Download - Digital Education Revolution In New South Wales DER-NSW Evaluation Conclusions on student and teacher engagement and ICT use Report presented by

DER-‐NSW, 2013 final report 1

Digital Education Revolution

In New South Wales

DER-NSW Evaluation

Conclusions on student and teacher engagement and ICT use

Report presented by

Dr Sarah Howard & Mr Adrian Mozejko (BSc, GDipEd, MSc, MEd)

University of Wollongong

July2013

DER-‐NSW, 2013 final report i

Evaluation team members The Evaluator

Dr Sarah Howard is a Lecturer in Education, in Information and Communication Technologies, at the University of Wollongong. She is the external evaluator for the DER-NSW.

The DER-NSW Evaluation Advisory Group

Dianne Butland, NSW Parents and Citizens Association

Chris Turner, Manager School Audit

Mark Watson, NSW Secondary Principals’ Council

Dianne Marshall, Program Director Digital Education Revolution NSW

Gerry McCloughan, Assistant Director, Educational Measurement and School Accountability

Terry O’Brien, Policy Leader Digital Education Revolution NSW

Bruce Stavert, Project Officer Digital Education Revolution NSW

Bill Tomlin, Manager Planning and Evaluation SEPE

This evaluation was commissioned by the Digital Education Revolution - NSW Program of the NSW Department of Education and Communities. The project is overseen by an Evaluation Advisory Group. This document presents comparison data gathered from year 10 students and teachers in 2011 and 2013. © State of New South Wales through the NSW Department of Education and Communities, DER-NSW, 2013

The Digital Education Revolution in NSW is funded by the Australian Government’s National Secondary School Computer Fund


1 Table of contents

1 Table of contents ................................................................................................................................................................... 1

2 Table of figures ...................................................................................................................................................................... 2

3 Executive Summary ............................................................................................................................................................. 4

4 Introduction ............................................................................................................................................................................. 5

5 Demographic data ................................................................................................................................................................. 7

5.1 Participation ........................................................................................................................................................ 7

5.1.1 Descriptive details of the sample .................................................................................... 7

5.2 Research questions ......................................................................................................................................... 13

5.2.1 RQ#1: How does the DER-NSW program influence teacher pedagogy? ..................... 13

5.3 RQ#2: Students’ engagement ..................................................................................................................... 22

5.3.1 School engagement ....................................................................................................... 22

5.3.2 ICT engagement ............................................................................................................ 23

5.4 Frequency of NSW-DER laptop usage at school ................................................................................ 27

5.5 Frequency of NSW-DER laptop and other computer usage outside of school ......................... 28

5.6 Confidence with ICT ..................................................................................................................................... 29

5.7 Student outcomes ........................................................................................................................................... 35

5.7.1 RQ#4: How does school leadership influence teacher and student engagement in the DER-NSW? ................................................................................................................................... 38

5.8 Summary ........................................................................................................................................................... 40

6 References ............................................................................................................................................................................. 43


2 Table of figures Figure 5-‐1 Teacher respondent gender distribution 2010 to 2013 .................................................................. 7

Figures 5-‐2a and 5.2b Year 10 student gender distribution in 2011 and 2013 ........................................... 8

Figure 5-3 Are you of Aboriginal or Torres Strait Islander origin? (%) .......................................................... 9

Figure 5-4 How many years have you been in an Australian school? (%) ....................................................... 9

Figures 5-5a and 5-5b Do students have access to computers and internet at home? (2013) ................ 10

Figure 5-6 Distribution of teachers' years of teaching ......................................................................................... 11

Figure 5-7 Distribution across KLAs ......................................................................................................................... 12

Figure 5-8 DER-NSW laptop use at school “How often do you use the DER-NSW laptop when teaching?” ............................................................................................................................................................................ 14

Figure 5-9 “How often do you use the DER-NSW laptop when teaching?” ................................................. 15

Figure 5-10 “On average, how frequently do you use a computer to perform the following?” ............ 16

Figure 5-11 Teachers’ beliefs about computers in teaching and learning .................................................... 18

Figure 5-12 “It is important for me to work with a computer” ......................................................................... 19

Figure 5-13 Frequency of using student-centred lessons .................................................................................... 20

Figure 5-14“It is important for students to work with a computer” ................................................................ 21

Figure 5-15 Students' engagement with their school: “My school is a place where…” ........................... 22

Figure 5-16 Student engagement with ICT ............................................................................................................... 23

Figure 5-17 Year 10 engagement with technology by home access to another computer ........................ 24

Figure 5-18 Year 10 ATSI and Non-ATSI engagement with technology (2011 and 2013) ....................... 25

Figure 5-19 Students by years attending an Australian school versus ICT engagement .......................... 26

Figure 5-20 Frequency of NSW-DER laptop usage at school ............................................................................ 27

Figure 5-21 Frequency of NSW-DER laptop usage outside of school ............................................................ 28

Figure 5-22 Changes in confidence versus ICT factors ....................................................................................... 29

Figure 5-23 Increasing confidence using ICT ......................................................................................................... 30

Figure 5-24 Students’ confidence performing ICT practice categories by home access to another computer ............................................................................................................................................................................... 31

Figure 5-25 Non-Indigenous and indigenous students’ confidence with ICT ............................................... 32


Figure 5-26 Confidence performing ICT practice categories by years attending an Australian school .................................................................................................................................................................................................. 33

Figure 5-27 Frequency of students performing ICT practices in school ........................................................ 34

Figure 5-28 Student outcomes when using ICT ....................................................................................................... 35

Figure 5-29 ATSI and Non-ATSI learning outcomes with ICT .......................................................................... 36

Figure 5-30 Students by years attending an Australian school and ICT learning outcomes ................... 37

Figure 5-31 Teachers’ agreement ................................................................................................................................ 38

Figure 5-32 Teachers’ confidence using the DER-NSW laptops in teaching ............................................... 39


3 Executive Summary The following report considers final results of the evaluation of the Australian Commonwealth Government’s Digital Education Revolution in New South Wales (DER-NSW) one-to-one laptop program, in relation to student and teachers’ developing engagement and use of ICTs. Formal evaluation of the program was initiated by the New South Wales Department of Education and Communities (NSW DEC), in collaboration with the University of Wollongong (UOW), in an effort to determine how the DER-NSW program affected education across the state. This was the fourth year of data collection, which has expanded upon the original evaluation (2010-2013) for reasons outlined below.

The DER-NSW program provided an opportunity to evaluate the impact of laptops on teaching and learning in a large sample of students, teachers and parents. The four primary research questions explored in this evaluation were:

• How does the DER-NSW program influence teacher pedagogy? • What is the influence on students’ understanding, skills, and attitudes? • What are the consequences on students’ educational outcomes? • How does school leadership influence teacher and student engagement?

The project employed an evaluation approach focusing on survey methodology and qualitative evidence. The basic design of the evaluation consisted of two data collection phases in 2010 and 2011, three in 2012 and two in 2013. Data collected in 2010 intended to establish a baseline, or benchmark, of students’ and teachers’ information and communication technology (ICT) use, knowledge and perceptions. The evaluation was conducted in two phases. Phase 1 2010 surveyed the entire Year 9 student population across the state, as well as all secondary school teachers. Phase 2 2010 included five school case studies across NSW.

Analysis of the 2012 data collection (year 11) indicated that the HSC was a possible inhibiting factor on capacity development into Stage 6 compared to that occurring in Stage 5 (years 9 and 10). The Evaluation Advisory Group requested the administration of an additional survey of year 10 2013 (teachers and students) to enable confirmatory reporting on:

1. The growth of teacher capacity over three years 2. The impact of this on Stage 5

The following report explores student data collected in 2011 and 2013 from participants in Year 10, Year 11 data from 2012 on key factors, and teacher data across all four years. Overall, students in 2013 reported the same engagement with school. However, levels of ICT engagement and confidence, and learning outcomes with ICT, all showed significant gains. While students reported an overall decrease in the usage of other computers and NSW-DER laptops outside of school, there were significant increases in the frequency of ICT usage.


Year 11 students in 2012 have reported significantly more engagement with school, significantly lower beliefs about ICT outcomes, as well as less ICT usage and utilisation of their DER-NSW laptops, and similar levels of ICT engagement (compared to Year 10 students in 2013). This finding is consistent with case study data in 2012 suggesting the nature of HSC curriculum did not encourage ICT integration, and that students were more comfortable preparing for their exams using more traditional methods.

Teachers reported a significant increase in frequency of laptop use over the four years, particularly gains in teachers using the laptop a few times a week to many times a day. This was reflected in a steady increase in frequency of different ICT-related tasks. In the 2013 findings, an increase in the use of student-centred practices was also observed. Overall, increasing laptop use and ICT-related tasks in teaching, coupled with increases in student-centred tasks suggests teaching change has occurred over the past four years. However, case study data has shown that much of this change has occurred in Years 9 and 10 (Stage 5), not in Years 11 and 12 (Stage 6). Teachers also reported increasing agreement and steady positive beliefs about their teaching and workplace over the four years. Teachers “have become more enthusiastic about teaching since participating in the DER-NSW program” and are becoming more confident in using the laptops.

4 Introduction The Digital Education Revolution in the NSW (DER-NSW) one-to-one laptop program was part of the Australian Commonwealth Government Digital Education Revolution initiative. The DER-NSW was funded through the National Secondary School Computer Fund, and aimed to provide a highly specialised wireless laptop to all students in years 9 to 12 by 2012.

The NSW Department of Education and Communities (NSW DEC) has initiated a formal evaluation of the educational impact of the DER-NSW. The project has been conducted as collaboration between the NSW DEC and the University of Wollongong (UOW). Details on the 2010 and 2011 stages of the evaluation have been published in:

Howard, S. K. & Carceller, C. (2011). DER-NSW 2010: Implications of the 2010 data collection. Sydney: New South Wales Department of Education and Communities.

Howard, S. K., Thurtell, E. & Gigliotti, A. (2012). DER-NSW evaluation: Report on the implications of the 2011 data collection. Sydney: New South Wales Department of Education and Communities.

Howard, S. K. & Gigliotti, A. (2013). DER-NSW evaluation: Report on the implications of the 2012 data collection. Sydney: New South Wales Department of Education and Communities.


Details on the evaluation conceptual framework, research design, trends and findings are presented in these reports. The current discussion presents a comparison of Year 10 student data collected from 2011 and 2013, Year 11 from 2012, and teacher data across all four years. The aim of this report is to confirm changes in students’ and teachers’ experiences with, and beliefs about, ICT after receiving the DER-NSW laptops.

The data presented in the following section includes student and teacher questionnaire results. Year 10 student data was gathered in Term 1 & 2 of the 2011 school year (March – May). Year 11 (Stage 6) student data was collected in 2012 from all government secondary and central schools in NSW with a Year 11 cohort. Finally, in 2013, Year 10 students were surveyed during the same period. Student data was collected through three online questionnaires. The questionnaires were available through Survey Monkey. In both years, URLs to the questionnaires were made available to Year 10 students through their NSW DEC email addresses. Teacher questionnaire data was collected at the same time, using the same procedure.

In 2011, students’ responses were collected through questionnaires; StuA and StuB (see Appendices A & B). The questionnaires were mostly completed in class and sometimes at home. The StuA questionnaire addressed students’ existing access to, and knowledge of, ICT, as well as their beliefs about core subject areas (e.g. English, Maths, etc). The StuB questionnaire gathered data about students’ initial understanding of ICT in learning. Students were randomly allocated by school to either questionnaire StuA or StuB. In 2013, the two questionnaires were combined. Teachers all completed the same questionnaire, in all four years. The questionnaire addressed their computer and laptop use, confidence using ICT, beliefs about technology integration in teaching and learning as well as beliefs about their school culture.


43% 41%

39% 41%

57% 59%

61% 59%

35

40

45

50

55

60

65

70

2010 2011 2012 2013

Male Female

5 Demographic data 5.1 Participation In 2013, 2,831 teachers completed the DER-NSW online questionnaire. A total of 10,602 students completed the DER-NSW evaluation questionnaires in 2011 (7,482 students completed student questionnaire A and 3,120 completed student questionnaire B). In 2013, there was moderate participant attrition in the student population, with 9,146 students participating. Full demographic data from past years can be found in the earlier reports, (links provided in the previous section).

5.1.1 Descriptive details of the sample The gender distribution of teachers has remained relatively stable across the years surveyed (Figure 5-1). While the 2010 distribution is consistent with the New South Wales Department of Education and Training Annual Report (2009) (57.4% female and 42.6% male, N = 10,389 secondary classroom teachers), the 2011 to 2013 participation rates in the questionnaire show a slightly increasing bias towards female teachers.

Figure 5-‐1 Teacher respondent gender distribution 2010 to 2013


In 2013, the Year 10 student population responding to this item consisted of 18% more female participants than males, as presented in Figures 5-2a. This shows a decrease in the proportion of males participating, as compared with the data from 2011 (26% more females participating than males). This is significantly skewed from normal gender distribution. In 2012, of those who reported their gender, the Year 11 student population consisted of approximately 21% more female participants than males (60% females and 39% males).

Figures 5-‐2a and 5.2b Year 10 student gender distribution in 2011 and 2013

Figure 5-2 shows that in 2013, 6.3% (n = 569) of responding Year 10 students indicated that they were of ATSI descent. This shows an increase in the proportion of responding ATSI students compared with the 2011 data (n = 583, 5.3%), and a significant increase from the 2010 data (n = 2,480, .8%). The 2012 data from Year 11 students also shows an increase in the proportion of responding ATSI students (n = 276, 5.8%).

The ABS (2013) school data shows 5% of students in NSW schools are of ATSI descent. Therefore, the samples from 2011 to 2013 approximately reflect the ATSI population, and may indicate increasing ATSI participation or an increasing ATSI population, or both.

37%

63%

2011

Male

Female

41%

59%

2013


Figure 5-3 Are you of Aboriginal or Torres Strait Islander origin? (%)

Note – 0.4% of students in 2011, and 0.5% of students in 2013 did not respond to this item.

From 2011 to 2013, there was a 0.7% increase in the number of students reporting that they had been in an Australian school for less than a year, and a 1% decrease in the number of students reporting that they had been in an Australian school for one to three years (see Figure 5-3). In 2012, Year 11 students indicated they had been in an Australian school for less than three years (n = 233, 9.2%) in a greater proportion than those reported in 2011 (n = 329, 7.9%) and 2010 (n = 1,060, 3.2%). This may be partially due to the Year 11 students having been in school for an extra year. In addition, this may also suggest that students who leave at the end of Year 10 are more likely to have been in school for more than three years.

Figure 5-4 How many years have you been in an Australian school? (%)

Note – 45% (n = 4022) of students in 2013, and 31% (n = 4658) of students in 2011 did not respond to this item.

94.7

3.6 0.6 1.1

93.7

4.3 0.6 1.4 0%

10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

No Aboriginal Torres Strait Islander Both

2011 2013

3.1 4.8 8.2

84.0

3.8 3.8 8.4

84.1

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

less than 1 year 1 to 3 years 3 to 7 years more than 7 years

2011 2013


Yes 91.6%

No 8.4%

Do you have access to a computer at home other than your DER-‐NSW laptop?

Yes 78.7%

No 14.8%

I Don’t Know 6.5%

At home can you connect to the internet on your DER-‐NSW laptop?

Figure 5-5a shows that in 2013, the majority (91.6%) of Year 10 students reported having access to another computer at home, which is similar to what was reported by Year 10 students in 2011 (93.2%), and is also similar to what was reported by Year 11 students in 2012 (94.0%).

Figures 5-5a and 5-5b Do students have access to computers and internet at home? (2013)

Students who identified as being enrolled in an Australian school for less than three years were less likely to have access to another computer at home (77% in 2011 and 71% in 2013), compared to students who have been in an Australian school for three or more years (94% in 2011 and 92% in 2013). While the statistics show a good coverage of home access to computers (Figures 5-5a and 5-5b), they do not indicate how much access a student may have to this other computer, such as how many family members share it. Further, we do not know what kind of internet access is at home, e.g. broadband, wireless, limited download, etc. Therefore, it is difficult to be clear about what kind of internet access students may have.

The 2013 teacher questionnaire response rate (n = 2,776) has slightly decreased from 2012 (n = 2,806) and decreased significantly from 2011 (n = 4,227) and 2010 (n = 4,575). The distribution of teachers’ years of teaching experience has remained relatively stable. Figure 5-6 shows the distribution of teachers over the years surveyed.


4%

9% 11%

14%

15%

11%

36%

2013 4%

9%

12%

14%

13% 12%

36%

2012

0-‐1

2-‐3

4-‐6

7-‐10

11-‐15

16-‐20

21+

5%

8%

13%

13%

12%

13%

36%

2011

Figure 5-6 Distribution of teachers' years of teaching

Note – between 1% and 2% of teachers did not respond to this item over the three years surveyed.

The over-representation of teachers who have been teaching for over 21 years has remained consistent from 2010 to 2013 (36%), which is consistent with the teaching population of NSW. 11-15 years has gradually increased from 12% in 2010 to 15% in 2013.

Distribution of teachers across Key Learning Areas (KLAs) has also remained relatively consistent in teacher respondents from 2010 to 2013 as can be seen from Figure 5-7.

4%

8%

12%

14%

12% 13%

36%

2010


Figure 5-7 Distribution across KLAs

Note – “Special Education” KLA category added in 2011 survey

15%

13%

19% 15%

7%

9%

20%

2%

2010 English

Mathematics

Science

HSIE

PDHPE

Creative Arts

Technology

Languages

15%

13%

16%

16%

7%

8%

18%

2% 5%

2011

15%

14%

14%

14% 8%

8%

17%

3% 6%

2012 English

Mathematics

Science

HSIE

PDHPE

Creative and Performing Arts Technology

Languages

Special Education

15%

14%

14%

15% 7%

9%

17%

2% 7%

2013


5.2 Research questions The following sections address teacher and student data that provide insight into each of the four research questions. A summary of key findings related to each research question is at the end of each section. Comparisons and trends from 2010 to 2013 will be noted. The Phase 1 reporting for 2011 will be concluded with an overall summary of the questionnaire findings.

5.2.1 RQ#1: How does the DER-NSW program influence teacher pedagogy? This question seeks to identify if, and how, teachers may have changed their practice, as a result of participation in the DER-NSW program. This analysis will look at how the DER-NSW laptops may have influenced teachers’ pedagogy in two areas: personal beliefs about teaching with ICT and beliefs related to KLAs. Research has shown positive experiences with, and attitudes towards, technology are predictive of teachers’ technology integration in the classroom. Individuals who have positive beliefs about technology integration are less likely to anticipate anxiety related to technology use in the classroom and more likely to develop their confidence through repeated use (Mueller, Wood, Willoughby, Ross, & Specht, 2008, p. 1533). Therefore the following analysis considers teachers’ use of laptops with their beliefs about ICT in teaching, as well as some pedagogical beliefs.

5.2.1.1 Teachers’ use of computers Overall use of the DER-NSW laptops by teachers in teaching has significantly changed from 20111 (M = 5.25, SD = 2.74) to 2012 (M = 5.13, SD = 2.92) and to 2013 (M = 5.46, SD = 2.96)2. A closer look at the distribution (see Figure 5-8) may indicate that those who are already using their laptops are now using them more, with 41.5% of teachers using their DER-NSW laptops “Many times a day”.

1 As the 2010 survey questionnaire referenced all computers instead of DER-‐NSW laptops specifically, 2010 has been omitted from the analysis 2 t(8191) = 8.134, p < .001, η2 = .002


13.6

5.0

3.1

4.0

5.0

5.7

12.2

9.8

41.5

14.2

5.3

4.0

5.3

7.3

6.4

13.9

10.8

32.9

12.3

3.8

3.8

4.8

7.6

8.0

19.7

10.8

29.4

0 10 20 30 40 50

Never

1-‐3 times a year

Once a term

Once a month

2-‐3 times a month

Once a week

2-‐4 times a week

Once a day

Many times a day 2011 2012 2013

Figure 5-8 DER-NSW laptop use at school “How often do you use the DER-NSW laptop when teaching?”

Note: Units are %


4.22

5.71

5.03

5.83

5.92

5.58

5.59

5.50

5.09

5.11

4.59

5.45

4.68

5.60

4.93

5.28

5.40

5.36

4.81

4.36

4.57

5.36

4.73

5.76

5.68

5.49

5.16

5.00

5.10

4.96

0 1 2 3 4 5 6 7 8

Other

English

Mathematics

Science

HSIE

PDHPE

Creative Arts

Technology

Languages

Special Education 2011 2012 2013

There were significant differences in laptop usage in teaching between the KLAs from 2011 to 20133 (see Figure 5-9).

Figure 5-9 “How often do you use the DER-NSW laptop when teaching?”

Note: 0 = ‘Never’, 6 =’2-4 times a week’, 8 =’Many times a day’

KLAs that showed a significant increase in laptop usage include Special Education, Technology, Creative and Performing Arts, HSIE, Mathematics, and English. The only overall decreases reported were in English and Other.

Figure 5-10 shows significant changes in the frequency of teachers’ computer usage for specific tasks over the four years4 surveyed across a range of tasks.

3 2011: F(8, 3271) = 9.03, p < .001, η2 = .022; 2012: F(8, 2256) = 4.01, p < .001, η2 = .014; 2013: F(9, 2336) = 6.07, p < .001, η2 = .023 4 Some items were not surveyed in 2010.


6.72

6.79

5.82

4.87

3.29

4.46

3.93

2.54

2.92

3.19

2.00

3.50

2.33

6.76

6.81

5.66

4.47

2.97

4.09

3.75

2.14

2.51

3.11

1.70

3.32

1.87

6.40

6.49

5.28

3.97

2.59

3.75

3.27

2.10

2.24

2.99

1.49

2.88

1.78

6.40

6.62

3.80

2.33

3.49

2.00

2.20

3.39

1.20

1.63

0 1 2 3 4 5 6 7 8

Research and develop lesson plans and curriculum design.

Develop instructional materials .

Provide online resources for students to access

Use PowerPoint in classroom instruction

Create and/or maintain website(s) as part of one or more of your classes

Use group email lists.

Communicate with parents and/or students )

Participate in online discussion forums and interact with other educators online

Share your teaching resources online for other educators to access and use

Engage in self-‐assessment

Post examples of your students' work online to share

Answer student questions about homework or assignments through email.

Use online simulation sites to use equipment that you do not otherwise have

access to for lessons or projects

2010 2011 2012 2013

Figure 5-10 “On average, how frequently do you use a computer to perform the following?”

Note: 0 = ‘Never’, 6 =’2-4 times a week’, 8 =’Many times a day’


Overall, results show that teachers in 2013 were performing all activities significantly more frequently5, with the exception of “Engage in self-assessment”, which is now occurring significantly less6. Findings show that teachers most frequently “develop instructional materials” and “research and develop lesson plans” using a computer. The least frequently performed tasks have been “use online simulation sites” and “post examples of student work online”. However, many of the more online focused tasks, such as maintaining a website and sharing teaching resources online have shown steady increase over the past four years. This is likely a result of consistent access to a computer, increased internet access in schools, teacher up-skilling, or a sampling bias towards teachers who are more likely to perform this kind of task.

5.2.1.2 Teachers’ beliefs about computers In regard to how important it is to work with a computer, teachers’ beliefs were overall consistent over the four years (see Figure 5-11).

Teachers were in “agreement” to “strong agreement” in regard to it being important for them, and for students, to work with computers. They showed slightly weaker agreement that computers should be a “fun” part of learning and teaching. In real terms, there was little change in teachers’ beliefs between 2010 and 2013. Statistically there were small differences between each of the four years, but this is mostly attributed to a small drop in agreement on the items, followed by a small rise again7.

5 F(3, 12873) = 48.02, p < .001, η2 = .011 6 F(3, 13398) = 70.39, p < .001, η2 = .003 7 Comparison of means shows very small but significant differences between the years on all of the variables except for the first. In order of how they appear in Figure 2-‐11: (2, 10,931) = 1.90, p = .150, η2 = .000; F(2, 10,925) = 10.99, p < .001, η2 = .002; F(2, 10,893) = 9.09, p < .001, η2 = .002; F(2, 10,902) = 10.62, p < .001, η2 = .002


3.65

3.43

3.14

3.27

3.41

3.15

3.61

3.40

3.09

3.22

3.36

3.13

3.59

3.37

3.08

3.21

3.36

3.13

3.61

3.44

3.15

3.27

2.86

3.10

1 2 3 4

It is imporatnt to me to work with a computer

It is important for students to work with computers

I think working and teaching with computers is fun

I thinkl working with a computer should be a fun part of student

learning

I use a computer because I'm interested in computer

technology

Time goes by quickly when I am working with a computer

2010 2011 2012 2013

Figure 5-11 Teachers’ beliefs about computers in teaching and learning

1 = “Strongly disagree”, 2 = “Disagree”, 3 = “Agree” and 4 = “Strongly agree”


3.66

3.54

3.69

3.67

3.66

3.65

3.69

3.64

3.65

3.64

3.48

3.60

3.65

3.47

3.67

3.64

3.62

3.62

3.55

3.50

3.60

3.61

3.48

3.59

3.63

3.61

3.69

3.56

3.47

3.62

3.65

3.59

3.63

3.60

3.60

1 2 3 4

English

Mathematics

Science

HSIE

PDHPE

Creative Arts

Technology

Languages

Special Education

2010 2011 2012 2013

There has been some variation in teachers’ beliefs about the use of computers between the KLAs (see Figure 5-12).

Figure 5-12 “It is important for me to work with a computer”



2.08 1.83 1.83 2.03 1

2

3

2010 2011 2012 2013

Findings show that, over the four years, all subjects except for Special Education have shown slight increases in teachers’ belief, indicating that it is increasingly important for them to use computers.

Overall, results show that teachers’ beliefs about the use of computers have significantly increased over the four years (2010, M = 3.61, SD = .60: 2011, M = 3.59, SD = .63: 2012, M = 3.60, SD = .61: 2013, M = 3.65, SD = .60) 8. Similar patterns9 are visible in teachers’ beliefs, within the KLAs, in regard to how important it is for students to use computers (see Figure 5-14). Results suggest that teachers’ beliefs about students’ use of computers was significantly less positive in 2011 (M = 3.37, SD = .67) and 2012 (M = 3.40, SD = .66) as compared to 2010 (M = 3.44, SD = .63) and 2013 (M = 3.43, SD = .68) 10.

Results suggest that shifts in teachers’ pedagogy, to embrace a more student-centred approach to teaching when using laptops in the classroom, has remained constant between 2011 and 2012.

Figure 5-13 Frequency of using student-centred lessons

0 = “Never”, 1 = “Less than half”, 2 = “More than half”, 3 = “Every lesson”

Figure 5-13 shows that the use of student-centred lessons has decreased slightly between 2010 and 2012. However, it can be observed that this change was primarily between 2010 and 2011, with use of student-centred practices remaining constant between 2011 and 2012, and then significantly increasing in 201311.

8 F(3,13543) = 5.24, p < .005, η2 = .001 9 F(3,13535) = 7.62, p < .001, η2 = .002 10 t(6957) = 2.48, p < .05 11 F(1, 4723) = 86.37, p < .001, η2 = .018


3.66

3.54

3.69

3.67

3.66

3.65

3.69

3.64

3.65

3.64

3.48

3.60

3.65

3.47

3.67

3.64

3.62

3.62

3.55

3.50

3.60

3.61

3.48

3.59

3.63

3.61

3.69

3.56

3.47

3.62

3.65

3.59

3.63

3.60

3.60

1 2 3 4

English

Mathematics

Science

HSIE

PDHPE

Creative Arts

Technology

Languages

Special Education

2010 2011 2012 2013

Figure 5-14“It is important for students to work with a computer”



5.3 RQ#2: Students’ engagement This section presents result on research question #2, considering students’ engagement in school and with ICTs.

5.3.1 School engagement Figure 5-‐15 shows that Year 10 students in 2013 show the same engagement with school as reported in 2011. However, Year 11 students in 2012 showed significantly more engagement with school (Y11 2012, M=2.97, SD=.71; Y10 2013, M=2.88, SD=.77)12 compared to the Year 10 cohorts while their perceptions of success did not differ.

Figure 5-15 Students' engagement with their school: “My school is a place where…”

Note. ‘1’ = Strongly disagree, ‘4’ = Strongly agree

12 F(1, 5174) = 37.89, p < .001, η2 = .010, with “I am a success as a student” not included due to no significance

2.71

2.70

2.91

2.83

3.07

3.04

2.94

2.79

2.77

2.98

3.00

3.15

3.04

3.05

2.71

2.65

2.93

2.81

3.09

3.05

2.93

1 2 3 4

I really like to go each day.

I iind learning is a lot of fun.

I am treated with respect by other students.

I am given the chance to do work that really interests me.

I have acquired skills that will be of use to me when I leave school.

I am a success as a student.

Teachers help me to do my best. 2011 2012 2013


5.3.2 ICT engagement The following figures present engagement in ICT for the entire participating Year 10 populations. In addition, some figures have the Year 11 student data from 2012 added for general reflection and comparison. Three minority sub-groups from the Year 10 cohorts have been further explored with analyses of their ICT confidence and engagement. These sub-groups include: • No access to another computer at home • Aboriginal and Torres Strait Islanders (ATSI) • Students who have been in an Australian school for less than three years (< 3Y) Figure 5-16 shows that there were significant increases in Year 10 student engagement with ICT from 2011 to 2013 (2011, M=2.86, SD=.85; 2013, M=3.03, SD=.82)13. In addition, in 2012, Year 11 students reported slightly higher levels of ICT engagement than Year 10 in 2011, but slightly lower engagement than Year 10 students in 2013.

Figure 5-16 Student engagement with ICT


There are two possible influences on differences in ICT engagement between the Year groups. First, in regard to Year 10 students between 2011 and 2013, it is likely increased use of laptops and computers by their teachers and higher quality of the laptops are affecting their engagement. Between, Year 10 2011 and Year 11 2012, it is likely their increased familiarity with using ICT affects their engagement. However their teachers’ use of laptops and computers in Year 11 would be less frequent than in Year 10.

13 t(4825) = -‐9.15, p < .001

3.03

3.16

2.92

3.03

2.96

3.08

2.87

3.01

2.80

3.00

2.75

2.91

1 2 3 4

It is very important to me to work with a computer

I think working with a computer is fun

I use a computer because I am very interested

Time goes by quickly when I am working with the computer

2011

2012

2013


Figure 5-17 shows, in 2011 students with “No access” to another computer at home reported no differences in engagement with ICT as compared to students with “Access” to another home computer (Access, M=2.87, SD=.85; No, M=2.87, SD=.95). This also held true for 2013 (Access, M=3.04, SD=.82; No, M=2.97, SD=1.00) and may be due to all students having access to a computer, owing to the NSW-DER laptop program. Figure 5-17 also shows that for students who reported having “Access” to another computer at home, there were significant increases in their engagement with ICT from 2011 to 2013 (2011, M=2.87, SD=.85; 2013, M=3.04, SD=.82)14. For students who reported having “No access” to another computer at home, there was only one item which returned a significant increase from 2011 to 2013; “It is very important for me to work with a computer” (2011, M=2.82, SD=.92; 2013, M=2.98, SD=1.00).15 Students with “No access” to another computer at home were increasingly viewing working with a computer as important.

Figure 5-17 Year 10 engagement with technology by home access to another computer


Figure 5-18 shows that in 2011, there were no significant differences in ICT engagement between ATSI and Non-ATSI students (Non-ATSI, M=2.86, SD=.84; ATSI, M=2.78, SD=1.05)16. However in 2013, Non-ATSI students showed significantly more engagement in

14 t(4430) = 25.95, p < .001 15 t(914) = -‐2.11, p < .05 16 t(131) = -‐.79, p = .413

2.91

2.75

3.00

2.80

3.03

2.92

3.16

3.03

2.88

2.76

3.00

2.82

2.96

2.83

3.09

2.98

1 2 3 4





Access 2011 Access 2013 No access 2011 No access 2013


ICT for all items except for the first: “Time goes by quickly when I am working with the computer” (Non-ATSI, M=3.03, SD=0.84; ATSI, M=2.97, SD=1.04)17. Figure 5-18 also shows that from 2011 to 2013, there were significant gains in Non-ATSI student engagement with ICT (2011, M=2.86, SD=.84; 2013, M=3.03, SD=.84)18. However, for ATSI students (2011, M=2.78, SD=1.05; 2013, M=2.91, SD=1.03)19 there was only one significant gain: “Time goes by quickly when I am working with the computer” (2011, M=2.75, SD=1.13; 2013, M=2.97, SD=1.04)20. Nevertheless, there was no significant difference between ATSI students in 2013 compared to Non-ATSI students in 2011 (ATSI 2013, M=2.91, SD=1.03; Non-ATSI 2011, M=2.86, SD=0.84)21. Overall, ATSI students have reported an increase in their engagement with ICT. While there is still a gap, it should be noted that ATSI students are now reporting the same engagement with technology as Non-ATSI students were reporting in 2011.

Figure 5-18 Year 10 ATSI and Non-ATSI engagement with technology (2011 and 2013)


17 t(537) = -‐1.19, p = .236 18 t(6568) = -‐9.98, p < .001 19 t(191) = -‐1.04, p = .299 20 t(181) = -‐1.98, p < .05 21 t(412) = -‐.301, p = .551

2.92

2.75

3.00

2.80

3.03

2.92

3.17

3.04

2.75

2.68

2.94

2.75

2.97

2.79

3.00

2.87

1 2 3 4





Non-‐ATSI 2011 Non-‐ATSI 2013 ATSI 2011 ATSI 2013


Figure 5-19 shows no significant differences in ICT engagement between the groups in 2011 (> 3Y, M=2.90, SD=.86; < 3Y, M=2.92, SD=1.01)22. However, in 2013 there was significantly less engagement with ICT across all items for students who had attended an Australian school for less than three years (> 3Y, M=3.10, SD=.83; < 3Y, M=2.90, SD=1.07)23. Figure 5-19 also shows that from 2011 to 2013, there were significant increases in ICT engagement across the board for students with three or more years of Australian schooling (2011, M=2.90, SD=.86; 2013, M=3.10, SD=.83)24. However, there were no significant differences in ICT engagement for students reporting less than three years. One item “Time goes by quickly…” did return a large mean difference (2011, M=3.04; 2013, M=2.91), but there was no statistical significance25.

Figure 5-19 Students by years attending an Australian school versus ICT engagement

Note - ‘1’ = Strongly disagree, ‘4’ = Strongly agree

22 t(44) = .326, p = .683 23 t(359) = 3.56, p = .012 24 t(2570) = -‐7.53, p < .001 25 t(441) = -‐.80, p = .270

2.96

2.79

3.01

3.10

3.08

2.99

3.20

3.11

3.04

2.80

3.00

2.88

2.91

2.77

2.96

2.97

1 2 3 4





> 3Y 2011 > 3Y 2013 < 3Y 2011 < 3Y 2013


5.4 Frequency of NSW-DER laptop usage at school Considering how students’ understanding, skills and attitudes towards ICT and ICT integration may change, it is important to know how much they use ICT and how confident they feel using it. In 2013, 47.3% of participants indicated that they used a computer at school “Many times a day”, showing a significant increase (9.5%) from 38.8% in 2011. This difference may be primarily attributed to the significant decrease in the number of students using a computer “Once a day” to “2-4 times per week”, which was 28.1% in 2013, down from 36.1% in 2011 (8%). Students indicating only using a computer “2-3 times a month” to ‘Never’ did not significantly change from 2011 (16.7%) to 2013 (17.5%). Students, who were already using their DER-NSW laptops at least once a week, may now be reporting that they are using them more often.

Figure 5-20 reports that Year 11 2012 students are using their laptops much less at school as compared to Year 10 students. Again, this point to two possible influencing factors: 1) the usability of their laptops and 2) frequency of laptops and computers used in class by their teachers.

Figure 5-20 Frequency of NSW-DER laptop usage at school

7.2%

2.0%

1.9%

2.3%

4.0%

7.1%

16.4%

11.7%

47.3%

16.7%

8.3%

5.3%

6.5%

7.7%

8.9%

17.2%

10.0%

19.3%

6.8%

1.8%

1.5%

2.4%

4.2%

8.4%

20.1%

16.0%

38.8%

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%

Never

1-‐3 times a year

Once a term

Once a month


Once a week

2-‐4 times a week

Once a day

Many times a day

2011 2012 2013


5.5 Frequency of NSW-DER laptop and other computer usage outside of school

Figure 5-21 shows a 6.5% decrease in 2013 in the frequency of NSW-DER laptop use outside of school for students reporting a usage of “Many times a day” to “2-4 times a week” (62.2% in 2011 and 55.7% in 2013). There was also a 6.9% increase in 2013 for students reporting low usage; “2-3 times a month” to “Never” (25.5% in 2011 and 32.4% in 2013). The most significant increase in 2013 was for students reporting “Never” (3.8%) while the most significant decrease in 2013 was for students reporting “Once a day” (3.8%). Students reporting “Once a week” did not significantly change.

Figure 5-21 also reveals that Year 11 students are using their laptops much less outside of school as compared to Year 10 students.

Figure 5-21 Frequency of NSW-DER laptop usage outside of school

Students are reporting a decrease in the usage of other computers (4.9%) and NSW-DER laptops outside of school from 2011 to 2013. This may be due to the recent uptake of mobile/smart phone and tablet computer usage, which students may not regard as a “computer”.

16.0%

3.4%

2.1%

4.5%

6.4%

12.0%

21.3%

17.8%

16.6%

29.4%

6.5%

3.7%

4.9%

7.4%

11.1%

16.1%

10.6%

10.3%

12.2%

2.6%

1.8%

3.3%

5.6%

12.4%

23.3%

21.6%

17.3%

% 5% 10% 15% 20% 25% 30% 35%

Never

1-‐3 times a year

Once a term

Once a month


Once a week

2-‐4 times a week

Once a day

Many times a day 2011 2012 2013


5.6 Confidence with ICT The following graph shows students’ confidence across three categories of ICT related tasks: Productivity, Creation, and Data Manipulation. The following items under the heading “How well can you do each of the following…” have been used for each category: Factor 1 – Productivity

“... Write a first draft” “... Edit your written work” “... Take notes” “... Write, send, and receive emails” “... Organise your work on the computer” “... Research information on people, things, or ideas” “... Create simple PowerPoint presentations” “... Download images” “... Download pictures”

Factor 2–Data Manipulation “How well can you do each of the following…”

“... Enter data into a database or spreadsheet” “... Use a spreadsheet to plot a graph” “... Create a database or spreadsheet”

Factor 3 – Creation “How well can you do each of the following…”

“... Create multi-media presentations” “... Make a podcast” “... Create a movie”

Figure 5-22 Changes in confidence versus ICT factors

Note – ‘1’ = I don’t know what this is, ‘2’ = I know what this is but I can’t do it,’3’ = I can do this with help…, ‘4’ = I can do this very well by my self

3.47 3.01 3.22 3.84 3.51 3.26 1

2

3

4 2011 2013


Students are indicating an overall significantly increasing confidence in ICT practices (Figure 5-22; 2011, M = 3.19, SD = .77; 2013, M = 3.66, SD = .65)26. From 2011 to 2013, every component of each factor (except one) returned significant increases in ICT confidence27. The factor returning a significant decrease was “Make a podcast”28. It is important to note that levels of confidence on these tasks may have been higher than actual levels of expertise due to self-reporting, since students may have felt they were better at using ICT than they truly were (Neuman, 2006).

Figure 5-23 Increasing confidence using ICT

Note – ‘1’ = I don’t know what this is, ‘2’ = I know what this is but I can’t do it,’3’ = I can do this with help…, ‘4’ = I can do this very well by my self

26 F(1, 14976) = 1878.11, p < .001, ηp2 = .108 27 Each component recorded p < .001 with ηp2 ranging from .042 to .226 28 F(1, 14468) = 75.76, p < .001, ηp2 = .005

3.5 3.4 3.5 3.5 3.5 3.5 3.5 3.5 3.4 3.2 3.0 2.9 3.3 3.2 3.2 3.8 3.8 3.8 3.9 3.8 3.9 3.9 3.9 3.8 3.6 3.5 3.5 3.6 2.8 3.4 1

2

3

4

2011 2013


3.38 2.99 2.88 3.87 3.54 3.28 3.22 2.78 2.78 3.52 3.11 2.98 1

2

3

4 Access 2011 Access 2013 No Access 2011 No Access 2013

Figure 5-24 Students’ confidence performing ICT practice categories by home access to another computer

Note -‐ ‘1’ = I don't know what this means, ‘2’ = I know what this means but I can't do it, ‘3’ = I can do this with help from someone, ‘4’ = I can do this very well by myself

Figure 5-24 shows that students with “Access” to another computer at home are significantly more confident with ICT. In 2011, there were significant differences in Productivity (Access, M=3.38, SD=.65; No, M=3.22, SD=.86)29 and Data Manipulation (Access, M=2.99, SD=.84; No, M=2.78, SD=1.00)30 between those with “Access” to another computer and those with “No access”. In 2013, significant differences between the two group groups continued (Access, M=3.56, SD=.68; No, M=3.20, SD=1.02)31.

For both groups of students, reported gains from 2011 to 2013 were significant across all factors (Access 2011, M=3.08, SD=.83; Access 2013, M=3.56, SD=.68: No access 2011, M=2.93, SD=.98; No access 2013, M=3.20, SD=1.02)32. In addition, students reporting “No access” in 2013 now have approximately the same confidence as students who were reporting “Access” in 2011 (Access 2011, M=3.08, SD=0.83; No access 2013, M=3.20, SD=1.02).

29 F(1, 7758) = 24.13, p < .001, ηp2 = .003 30 F(1, 7743) = 27.74, p < .001, ηp2 = .004 31 F(1, 7268) = 195.65, p < .001, ηp2 = .026 32 For students with “Access”: F(1, 14016) = 2071.23, p < .001, ηp2 = .125; For students with “No access”: F(1, 1037) = 27.65, p < .001, ηp2 = .026


Figure 5-25 Non-Indigenous and indigenous students’ confidence with ICT

Note - ‘1’ = I don't know what this means, ‘2’ = I know what this means but I can't do it, ‘3’ = I can do this with help from someone, ‘4’ = I can do this very well by myself

Figure 5-25 shows that indigenous (ATSI) students have reported significantly less ICT confidence. In 2011, ATSI students reported feeling significantly less confident with ICT across all factors (Non, M=3.08, SD=.83; ATSI, M=2.83, SD=.98)33. In 2013, this trend continued and deepened (Non, M=3.56, SD=.71; ATSI, M=3.12, SD=.98)34.

Both groups showed significant gains from 2011 to 2013 in ICT confidence, although the gains for Non-ATSI students were stronger (Non 2011, M=3.08, SD=0.83; Non 2013, M=3.56, SD=.71: ATSI 2011, M=2.83, SD=.98; ATSI 2013, M=3.12, SD=.98)35. Overall, ATSI students have reported real gains in confidence in all three areas. It is important to note that in 2013, ATSI students were feeling as confident with ICT as non-ATSI students had been reporting in 2011.

33 F(1, 7668) = 38.54, p < .001, ηp2 = .014 34 F(1, 7257) = 214.18, p < .001, ηp2 = .080 35 For Non-‐ATSI: F(1, 14042) = 2079.94, p < .001, ηp2 = .123; For ATSI: F(1, 778) = 18, p < .05, ηp2

= .022

3.39 2.99 2.88 3.87 3.53 3.28 3.07 2.76 2.68 3.37 3.04 2.95 1

2

3

4 2011 Non

2013 Non

2011 ATSI

2013 ATSI


Figure 5-26 Confidence performing ICT practice categories by years attending an Australian school

Note - ‘1’ = I don't know what this means, ‘2’ = I know what this means but I can't do it, ‘3’ = I can do this with help from someone, ‘4’ = I can do this very well by myself

Figure 5-26 shows that both groups of students, those that had been in an Australian school for less than three years and those that had been in for more than 3 years, felt relativity confident performing most ICT related tasks. Whilst both groups of students reported feeling confident, those who had attended for more than three years felt significantly more confident across all factors in 2011 (> 3Y, M=3.11, SD=.84; < 3Y, M=2.85, SD=.98)36, and this effect has deepened in 2013 (> 3Y, M=3.56, SD=.74; < 3Y, M=3.05, SD=1.00)37.

For students attending Australian schools for less than 3 years, there were no significant changes from 2011 to 2013. For students of more than 3 years, there were significant increases across all factors (2011, M=3.11, SD=.84; 2013, M=3.56, SD=.74)38. It is important to note that in 2013, students of less than three years were feeling as confident with ICT as students of more than three years had been reporting in 2011.

36 F(1, 4186) = 16.03, p < .001, ηp2 = .011 37 F(1, 4034) = 106.11, p < .001, ηp2 = .072 38 F(1, 7696) = 893.37, p < .001, ηp2 = .101

3.39 2.99 2.88 3.07 2.76 2.68 3.87 3.53 3.28 3.37 3.04 2.95 1

2

3

4 2011 > 3Y

2011 < 3Y

2013 > 3Y

2013 < 3Y


Figure 5-27 Frequency of students performing ICT practices in school

Scale ‘0’ = Never, ‘4’ = 1-3 times a month,’6’ =2-4 times a week, ‘8’ = Many times a day

Figure 5-27 shows that there were significant increases from 2011 to 2013 in students’ ICT usage (2011, M=2.33, SD=2.40; 2013, M=4.10, SD=2.63)39.

Overall, Year 11 students in 2012 reported significantly less ICT usage compared to Year 10 students in 2013 (Creation returned no significant difference while Productivity40 and Data Manipulation41 reported significantly less usage for Year 11 in 2012).

Comparing Year 11 2012 to Year 10 2010, Year 11 reported significantly less Productivity42 and Data Manipulation43. Year 11 did show slightly higher frequency of Creation tasks, but this did not return any significant difference.

39 F(1, 11938) = 2784.74, p < .001, ηp2 = .158 40 F(1, 8936) = 1293.78, p < .001, ηp2 = .114 41 F(1, 8892) = 61.32, p < .001, ηp2 = .006 42 F(1, 6567) = 730.53, p < .001, ηp2 = .092 43 F(1, 6491) = 450.90, p < .05, ηp2 = .017

2.75 1.43 1.96 2.60 2.58 2.57 4.95 3.11 2.53 1

2

3

4

5

6

7

8

2011 2012 2013


5.7 Student outcomes Year 10 students continued to indicate agreement that using a computer improved their learning outcomes. Figure 5-‐28 shows that there were significant gains across all items from 2011 to 2013 (2011, M=2.72, SD=1.02; 2013, M=2.87, SD=.94)44.

Year 11 students in 2012 reported significantly lower ICT outcomes across all items (Year 11 2012, M=2.41, SD=1.00; Year 10 2013, M=2.87, SD=.94)45. Again, this possibly reflects differences in curriculum aims and desired outcomes.

Figure 5-28 Student outcomes when using ICT


This increase is also possibly a function of students having more exposure to technology and therefore more confidence, as more confidence relates to more positive attitudes (Mueller, Wood, Willoughby, Ross, & Specht, 2008). This would be compounded through teachers’ increased use of technology in the classroom, thus developing a possible stronger connection between technology use and learning in Year 10. 44 F(1, 8778) = 53.29, p < .001, ηp2 = .006 45 F(1, 7581) = 181.44, p < .001, ηp2 = .041

2.91

3.07

2.76

2.69

2.93

2.47

2.62

2.40

2.18

2.38

2.72

2.95

2.54

2.54

2.84

1 2 3 4

My work is more creative when I use a computer

Using a computer helps me to be more organised

My writing improves when I use a computer

I feel that I understand what we are doing in class better when I use a computer

Using computers in school makes learning more fun

2011 2012 2013


The following figures look at the differences between outcomes of ATSI and Non-ATSI students, and students who have been in an Australian school for less than three years compared to students of more than three years. Since there were no significant differences in student outcomes between those who reported not having access to another computer at home versus those who did, this item will not be reported on.

Figure 5-29 ATSI and Non-ATSI learning outcomes with ICT


Figure 5-29 shows that in both 2011 and 2013, the differences between ATSI and Non-ATSI ICT outcomes were significant (Non 2011, M=2.71, SD=1.01; ATSI 2011, M=2.79, SD=1.09: Non 2013, M=2.88, SD=0.92; ATSI 2013, M=2.74, SD=1.14)46.

46 For 2011: F(1, 2883) = 15.54, p < .001, ηp2 = .015; For 2013: F(1, 5838) = 27.26, p < .001, ηp2 = .013

2.77

2.85

2.67

2.63

2.79

2.79

2.93

2.61

2.66

2.95

2.91

3.09

2.77

2.69

2.94

2.72

2.96

2.53

2.53

2.84

1 2 3 4





Using computers in school makes learning more fun

2011 Non 2013 Non 2011 ATSI 2013 ATSI


2.77

2.87

2.58

2.56

2.80

2.78

2.86

2.57

2.51

2.90

2.96

3.11

2.78

2.73

2.96

2.76

2.98

2.56

2.57

2.87

1 2 3 4





Using computers in school makes learning more fun 2011 > 3Y 2013 > 3Y 2011 < 3Y 2013 < 3Y

There were significant gains across all items from 2011 to 2013 for Non-ATSI students (2011, M=2.71, SD=1.01; 2013, M=2.88, SD=0.92)47 while there were no significant gains for ATSI students. It is important to note that in 2013, ATSI students were reporting the same ICT learning outcomes as non-ATSI students had been reporting in 2011.

Figure 5-30 Students by years attending an Australian school and ICT learning outcomes


Figure 5-30 shows that in both 2011 and 2013, the differences between these groups’ outcomes were significant (> 3Y 2011, M=2.75, SD=1.01; < 3Y 2011, M=2.72, SD=1.06: > 3Y 2013, M=2.91, SD=0.93; < 3Y 2013, M=2.72, SD=1.07)48. There were significant gains across all items with small effect sizes from 2011 to 2013 for students who were at an Australian school for more than three years (2011, M=2.75, SD=1.01; 2013, M=2.91, SD=0.93)49. There were no significant differences for students of less than three years.

47 F(1, 8224) = 58.15, p < .001, ηp2 = .007 48 2011: F(1, 1497) = 7.56, p < .05, ηp2 = .013; For 2013: F(1, 3241) = 12.22, p < .001, ηp2 = .011 49 F(1, 4423) = 30.94, p < .001, ηp2 = .006


5.7.1 RQ#4: How does school leadership influence teacher and student engagement in the DER-NSW?

Literature indicates that leadership is a key factor in teachers feeling supported in using ICT and thus fostering positive views of its use (Law, Pelgrum, & Plomp, 2008). Strong leadership may also promote teachers’ engagement with the school and enables more successful integration of new programs. Teachers participating in the questionnaire were largely positive, and increasingly positive, over the four years assessed. The following section presents a comparison of results of teacher engagement in the DER-NSW program and beliefs about how school leaders supported the laptop program.

Teachers overall reported agreement and positive beliefs about their teaching and workplace (see Figure 5-31).

Figure 5-31 Teachers’ agreement

Note - ‘1’ = Strongly disagree, ‘4’ = Strongly agree, scale of figure is reduced to illustrate changes over the three years.

The above figure shows that there was a significant increase in agreement across the three items between 2010 and 201150. Reported beliefs were not significantly different between 2011 and 201251 or 2012 to 201352.

In addition to positive beliefs about teaching and the school community, results indicate that teachers “have become more enthusiastic about teaching since participating in the DER-NSW program”. Results show a slightly increasing agreement between 2010 (M = 2.34, SD = .86) and 2011 (M = 2.38, SD = .88). Larger gains were observed between 2011 and 2012 (M =

50 All three items, p < .001, η2 = .011, .05, .05, .03 (respectively) 51 All three items, p > .400, η2 = .000 52 All three items, p > .065, η2 < .001

3.00

3.18 3.20 3.21

3.13

3.27 3.29 3.31 3.22

3.29 3.30 3.33

2.90

3.10

3.30

3.50

2010 2011 2012 2013

I feel respected in the school community

I enjoy coming to work

I feel successful as a teacher


2.84

2.98 3.02

3.09

2.91

3.09 3.12

3.19

2.8

3

3.2

3.4

2010 2011 2012 2013

I feel coniident that I can effectively use the laptop in my classes

I feel that I have the necessary skills to use the laptops to my everyday teaching

2.44, SD = .90) and 2012 to 2013 (M = 2.56, SD = .91). Overall, there was a significant increase in “enthusiasm” between 2010 and 201353.

Teachers reported increasing confidence using the laptops in their teaching, over the three years (see Figure 5-32).

Figure 5-32 Teachers’ confidence using the DER-NSW laptops in teaching

Note - ‘1’ = Strongly disagree, ‘4’ = Strongly agree, scale of figure is reduced to illustrate changes over the three years.

On both items, feeling confident and having the necessary skills, teachers have reported significant yet gradual increases in agreement between 2010 and 201154, as well as between 2011 and 201255 and 2012 to 201356. Teachers are becoming more confident in using the laptops.

53 F(3, 11594) = 18.14, p < .001, η2 = .011 54 F(2, 9373) = 62.92, p < .001, η2 = .013 55 F(2, 9318) = 42.56, p < .001, η2 = .009 56 Feeling confident: F(1, 4686) = 21.585, p < .001, η2 = .008; and skills: F(1, 4714) = 18.103, p < .001, η2 = .008


5.8 Summary The aim of this final report was to confirm trends observed over the three years of the program evaluation. The above reporting has presented differences between Year 10 (Stage 5) and Year 11 (Stage 6) students, and three sub-groups between 2011 and 2013 on Research Question #2 (students skills, beliefs and understanding) on four points: 1) Engagement, 2) Frequency of computer use, 3) Confidence using ICTs, and 4) Learning outcomes using ICTs. We have also presented the teachers’ results on Research Questions #1, impact on teachers’ pedagogy, and #4, influence of leadership. The four main findings from the DER-NSW evaluation, confirmed in the above analysis, were 1) significant differences between disadvantaged sub-groups and the larger student population on ICT engagement and use, 2) students in Year 10 (Stage 5) report more frequent use of laptops and stronger positive beliefs about ICT use than those in Year 11 (Stage 6); 3) teachers’ positive beliefs about ICT and frequency of use have positively increased; and, 4) teachers are reporting increases in student-centred practices, as a result of laptop use. First, in regard to the three sub-groups, in 2011, there was no significant difference in ICT engagement between ATSI and Non-ATSI students. However, in 2013, while ATSI students still showed significantly less ICT engagement, ATSI students are now reporting the same engagement with technology as Non-ATSI students were reporting in 2011. This finding suggests that through participation in the DER NSW, whether it be personal use, increased exposure or awareness of ICTs that their engagement has increased Further research is needed to determine which factors are influencing increased engagement. In addition, students without home access to another computer were increasingly viewing working with a computer as important. In regard to <3Y students, they have continued to report significantly lower ICT engagement than the wider population. Sub-groups continued to show significantly lower outcomes in 2013. Nevertheless, gains in ATSI students reported similar beliefs about ICT learning outcomes to non-ATSI students had reported in 2011. Again, this finding suggests a positive impact on ATSI learning outcomes from participation in the DER-NSW. Further research is needed to determine the actual effect. Findings from these three sub-groups varied significantly, within and between groups, but the overall lower engagement with and use of ICT suggests a digital divide. Further research, specifically investigating trends observed within these group, as well as between and with the larger population, are necessary. Second, when compared to Year 10 students in 2013, Year 11 students in 2012 reported significantly more engagement with school, significantly less positive beliefs about ICT outcomes and less ICT usage and utilisation of their DER-NSW laptops. However, they reported similar levels of ICT engagement. This finding suggests Year 11 students liked to use ICT as much as Year 10 students, but they did not feel it supported learning. This is consistent with reporting of significantly less ICT use in school by Year 11 students. Overall, findings suggest that in Year 11, Stage 6, students are using the laptops and ICT significantly less. This finding is confirmed with results from the 2012 case studies that students used their laptops less in Year 11:


I realised that it just became easier to study, getting into the older grades, it was easier to study from pen to paper than on the laptops and pen to paper is usually more reliable than the technology I found. We’ve had a few problems with the laptops. (School 3)

Year 10 students in 2013 reported significantly higher frequencies of ICT usage that 2011 and Year 11 students in 2012. Confidence performing ICT tasks significantly increased from 2011 to 2013, as well as positive beliefs about learning outcomes when using ICTs. Students continued to report the most confidence in performing Productivity tasks.

Students increasing use and positive beliefs about ICT, at least in Year 10, can in part be explained by consistent increases in teachers’ use of ICTs and increasing positive beliefs about ICT in learning and teaching. Over the four years, teachers have reported increasing laptop use, positive beliefs about ICT-related tasks in teaching. However, teachers’ comments in the 2012 case studies (see Howard, S. K. & Gigliotti, A. (2013). DER-NSW evaluation: Report on the implications of the 2012 data collection. Sydney: New South Wales Department of Education and Communities.) suggests that these increases are happening in Years 9 and 10 (Stage 5), not at Stage 6:

I know that my Year 11s going into Year 12s next term, I’m going to insist that they have an exercise book and that extended writing tasks, drafts, begin in an exercise book… to develop their handwriting skills [for the exam]. Some of them actually said their spelling… they’ve lost their ability to spell accurately.

There’s a lot of time, a lot of preparation that is needed to [the laptops]… [Students] just have that attitude that the computer is their possession to play games on…We’ve got a program and we’ve got to get through it and we’ve got a timeline on topics that have got to be covered so the exams are all consistent from all classes at that time… Year 11s exam coming up in Week 10, everyone’s expected to be up to a certain level. If we were [using the laptops] we’d still be in chapter one.

The curriculum constraints and limited use of laptops in Year 11 is reflected in students reporting less use of the laptops in school. However, students in Stage 5 have reported an increased of laptop use in Year 10, which suggests teachers increased use of the laptops is happening at this level. This leads to the final finding of the evaluation, that increase in ICT use has also resulted increasing frequency of student-centred tasks. This finding is confirmed through teachers comments in the case studies; teachers believed the laptops supported different ways for students to engage in learning:

What I like about [using the laptops] is they see each other’s work more with the laptops. After they [make the video], they upload it onto Edmodo and we watch it on


the interactive whiteboard together – all of them – and then they go onto Edmodo and they comment. We do “Wows and Wonders” – they go “I think you took really good photos. They were really clear, but I wonder if your writing – your text explanation – could have been a bit clearer. I was a bit confused with this particular sentence” – or whatever it is… I think with the laptops it has exposed kids to other kids’ work a lot more because otherwise they were just written like an experiment report and that was just me looking at it and no-one else and it would be just me.

Teachers have also reported becoming increasingly confident using the laptops in teaching and learning, as well as increasing agreement and positive beliefs about their teaching and workplace. Most importantly, teachers have reported feeling increasingly engaged in teaching since the introduction of the DER-NSW laptop program. The effect of teachers’ use of ICT students beliefs and frequency of use is strongly suggested through differences in Year 10 and 11 student use of the laptops, and teachers’ beliefs about the impact of the HSC on their use of ICT. This finding has significant implications for how leadership addresses design of HSC curriculum to appropriately integrate technologies and ensure continued development of students’ ICT engagement and developing skills. Finally, since the inception of the DER-NSW program, teachers and students have reported increasing engagement and confidence using ICTs, which was one of the main aims of the initiative. As teachers’ gains have been steady over the past four reporting years, there is every indication this progress will continue as long as the resources and leadership support are consistent.


6 References

AIHW. (2004). Rural, regional and remote health: a guide to remoteness classifications. Retrieved March 21, 2011, from http://aihw.gov.au/publication-detail/?id=6442467589

Australian Bureau of Statistics. (2013). COMMENTARY ON ABORIGINAL AND/OR TORRES STRAIT ISLANDER STUDENTS. Retrieved July 2013, from http://www.abs.gov.au/AUSSTATS/[email protected]/mf/4221.0/ http://www.abs.gov.au/AUSSTATS/[email protected]/Latestproducts/4221.0Main%20Features402012

Learning Cultures Consulting Inc. (2006). One-to-one mobile computing: Literature review. Edmonton, Alberta, Canada: Alberta Education.

Mueller, J., Wood, E., Willoughby, T., Ross, C., & Specht, J. (2008). Identifying discriminating variables between teachers who fully integrate computers and teachers with limited integration. [doi: DOI: 10.1016/j.compedu.2008.02.003]. Computers & Education, 51(4), 1523-1537.

Neuman, L. W. (2006). Social research methods: Qualitative and quantitative methods (6th ed.): Pearson.

Schoen, L. T., & Teddlie, C. (2008). A new model of school culture: A response to a call for conceptual clarity. School Effectiveness and School Improvement, 19(2).