A CORPUS-BASED LINGUISTIC PROFILING OF IDGH AND LOW STUDENT ENGAGEMENT CLASSROOMS IN SINGAPORE SCHOOLS

TI • TI 43 nuaqmgnong

Abstract

Taking a corpus-linguistic approach to conversational analysis, this study investigates whether there is a strong correlation between teachers' talk and students' engagement levels in the classrooms of Singaporean primary and secondary schools. To identify the linguistic features of high and low engagement classrooms, we adopt two sources of authentic materials collected in the Singapore Corpus of Research in Education (SCoRE) (Hong 2005): the coding data from 455 observed authentic classroom practices as well as the 2.3 million words of transcripts of recorded classroom interactions. Linguistic and statistical analyses of the linguistic patterns used in the sample lessons of high and low student engagement classrooms revealed that the use of a list of linguistics features in the teachers' talk varies systematically across classes of different subjects, levels and streams. Thus, linguistic profiles of the teachers' talk in high and low student engagement classrooms are identified and statistically justified These profiles can be used as an overall picture of current practice in the classes where students' engagements vary across subjects and levels, and can also be useful in fUture school intervention and teachers' professional development.

Keywords: Corpus, Classroom Discourse, Engagement, Linguistic Profiling

Introduction

Student engagement is undoubtedly a major factor of students' performance in schools and thus for a long time it has been a very hot topic in traditional education research and it is still the focus of many recent studies (Hake 1998, Brewster & Fager 2000, Willms 2003; Norris, Pignal & Lipps 2003; Rosario 2006; Harmin & Toth 2006; Laitsch 2007; Yeh 2007; to name just a few). Student engagement in learning has also been identified to be associated with reduced dropout rates and increased levels of student success (Blank 1997; Dev 1997; Kushman et a12000; Woods 1995; to list just a few). Getting students engaged during class have always been a challenge to teachers. Many factors, such as students' age, self-efficacy, and gender, contribute to students' level of engagement in learning. For example, studies have shown that student engagement declines as students get older. Students' self-efficacy towards learning can also affect their level of engagement in learning (Anderman & Midgley 1997 & 1998). Students, who attribute poor performance to a lack ofattainable skills, rather than to some innate personal deficiency, are more likely to re-engage themselves in a task and try again. Marks (2000) found that girls are more engaged in instructional activities than boys across all levels. Peer influence is another factor that contributes to student engagement in learning, especially for high school-age students. As students grow older, their motivation to engage in learning may be influenced by their peers just as much as, if not more than it is by teachers, parents, and other adults. While peer influences can be either positive or negative, it is not uncommon for older students to discourage one another from actively participating in school (MacIver & Reuman 1994).

Although teachers have no control over students' background factors as such gender, social economic status etc., research has shown that there are ways to make their classroom instructions and assigned work more engaging and more effective for students at all levels. For example, Marks (2000) found that authentic instructional work, classroom support, and parental involvement are all correlated with student engagement.

At present, most studies on student engagement has focused on students' characteristics (such as age, attitudes), situational characteristics (such as subjects, social economic status) and teacher instruction types (such as classical, authentic). Taking a corpus linguistics approach to classroom discourse analysis, this study differs from prior studies by looking at whether there is a correlation between teachers' talk and students' engagement levels in

43 Huaqing HONG is a Lead Research Associate with the Centre for Research in Pedagogy and Practice (CRPP), National Institute of Education (NIE), Nanyang Technological University (NTU), Singapore. He has been working as university lecturer, teacher trainer and education researcher both in China and Singapore since 1992. He has published in refereed journals like World Englishes, Language and Education, Language Policy, and Journal of Chinese Sociolinguistics, and presented at a dozen of international conferences in Czech, UK, France, New Zealand, China, USA, Australia, Hong Kong, Singapore, etc. His research interests include computational linguistics, corpus linguistics, discourse analysis, and cross-cultural communication and translation studies. Contact detail: Address: B1k5, B3-CRPP, 1 Nanyang Walk, Singapore 637616, Telephone: +65 6216 6269, Email: huaging.hong@nie.e@.§g

148

the classrooms. It adopts some advanced statistical analyses like factor analysis and cluster analysis to classroom dis(;owrse, and tries to profile the linguistic featwres correlated with students' engagement levels in a large corpus

real-life classroom interactions.

data for the present study is a sub-corpus of the SCoRE project (see more at http://score.crpp.nie.edu.sg). The below is the breakdown of the data used for the present study. This sub-corpus consists of a total of 121

'''<I''n''''r~' 455 lessons recorded in 40 primary and secondary schools in Singapore. The 253.3 hours recordings of lessons were manually transcribed to machine readable texts for the corpus compilation. These class

sessions were sampled on a common design and a group of experienced researchers were assigned to observe the in the classrooms to code the actual practices with a list of designated featwres in the Singapore Coding

(Luke et aI2004). Engagement levels were then generated from the coding data.

compare between subjects, levels and streams, 216 Primary Five (P5) lessons of 3 streams (EMI, EM2, and were sampled, while 239 Secondary Three (S3) lessons of 3 steams (EXP, NA, and NT) were collected.

4 main school subjects were selected for this study and the size of this corpus for this study is over 2.3 million words (excluding event markers, comments, punctuation marks and other non-utterance tags in the transcripts). It is believed that the sampling method and the corpus size are good enough for the present study.

Subiect Grade Stream Teachers Lessons Dur(hrs) Tokens

EMI 5 16 14.7 145277 P5 EM2 7 18 12.6 111106

English EM3 1 3 21.6 178724

EXP 9 30 25.5 243064

S3 NA 3 11 8.1 79695

NT 4 13 9.0 85282

EMI 4 11 6.3 57945 P5 EM2 7 29 13.6 121659

Math EM3 2 9 8.8 76934

EXP 8 26 14.2 140976 S3 NA 4 16 9.4 93853

NT 7 25 11.2 104427

EMI 6 25 9.4 83741 P5 EM2 4 30 11.3 101994

Science EM3 3 29 1.5 12114

EXP 11 46 15.6 143124 S3 NA 4 14 4.8 38850

NT 4 18 7.2 75701

EMI 3 8 5.3 32196 P5 EM2 9 26 10.4 100859

Social EM3 4 12 5.9 50141 Studies EXP 8 25 17.8 148900

S3 NA 4 15 9.2 86095

NT 0 0 0.0 0

TOTAL 2 Levels 6 Types 121 455 253.3 2312657

Table 1: Breakdown of the corpus data for the analysis

149

Student Engagement Levels

Observers scored the levels of student engagement during class on a 4-point scale. Thus each transcript has an associated score for student engagement level. Some moderation has been made when all the scores were collected, and as the distribution of score was skewed, the engagement was then categorized with statistical analysis and exporters' judgment (please refer to for more). As shown in the table below, the categorization resulted in 292 high student engagement (score=4) and 163 low student engagement (score~3) classes/transcripts. With the distribution across social variables, such as School Subjects, Grades, Teacher Gender, Teacher Age, Teacher Qualification and Teaching Experience, the linguistic profiles of high engagement classes were then compared to those low engagement classes in next section.

School Subiect Engagement Social Total

Science Mathematics English Studies

Low 32 50 49 32 163

High 59 66 113 54 292

Total 91 116 162 86 455

Engagement Grade Level Total

Primary 5 Secondary 3

Low 62 101 163

High 154 138 292

Total 216 239 455

Engagement Teacher Gender Total Male Female

Low 47 116 163

High 80 212 292

Total 127 328 455

Engagement Teacher Qualification Total

Certificate Diploma Bachelor Post-graduate

Low 15 13 128 7 163

High 23 32 228 9 292

Total 38 45 356 16 455

Engagement TeacherAgeCYear) Total <25 25-30 30-35 35-40 40-45 45-50 >=50

Low 3 66 43 17 9 3 22 163

High 7 77 86 58 27 11 26 292

Total 10 143 129 75 36 14 48 455

Engagement Teachin Experience (Year Total <3 3-5 5-10 10-15 15-20 20-25 >=25

Low 50 7 60 14 4 8 20 163

High 67 21 104 44 12 17 27 292

Total 117 28 164 58 16 25 47 455

Table 2: Statistics of high and low engagement lessons by school subject, grad e level, teacher gender, teacher age, teacher qualification and teaching experience

150

; .. n.rll.<il·'C Features Selected for the Analysis

Linguistic features were automatically tagged by making use of the online version ofWmatrix corpus analysis and comparison tool (Rayson 2003 & 2008). Although the set of linguistic features identified included both grammatical and semantic features, only the grammatical features were used for the analysis in this study as the accuracy rate of the semantic tagging is not as high as expected. A total of 132 grammatical features were identified, but only 43 features were selected in the analysis. Many features were not included because they were either extremely rare in the conversations, or have little or no variation across transcripts. Still some features were not included because they had low communality estimates, suggesting that they shared very little variance with the overall factorial structure of this analysis. The set of 43 features is listed in the table below.

Table 3: List of grammatical features used in analysis

151

A computer tool was then used to get the frequency counts for these grammatical features selected, and the data was then further processed for some advanced statistical analyses with SPSS version 15 in next section.

Factor Analysis

Counts of each linguistic feature were computed for each of the 455 transcripts of the lessons by means of a computer program. The frequency counts were fIrst normalized with respect to the total number of tokens in the transcripts. The standardized counts were then subjected to Factor Analysis with Principal Component extraction and Varimax with Kaiser Normalization (rotation converged in 10 iterations) using SPSS 15. Finally a fIve-factor solution was obtained as shown in the fIgure and table below.

6

5

4 (I) ::I ..... lIS

~3 (I) til ....

I%l 2

o

Scree Plot

9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43

Component Number

Figure 1: Scree Plot of the factor analysis result

152

Rotated Component Matrix

Grammatical Features

past tense of lexical verb (e.g. gave, worked)

was 3rd person sing. subjective personal pronoun (he, she)

did had (past tense) were preceding noun oftitIe (e.g. Mr., Prof.) singular proper noun (e.g. London, Jane, Frederick) 3rd person sing. objective personal pronoun (him, her) infinitive (e.g. to give ... It will work. .. ) infinitive marker (to) modal auxiliary (can, will, would, etc.) ing participle catenative (going in be going to) am 1st person sing. subjective personal pronoun (I) subordinating conjunction (e.g. if, because, unless, so, for) be, infinitive (To be or not... It will be .. ) that (as conjunction) do, base form (finite) base form oflexical verb (e.g. give, work) not, n't 2nd person personal pronoun (you) general adverb of (as preposition) interjection (e.g. oh, yes, um) wh-determiner (which, what) 1st person plural subjective personal pronoun (we) is plural common noun (e.g. books, girls) cardinal number, neutral for number (two, three .. ) singular cardinal number (one) singular letter of the alphabet (e.g. A,b) 3rd person plural subjective personal pronoun (they) common noun, neutral for number (e.g. sheep, cod, headquarters) singular determiner (e.g. this, that, another) 1st person sing. objective personal pronoun (me) have, base form (finite) past participle of lexical verb (e.g. given, worked) 3rd person sing. neuter personal pronoun (it) article (e.g. the, no) singular common noun (e.g. book, girl) possessive pronoun, pre-nominal (e.g. my, your, our) com arative after-determiner e .. more, less, fewer

Com onent 1 2 3 4 5

.835

.765

.676

.638 .345

.565

.506 .368

.495

.494 -.410

.473

.455

-.369

-.307 -.310

-.363

.414

-.371

.577

.563

.423 -.396 -.326

Table 4: List of five factors identified and the respective positive and negative loadings of the grammatical features

number of factors extracted was based on inspection of the scree plot, eigen-values, as well as interpretability factors. These five factors were interpreted as: (1) Narrative; (2) Simple Instructional; (3) First-person Object-oriented; (4) Explanatory; and (5) General versus Numerical. A set of factor scores was then

for each of the 455 transcripts observed. Factor scores, computed based on the regression method, were used to do a list of Logistic Regression analyses to identify the association patterns and statistical

~u .. _au\;t:: levels of these five factors against school subjects, grade levels, streams as well as teachers' social (e.g. age, gender, experience, qualification, etc.).

153

Results and Discussion

Comparing Across Subjects

The figure below shows the comparison of the profile of factor scores between high and low student engagement classes for different subjects. For English, it appeared that a Narrative and Explanatory style distinguished between high and low student engagement. For Math classes, the profiles did not differ much except that too much focus on what the teacher is going to do (First-Person Oriented) and simple instructions (Simple Instructional) was correlated with low student engagement. The same can be observed for Science classes. However, unlike Math, adopting a more Explanatory style was associated with high student engagement.

(a)

(cl

Narrative Simple First vs Object Explanatory Instructional

General vs Numerical

Figure 2: Linguistic profiles of high (black bars) and low (white bars) student engagement for (a) English, (b) Math, and (c) Science classes.

154

, that in Math classes, the linguistic profile of teachers' talk is less correlated with students' engagement. with the exception of one factor, the linguistic profiles do not differ dramatically across high and low

engagement for English and Science. Notable trends are that (1) for Math and Science classes, Simple 'fr11,~tlOlnal and First-Person Oriented styles are associated with low student engagement, but this trend is not

in English classes, and (2) in general, a less Explanatory style is associated with low levels of student laSl:eIIlenL, although the difference is less pronounced for Math classes. We speculate that the fmdings reflect

low need for simple instructions during English classes, and the low provision of elaborate explanations Math classes. The latter is not intuitive given that there is generally a need to explain mathematical

'cOllcepIS during class. This could reflect a problem with the current mathematical instructions where students are '''nJULll-~''''~ formulas rather than encouraged to think. Further research in associating authentic instruction with

• 'linguistic profiling may reveal more.

Comparing Across Levels

The figure below shows the comparison of the profiles of factor scores between high and low student engagement classes for different levels. It can be observed that less Explanatory style was associated with low student engagement. However, for other factors, P5 and S3 showed very different trends. While for the first two factors -Narrative and Simple Instructional- there were no differences between high and low student engagement for P5, the S3 profiles showed that more Narrative style was associated with high student engagement while Simple Instructional style was associated with low student engagement. While focusing on what the teacher is going to do was associated (First-Person Oriented), and less General-oriented was associated with high student engagement in P5 classes, the reverse was true for S3 classes.

...... ~ C :::J

€ 1\1 --en ~ 8 UJ ... ~ 1\1

LL

.-. ::!

I:::: :::J

€ .!!.

~ 0 U U) ... 0 ti 1\1

LL

(a)

(b)

Narrative Simple First vs Object Explanatory Instructional

Generalvs Numerical

Figure 3: Linguistic profiles of high (black bars) and low (white bars) student engagement for (a) P5, and (b) S3 classes.

155

The comparison across levels reveals more interesting fmdings. The linguistic profiles for high and low student engagement clearly differs for the two levels and in different ways. This is especially the case for S3 where the linguistic profiles for high and low student engagement are almost opposites. The findings suggest that for S3, non-narrative, non-explanatory, first-person oriented, simple instructions are associated with low engagement. This finding is intuitive as one can predict that a teacher who gives non-story-like, simple description of what she's going to do and with no explanations given is likely to bore hislher students. On the other hand, story-like elaborations of events is much more likely to engage students in class. It is notable that this applies only to high­school level students. For young students, surprisingly, such speech style is not associated with high engagement. It is the explanation of what the teacher is doing that engages young students more.

Final Remarks

Taking a corpus linguistics approach to profile the linguistic patterns of teachers' talks in terms of students' engagement levels, this study has shown that there is a strong correlation between teachers' language use in classrooms and students' engagement levels. Differing from the prior research focused only on students' variables, this study has identified a systematic variation of the correlation patterns across school subjects, grade levels, streams as well as teachers' social variables, such as age, gender, experience, qualification, etc. It is believed that the investigation of this kind is one of the first few, if not the first one, which has identified and profiled the correlation patterns between linguistic features of teachers' talks and students' engagement levels, drawn on a large corpus of authentic classroom practices and advanced statistical analyses.

A limitation of this study is that there are just too many linguistic features and social variables to select for any form of quantitative comparison. As these are correlational data, causality cannot be established. We are not certain whether the speech styles resulted in the high/low levels of student engagement are a combined effect with other factors (e.g. social variables of schools, teachers and students) or vice versa. In addition, as student engagement is rated on a relatively restricted 4-point scale, the variance across classes is admittedly limited. Operational definition of student engagement is also simplistic, as it does not capture the full range of the types of student engagement (e.g. student engagement in different types of class phases, learning tasks, and activities). However, the criticism on rating of students' engagement level in classrooms is beyond this study's scope and research focus.

Despite its limitations, profiling the teachers' talk in real classroom activities of high and low student engagement classes can reveal interesting patterns. This influence of teachers' speech on student engagement has often been anecdotally acknowledged yet often neglected in quantitative studies, probably due to the difficulty in obtaining quality data. This study hopes to illustrate the usefulness of this corpus-based approach to studying student engagement - the profiles obtained can be used as an overall picture of current practice in the classes where students' engagements vary across subjects and levels, and they can also be useful in future school intervention and teachers' professional development.

Acknowledgements This paper makes use of the data from the research projects "Compiling a Multimodal Corpus Database of Education Discourse in Singapore Schools" (CRP7105AL) and "Computational Language Analysis of Singapore Classroom Interactions: The CRPP computational program (CRP20104JD) ", funded by the Ministry of Education, Singapore, through the Centre for Research in Pedagogy and Practice, National Institute of Education, Nanyang Technological University, Singapore. Thanks goes to Liobo Guo for his comments on the earlier draft of this paper, and I'm also grateful Chan Wai Yen and Papia Sultana, for their assistance with data analyses. The views expressed in this paper are all the author's and do not necessarily represent the views of the Centre or the Institute or the Ministry. For more about the center and the projects, please visit: http://www.crpp.nie.edu.sg.

References

Anderman, E. M., & Midgley, C. 1997. "Changes in personal achievement goals and the perceived goal structures across the transition to middle schools." Contemporary Educational Psychology, 22, 269-298.

Anderman, E. M., and Midgley, C. 1998. Motivation and Middle School Students. Champaign, IL: ERIC Clearinghouse on Elementary and Early Childhood Education.

Blank, W. 1997. "Authentic instruction." In W.E. Blank & S. Harwell (Eds.), Promising practices for connecting high school to the real world (pp. 15-21). Tampa, FL: University of South Florida.

156

in •• ~ .. ",,~,>r, C., & Fager, J. 2000. Increasing student engagement and motivation: from time-on-task to homework. portland, OR: Northwest Regional Educational Laboratory. http://www.nwrel.org/requestJoctOOI [Access date: 15105/2008].

Dev, P. C. 1997. "Intrinsic motivation and academic achievement: What does their relationship imply for the classroom teacher?" Remedial and Special Education, Vo1.18 (1), 12-19.

Hake, R. 1998. "Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses." American Journal of Physics, Vol. 66(1), 64-74.

Harmin, M., and Toth, M. 2006. Inspiring Active Learning: A Complete Handbook for Today's Teachers. Alexandria, VA: ASCD.

Bong, B. 2005. "SCoRE: A multimodal corpus database of education discourse in Singapore schools", In Proceedings of the Corpus Linguistics Conference Series, Vol. 1, No.1 (ISSN 1747-9398). Birmingham, UK, July 14-17,2005.

Kushman, J. W., Sieber, C., & Heariold-Kinney, P. 2000. "This isn't the place for me: School dropout." In D. Capozzi & D.R. Gross (Eds.), Youth at risk: A prevention resource for counselors, teachers, and parents (3rd ed., pp. 471-507). Alexandria, VA: American Counseling Association.

Laitsch, D. 2007. "Interactive engagement vs. traditional instruction in physics." ResearchBrief, Vol. 5(4), 23 April, 2007.

Luke, A., Cazden, C., Lin, A., & Freebody, P. 2004. A Coding Scheme for the Analysis of Classroom Discourse in Singapore Schools. Research Report, Centre for Research in Pedagogy and Practice, National Institute of Education, Singapore.

MacIver, D. J., and Reuman, D.A. 1994. "Giving their best: Grading and recognition practices that motivate students to work hard." American Educator, Vol. 17, 24-31.

Marks, H. M. 2000. "Student Engagement in Instructional Activity: Patterns in the Elementary, Middle, and High School Years." American Educational Research Journal, Vol. 37, 153-184.

Norris, C., Pignal, J., and Lipps, G. 2003. "Measuring school engagement." Education Quarterly Review, Vol. 9(2), 25-34.

Rayson, P. 2003. Matrix: A statistical method and software tool for linguistic analysis through corpus comparison. Unpublished PhD thesis, Lancaster University.

Rayson, P. 2008. Wmatrix: a web-based corpus processing environment. Computing Department, Lancaster University. http://ucrel.lancs.ac.uklwmatrixl.

Rosario, J. R. 2006. On engaging Latino students in their education: A resource guide to research and programs. http://www.elpuenteproject.comidefiles/On%20Engaging%20Latino%20Students%20in%20their%20Education­%20%20A %20Resource%20Guide%20to%20Research%20and%20Programs.pdf [Access date: 15/05/2008]

Willms, J. D. 2003. Student engagement at school: A sense of belonging and participation. Paris: Organization for Economic Cooperation and Development.

Woods, E. G. 1995. "Reducing the dropout rate." In School Improvement Research Series (SIRS): Research you can use. Portland, OR: Northwest Regional Educational Laboratory. http://www.nwrel.org/scpdlsirs/9/cOI7.html [Access date: 15/05/2008).

Yeh, S. 2007. "Improving engagement and achievement through rapid assessment." The Leader (e-newsletter), Spring 2007. http://www.education.umn.eduiedpailicensure/leaderl2007SpringlYeh.html [Access date: 15/05/2008]

157