Abstracts, introductions and discussions: How far do they differ in style?

Jointly published by Akadémiai Kiadó, Budapest Scientometrics,and Kluwer Academic Publishers, Dordrecht Vol. 57, No. 3 (2003) 389–398

Received March 28, 2003Address for correspondence:JAMES HARTLEYDepartment of Psychology, Keele UniversityStaffordshire, ST5 5BG, UKE-mail: [email protected]

0138–9130/2003/US $ 20.00Copyright © 2003 Akadémiai Kiadó, BudapestAll rights reserved

Abstracts, introductions and discussions:How far do they differ in style?

JAMES HARTLEY1, JAMES W. PENNEBAKER2, CLAIRE FOX1

1University of Keele (UK)2University of Texas, Austin, TX (USA)

Two computer-based style programs were used to analyse the Abstracts, Introductions andDiscussions of 80 educational psychology journal articles. Measures were made of the overallreadability of the texts as well as of sentence lengths, difficult and unique words, articles,prepositions and pronouns. The results showed that the Abstracts scored worst on most of thesemeasures of readability, the Introductions came next, and the Discussions did best of all. However,although the mean scores between the sections differed, the authors wrote in stylistically consistentways across the sections. Thus readability was variable across the sections but consistent withinthe authors.

Introduction

Many papers discuss the different components or sections of research articles.Useful introductory articles that aim to help writers of academic papers can be found onthe topics of Abstracts,1,2 Introductions,3,4 Methodology5 and Discussions.6,7 Otherbook chapters consider several of these components in combination.8–12

Whilst the book chapters tend to be more discursive, the research papers on thesetopics tend to be more quantitative. Thus Hartley et al.13 measured the readability of theIntroductions and the Discussion sections of famous papers in psychology andconcluded that the Introductions were harder to read than were the Discussions. Robertset al.14 compared the readability of the Abstracts and the manuscripts as a whole inarticles from Annals of Internal Medicine. They found no real differences between thereadability of the Abstracts and the full text but that the Abstracts contained shortersentences (average 17 words) than the main texts (average 21 words). Otherpsychologists have measured the readability of Abstracts in different educationalpsychology journals15–19 and Leydesdorff has examined how particular words occurtogether (co-words) in different sections of articles from bio-chemical journals.20, 21

J. HARTLEY et al.: Abstracts, introductions and discussions

390 Scientometrics 57 (2003)

Most of the more quantitative papers focus on one or two features of academicpapers, and use a limited number of measures. Today, with the advent of style-basedcomputer programs, it is easier to make more numerous quantitative comparisons than itwas some years ago. Furthermore, it is also possible to make more fine-grained analysesof the different sections within the articles rather than consider journal articles as awhole.

In this paper we report the results that we obtained when we used the LinguisticInquiry and Word Count (LIWC) program of Pennebaker et al.22 and Microsoft’s Office97 to assess the Abstracts, Introductions and Discussions of 80 journal articles ineducational psychology. LIWC reports the percentages of words in 71 differentcategories (e.g., emotional words, cognitive words, social words, personal words,quantities, present tenses, etc.). Office 97 provides statistics on various aspects ofwritten text, including average word and sentence lengths, percentages of passives, andthe overall Flesch Reading Ease (R.E.) score.23

The Flesch R. E. score is one of many measures of readability, or text difficulty, thathas now been computerized. In its original form:

R. E. = 206.835 – 0.846w – 1.015swhere w = the average number of syllables per 100 words

s = the average number of words per sentence.Table 1 shows the suggested relationships between R. E. scores, levels of difficulty

and likely reading age (in the USA in 1948).Our aim, in this study, was to assess the usefulness of using these measures in

providing more fine-grained analyses of the sections typically found within journalarticles.

Table 1. Flesch Reading Ease scores, text difficulty, and likely reading age (in 1948)R.E. Description RequiredScore of difficulty reading agea

90-100 Very easy 10 years80-89 Easy 11 years70-79 Fairly easy 12 years60-69 Standard 13-14 years50-59 Fairly difficult 15-17 years30-49 Difficult 18-21 years (undergraduate)

0-29 Very difficult GraduateaNote these ages are now dated and, therefore, must be treated as rough guides.


Scientometrics 57 (2003) 391

Method

In order to carry out this study we took 80 journal articles, 38 written by pairs and42 by individual authors, from four recent volumes of the Journal of EducationalPsychology (1997-2001). For each article we scanned the Abstract, a portion of theIntroduction and a portion of the Discussion section into a computer database. To makethe texts of the Introductions and the Discussions comparable we deleted any sub-headings and lengthy quotations from other sources, and we ended each extract with thelast complete sentence on the page. In point of fact, taking the extracts from theDiscussion sections was not as straightforward as we had expected because somearticles had summaries rather than conclusions, some had only conclusions, and somehad only discussions. In the event the average length of the Abstracts was 130 words(ranging from 89-168), that of the extracts from the Introductions was 516 words(ranging from approximately 350-700 words), and that of the Discussions was 665words (ranging from approximately 270-1450 words).

We then analysed all three sections using the measures provided by LIWC andOffice 97. And, because in an earlier analysis24 we had not found any substantialdifferences between the writing styles of the 38 pairs and the 42 individual authorswhen using these measures, we subjected the data obtained from each section to a one-way analysis of variance. In addition, in order to assess the consistency of the authors’writing styles from section to section, we also computed within-manuscript correlations.

Results

Tables 2 and 3 show the main results. Table 2 lists the mean scores and standarddeviations on all of the measures that differed significantly from each other according tothe article sections. Generally speaking, it can be seen that the Abstracts fare worst ofall on most measures indicative of readability, that the Introductions come somewherein between, and that the Discussions score best of all.

There were also a number of other measures (not reported here) where there weresignificant differences with the LIWC. However, these differences were deemed sosmall as to be of no practical importance (e.g. less than 1.0% difference between thethree means). For example, the mean percentage score for use of the pronoun ‘we’ was0.0% for the Abstracts, 0.18% for the Introductions, and 0.33% for the Discussions[F (2,158) = 322, p < 0.000].



Table 2. Significant main effects between the styles for Abstracts, Introductions and Discussions (N=80)Measure Abstracts Introductions Discussions F (2,78)Flesch Reading m 18.1 20.5 22.7 11.4Ease score s.d. 12.2 9.7 9.1

Sentence length m 22.8 27.4 25.2 34.2(in words) s.d. 4.6 4.5 4.4

% Passives sentences m 28.8 17.5 19.1 7.8s.d. 23.3 13.9 11.3

% Long words’a m 41.3 37.5 37.5 35.1s.d. 6.2 3.8 4.3

% ‘Unique words’b m 59.1 46.8 41.9 278.9s.d. 6.9 4.5 6.3

% ‘Dictionary m 57.7 55.9 61.1 27.5wordsc s.d. 7.7 5.6 4.7

% Articles m 5.9 6.2 7.3 16.6s.d. 2.6 1.9 2.0

% Prepositions m 13.6 14.0 14.6 5.9s.d. 3.0 2.0 1.5

% Pronouns m 1.1 1.7 1.9 17.3s.d. 1.3 1.0 1.1

Note: all F values are statistically significant, p < 0.01aDefined as words with 6 or more letters.b Type/token words: defined as the number of different words divided by the total number of words.cPercentage of words captured by one of the LIWC dictionaries.

Table 3 presents the simple Pearson correlations for the various word categoriesbetween the three sections of the manuscripts. Most striking is the fact that, apart frompassives and sentence lengths, the authors are remarkably consistent in how they useword categories from the Abstract to the Introduction, from the Abstract to theDiscussion, and from the Introduction to the Discussion. In short, it appears that theways in which our authors wrote their journal articles were highly reliable even though,as shown in Table 2, they may have changed the detailing from section to section.



Table 3. Intercorrelations between manuscript sections (N=80)Abstracts Abstracts vs Intros vs

Measure vs Intros Discussions DiscussionsFlesch Reading Ease score 0.64 0.74 0.62

Sentence length (in words) 0.02a 0.20# 0.26

% Passives sentences 0.09a 0.19a 0.01a

% Long words(1) 0.55 0.70 0.50

% ‘Unique words’(2) 0.32 0.45 0.37

% ‘Dictionary words’(3) 0.42 0.56 0.56

% Articles 0.51 0.43 0.54

% Prepositions 0.29 0.39 0.37

% Pronouns 0.32 0.34 0.65Note: all correlations are statistically significant, p < 0.01 (78df) except: a = ns and # p = 0.08.(1)Defined as words with 6 or more letters.(2)Type/token words: defined as the number of different words divided by the total number of words.(3)Percentage of words captured by one of the LIWC dictionaries.

Discussion

In considering the validity of these results we need to consider at least two particularissues. First, how far are they dependent upon the measures that we have used? And,second, how far does the format of an academic article affect what writers can actuallydo?

Problems in measuring stylistic variables in academic text

In this particular paper we have compared the overall readability of sections ofarticles by using a computer-based version of the Flesch Reading Ease measure,23despite the long debates about the validity of this measure.25,26 The essential ideaunderlying the Flesch score as noted earlier – and one that is often queried – is that onecan measure the difficulty of a piece of text by combining with a constant two main



measures – the lengths of the sentences, and the lengths of the words within thesesentences. Thus, putting the matter simply, all that is being said is that a text which haslong sentences and difficult words within them will be more difficult to read than a textthat contains short sentences and simple words. This, of course, is naïve, although itdoes have an element of truth. Long words are not always harder than short ones(compare ‘the average’ with ‘the mean’). Some sentences can be short, but difficult tounderstand (for example, God is grace). What makes academic text readable – to aconsiderable extent – is what the readers already know before they start, theirmotivation, and their appreciation of the genre and structure of the text.27,28

Despite these caveats, many studies have shown that measures such as the Fleschscore can be useful for comparing the readability of different texts and, moreparticularly, the readability of different versions of the same text.15-18, 29 It is ofinterest, therefore, to compare the scores obtained here with those found in other studies(see Table 4). Here it can be seen that the results of the present study – as far as theAbstracts are concerned – are in line with the average of the results obtained so far inother studies.

Table 4. A comparison of the mean Flesch scores for abstracts obtained in the present studywith those reported previously

Number of Mean FleschStudy abstracts scoresa

The present study 80 18Dronberger and Kowitz15 48 12Hartley16 14 26Hartley17 24 21Hartley and Benjamin18 30 21Tenopir and Jacso30 315 19Tenopir and Jacso32 379 15

Weighted average 17aNote: Different investigators use different ways of assessing Flesch R. E. Scores, so these results, whilstindicative, might not be directly comparable (see Refs 31 and 32).

In the present study, however, we also found significant differences between themeans of various sub-components of readability and word-categories for the threesections of the articles. These additional findings extend the ones reported in theIntroduction, and they strengthen the conclusion that Abstracts, Introductions andDiscussions differ in style.



Nonetheless, it remains to be said here that the computer-based measures that wehave used in our research have been little more than ‘glorified word-counts’. Otherresearchers have discussed more complex measures of readability and style that dependless on the surface features of the text and that it might be more useful to explore in thispresent context.33,34 One particularly promising approach is that of ‘systemic functionallinguistics’35 but this procedure does not yet appear to have been fully computerized.[However, for advances in this respect, see Refs 36, 37.]

Problems arising from the formats used in academic articles

The results obtained in this paper suggest to us that the more difficult the task, theharder the prose that is used by writers. It is a well-known fact that Abstracts aredifficult to write. Dense and complex material has to be fitted within a tight word limitand, sometimes, authors fail to manage it.17 Indeed, authors are often instructed to writetheir Abstract last.38 In contrast, Introductions have a well-known formula that makestheir writing comparatively more simple,4 but authors still have to summarise andcondense a good deal of material into producing an effective Introduction. In the finalDiscussion section authors need only to comment on what they have found and reportedin the Results section (and to link it to the earlier research that they have alreadydescribed). This is possibly an easier task than is writing the Introduction, or at least itis on a par with it.

Although these remarks are consistent with the results reported in this paper, andwith the findings of Hartley et al.13 on the readability of psychology journal articles,they are not consistent with those of Roberts et al.14 studying a medical research article.Here there were no significant differences between the readability of the Abstracts andthe full text.

Finally in this section we should note that our results do not address the issue ofdifferences between the writers in terms of their modes of writing or ‘cognitivestyles’.39-42 Indeed, our results suggest that although the authors probably differed intheir ways of writing, they were consistent with whatever style they used. Similarresults have been found in studies of the effects of using new technology on academicwriting. Here the end products of individual authors are very similar despite massivedifferences in the ways they have been produced.43,44



Conclusions

Computer-based technology has allowed us to extend the findings of previousresearch by showing that there are measurable differences between the stylistic featuresof different sections of journal articles. Thus it does not seem wise to talk anymoreabout the readability (or any other stylistic measures) of journal articles taken as awhole. Different sections of journal articles differ from each other. Although theseresults are clearcut, the overall picture nonetheless indicates that readability is not asimple issue, for the authors of our eighty papers appear to have taken these sectionaldifferences into account whilst writing. Thus whilst they have written in an identifiablestyle throughout the articles (as shown in Table 3) they have adjusted this style to meetthe requirements of the different sections.

Of course the findings reported in this paper (like those of Roberts et al.14) arerestricted to the data source – 80 papers from one particular journal published at oneparticular point in time. We need to note at this point that there are disciplinarydifferences in journal articles and in the ways that they are written.24 Chandler,39 forexample, noted with 107 academic writers that 73% of scientists reported that theyalways or often co-authored papers compared with only 10% of arts specialists, andHartley45 noted that single authors of psychology papers made significantly moreacknowledgements to colleagues for help with the writing of their papers than did pairsor larger groups. Thus further research is required on the single and multiple authoringof the different sections of journal articles in a variety of disciplines in order to qualifythe findings reported here.

Note: readers might be interested to know that the Flesch scores for the variouscomponents of this article are as follows: Abstract 18.4; Introduction 28.7; Discussion46.4. Although three authors were involved, the text was written and revised almost inits entirety by the first author. [See Refs 24, 40 and 46 for further discussions of howwriters collaborate.]

*

We are grateful to two anonymous referees for helpful suggestions regarding this paper and to AndrewKnipe and Gerry Markopoulos for technical assistance.



References

1. J. HARTLEY, Clarifying the abstracts of systematic reviews. Bulletin of the Medical Library Association,88 (4) (2000) 332–337.

2. F. W. LANCASTER, Indexing and Abstracting in Theory and Practice. The Library Association, London,1991.

3. U. K. AHMAD, Research article introductions in Malay: Rhetoric in an emerging research community. In:A. DUSAK (Ed.) Cognitive Styles in Academic Discourse, Mouton de Gruyter, Berlin, 1997, pp. 273–303.

4. J. M. SWALES, H. NAJJAR, The writing of research article introductions. Written Communication,4 (1987) 175–192.

5. S. E. MAXWELL, D. A. COLE, Tips for writing (and reading) methodological articles. PsychologicalBulletin, 118 (2) (1995) 193–198.

6. B. L. DUBOIS, Scientific Discourse: An Analysis of Biomedical Journal Articles’ Discussion Sections.(Advances in Discourse Processes, Vol XLVI) Ablex, Norwood, N.J., 1995.

7. A. HOPKINS, T. DUDLEY-EVANS, A genre-based investigation of discussion sections in articles anddissertations. English for Specific Purposes, 7 (1988) 113–121.

8. C. BAZERMAN, Shaping Written Knowledge. University of Wisconsin Press, Madison, Wisconsin, 1989.9. C. BERKENKOTTER, T. HUCKIN, Genre Knowledge in Disciplinary Communication. Erlbaum, Hillsdale,

N.J., 1995.10. J. M. SWALES, Genre Analysis: English in Academic and Research Settings. Cambridge University

Press, Cambridge, 1990.11. J. M. SWALES, C. B. FEAK, Academic Writing for Graduate Students: A Course for Non-native Speakers

of English. University of Michigan Press, Michigan, 1994.12. J. M. SWALES, C. B FEAK, English in Today’s Research World: A Writing Guide. University of

Michigan Press, Michigan, 2000.13. J. HARTLEY, M. TRUEMAN, A. J. MEADOWS, Readability and prestige in scientific journals. Journal of

Information Science, 14 (1988) 69–75.14. J. C. ROBERTS, R. H. FLETCHER, S. W. FLETCHER, Effects of peer review and editing on the readability

of articles published in Annals of Internal Medicine. Journal of the American Medical Association, 272(2) (1994) 119–121.

15. G. B. DRONBERGER, G. T. KOWITZ, Abstract readability as a factor in information systems. Journal ofthe American Society for Information Science, 26 (2) (1975) 108–111.

16. J. HARTLEY, Three ways to improve the clarity of abstracts, British Journal of Educational Psychology.64 (1994) 333–343.

17. J. HARTLEY, Improving the clarity of journal abstracts in psychology: the case for structure. ScienceCommunication, 24 (4) (2003) 366–379.

18. J. HARTLEY, M. BENJAMIN, An evaluation of structured abstracts in journals published by the BritishPsychological Society. British Journal of Educational Psychology, 68 (3) (1998) 443–456.

19. R. A. KING, A comparison of the readability of abstracts with their source documents. Journal of theAmerican Society for Information Science, 27 (2) (1976) 118–121.

20. L. LEYDESDORFF, In search of epistemic networks. Social Studies of Science, 21 (1991) 76–110.21. L. LEYDESDORFF, The Challenge of Scientometrics: The Development, Measurement and Self-

Organization of Scientific Communications. DSWO Press, Leiden University, The Netherlands, 1995.22. J. W. PENNEBAKER, M. E. FRANCIS, R. J. BOOTH, Linguistic Inquiry and Word Count: LIWC. Erlbaum,

Mahwah N.J., 2001.23. R. FLESCH, A new readability yardstick. Journal of Applied Psychology, 32 (1948) 221–223.24. J. HARTLEY, J. W. PENNEBAKER, C. FOX, Using new technology to assess the academic writing styles of

male and female pairs and individuals. Journal of Technical Writing and Communication, 33 (2003) (in press).



25. A. DAVISON, G. M. GREEN, Linguistic Complexity and Text Comprehension: Readability IssuesReconsidered. Erlbaum, Hillsdale, N.J., 1988.

26. J. A. LOXTERMAN, I. L. BECK, M. MCKEOWN, The effects of thinking aloud during reading on students’comprehension of more or less coherent text. Reading Research Quarterly, 29 (1994) 353–367.

27. J. HARTLEY, From structured abstracts to structured articles: A modest proposal. Journal of TechnicalWriting and Communication, 29 (3) (1999) 121–138.

28. M. LEA, B. STREET, Student writing and staff feedback in higher education: An academic literaciesapproach. In: M. LEA, B. STEIRER (Eds), Student Writing in Higher Education. Open University Press,Buckingham, 2000, pp. 32–46.

29. J. HARTLEY, Designing Instructional Text (3rd ed.) Kogan Page, London, 1994.30. C. TENOPIR, P. JACSO, Quality of abstracts. Online, 17 (3) (1993) 44–55.31. M. SYDES, J. HARTLEY, A thorn in the Flesch: Observations on the unreliability of computer-based

readability formulae. British Journal of Educational Technology, 28 (2) (1997) 143–145.32. S. L. MAILLOUX, M. E. JOHNSON, D. G. FISHER, T. J. PETTIBONE, How reliable is computerized

assessment of readability? Computers in Nursing, 13 (5) (1995) 221–225.33. A. GLOVER, G. HIRST, Detecting stylistic inconsistencies in collaborative writing. In: M. SHARPLES, T.

VAN DER GEEST (Eds), The New Writing Environment: Writers at Work in a World of Technology,Springer-Verlag, London, 1996, pp.147–168.

34. S. HARRISON, P. BAKKER, Two new readability predictors for the professional writer: Pilot trials.Journal of Research in Reading, 21 (2) (1998) 121–138.

35. M. A. K. HALLIDAY, Introduction to Functional Grammar (2nd edit.). Arnold, London, 1994.36. D. BIBER, Variation Across Speech and Writing. Cambridge University Press, Cambridge, 1988.37. E. VIDAL-ABARCA, H. REYES, R. GILABERT, J. CALPE, E. SORIA, A. C. GRAESSER, ETAT: Expository

text analysis tool. Behavior Research Methods, Instruments, and Computers, 34 (1) (2002) 93–107.38. R. A. DAY, How to Write and Publish a Scientific Paper (4th ed.). Cambridge University Press,

Cambridge, 1995.39. D. CHANDLER, The Act of Writing. University of Wales, Aberystwyth, 1995.40. M. SHARPLES, How We Write: Writing as Creative Design. Routledge, London, 1999.41. J. HARTLEY, A. BRANTHWAITE, The psychologist as wordsmith: A questionnaire study of the writing

strategies of productive British Psychologists. Higher Education, 18 (1989) 423–452.42. J. W. PENNEBAKER, L. A. KING, Linguistic styles: Language use as an individual difference. Journal of

Personality and Social Psychology, 77 (6) (1999) 1296–1312.43. J. HARTLEY, M. J. A. HOWE, W. J. MCKEACHIE, Writing through time: Longitudinal studies of the

effects of new technology on writing. British Journal of Educational Technology, 32 (2) (2001) 141–151.44. J. HARTLEY, E. SOTTO, J. W. PENNEBAKER, Speaking versus typing: A case-study on the effects of using

voice-recognition software on academic correspondence. British Journal of Educational Technology, 34(1) (2003) 5–16.

45. J. HARTLEY, Single authors are not alone: Colleagues sometimes help. Journal of Scholarly Publishing,34 (2) (2003) 108–113.

46. B. W. SPECK, T. R. JOHNSON, C. P. DICE, L. B. HEATON, Collaborative Writing: An AnnotatedBibliography. Greenwood Press, Westport Connecticut, 1999.

Abstracts, introductions and discussions: How far do they differ in style?

Documents

Transcript of Abstracts, introductions and discussions: How far do they differ in style?