JOURNAL OF RESEARCH IN SCIENCE TEACHING VOL. 24, NO. 3, PP. 267-278 (1987)

GENDER DIFFERENCES IN NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS SCIENCE ITEMS: WHAT DOES

“I DON’T KNOW’’ REALLY MEAN?

MARCIA C. LINN, TINA DE BENEDICTIS, KEVIN DELUCCHI, ABIGAIL HARRIS, and ELIZABETH STAGE

Lawrence Hall of Science, University of California, Berkeley, California 94720

Abstract

The National Assessment of Educational Progress Science Assessment has consist- ently revealed small gender differences on science content items but not on science inquiry items. This assessment differs from others in that respondents can choose “I don’t know” rather than guessing. This paper examines explanations for the gender differences including (a) differential prior instruction, (b) differential response to uncer- tainty and use of the “I don’t know” response, (c) differential response to figurally presented items, and (d) different attitudes towards science. Of these possible explana- tions, the first two received support. Females are more likely to use the “I don’t know” response, especially for items with physical science content or masculine themes such as football. To ameliorate this situation we need more effective science instruction and more gender-neutral assessment items.

Introduction

The National Assessment for Educational Progress (N AEP) science as- sessment has consistently revealed gender differences for 13 through 17 year- olds (Heuftle, Rakow, & Welch, 1983; NAEP, 1978a,b, 1979a,b). The mecha- nisms governing these differences have not been established. Therefore, recommendations for dealing with them are uncertain.

NAEP assessments differ from many science knowledge tests in that they allow respondents to choose “I don’t know” rather than guessing or leaving items blank. Patterns of choice for this response could shed light on the ob- served gender differences. A straightforward argument says that students who have received less formal instruction in science choose the “I don’t know” response for difficult items while their more experienced peers choose the correct answers. A more complex argument says that learners have different threshholds for uncertainty and that frequent users of “I don’t know” express uncertainty even when they are almost sure while others express uncertainty

8 1987 by the National Association for Research in Science Teaching Published by John Wiley & Sons, Inc. CCC 0022-4308l87/030267-12$04.00

268 LINN ET AL.

only when they have no idea at all. Related to the uncertainty argument, some learners may tolerate great levels of risk and guess when the likelihood of a correct response is 50%. Others may choose a response only if the likelihood is 90%. Another argument is that certain aspects of science content, such as the need to reason about figurally presented materials (graphs, rotating objects) might lead students lacking skill in spatial visualization to respond “I don’t know” to such items. This paper assesses the viability of each of these argu- ments.

Previous Findings

NAEP science assessments typically include two kinds of items. Content items assess information about science, including questions such as “What is evaporation?”. The NAEP confenf items differentiate among Biology, Physical Science, and Earth Science. Znquiry items ask students of design experiments and draw conclusions, and to apply principles, such as conservation of liquid. In addition, NAEP has included science, technology, and society items in recent assessments. These items test understanding of the relationships be- tween scientific advance and persistent social problems. This category also includes items assessing health and safety-related information and items requir- ing reasoning about social responsibility.

The NAEP 1976-77 and 1981-82 assessments revealed stable patterns of gender differences (NAEP, 1978a,b, 1979a,b). Although there was a slight de- cline in overall scores during this period, the decline was nonsignificant. This suggests that the well-publicized erosion of science knowledge characterizing NAEP results in the 1970s may be coming to an end.

Overall, on science information, males scored about five percentage points higher than females at both age 13 and age 17. The largest differences were found for physical science items, the smallest for biology, with earth science in between. This finding parallels course-taking in that males and females are equally likely to take biology but more males than females take physics. Inter- estingly, the physical science item used by Hueftle, et al. (1983), to illustrate this difference involved knowledge of where to place weights to stabilize a balance beam, a task involving mathematics problem-solving skill as well as science knowledge.

The science, technology, and society cluster included science information items having to do with health and environmental safety. Interestingly, females outperformed males in the health content area while males outperformed fe- males in the environmental content areas. These results parallel the small gen- der differences in biology and larger differences in physical science for content items.

Overall, on science inquiry there were no significant gender differences at age 13 or 17. When items involve analytic processes and multistep reasoning but do not depend on specific science content, no gender differences appear to arise.

Thus, NAEP assessments reveal gender differences on science content. These results are consistent with gender differences in course experience for 17-year-olds. Furthermore, no gender differences in inquiry items emerge, pre-

GENDER DIFFERENCES 269

sumably because these items really do not require science content knowledge. Mechanisms to explain gender differences on science content items were thus the focus of the present investigation.

Methods

The data used in this study came from the National Assessment of Educa- tional Progress public-use data tape (Version 1). Three NAEP Science Assess- ment booklets on the tape from the 1976-77 assessment (numbers 4,8, and 11) for seventeen-year-olds were analyzed. The National Assessment employs a deeply stratified, nationwide, three-stage probability sample design with over- sampling in extreme rural and in low income areas.

We constructed a “test” out of the cognitive items in each booklet. A total score on the test consisting of the number of correct responses was calculated for each respondent. Each person received only one of the science booklets. A total “I don’t know” score was constructed by counting the number of “I don’t know” responses for each student. To assess,whether there were gender differ- ences on items requiring spatial visualization, the items in booklet 4 were categorized by presence or absence of spatial visualization content. The au- thors judged each item as either requiring spatial visualization or not. Interrater agreement was very high. Disagreements arose for less than 10% of the items.

Level of difficulty was determined by the proportion of subjects passing an item. Three levels of difficulty were delineated, with separation points based on the distribution of the item p-values as illustrated in Table I. The easiest items covered science topics typically taught in middle-school science courses. These could be viewed as achievement test items since all students probably ‘received instruction on these topics.

To assess science content, the items were categorized as to whether they mentioned key physical science words, such as energy, velocity, temperature, or topics more popular with males, such as football, etc. (see Table I for a full list). This was deemed more appropriate then using the NAEP categories be- cause it a) combined physical and earth science content where the items clearly

TABLE I Key Words for Physical Science Content

Analysis

acceleration atomic model atomic structure cars chemical compound electron energy football heat matter molecule

organic particle power pressure radiation speed temperature velocity wave weight x-rays

270 LINN ET AL.

relied on physical science principles, b) eliminated items primarily measuring mathematics principles, and c) emphasized the features of the items respon- dents would pay attention to, the actual wording of the question or the subject matter relevant to the problem.

To assess how students felt about science, possibly a factor in their accep- tance of uncertainty when choosing responses, we used the science attitude scales administered as part of booklet 4.

Results

Reliability of Tests

The distribution of total score and the split-half correlations for “I don’t know” responses and total score indicated that the “test” constructed for each booklet was reliable. The total scores varied widely across subjects, with no ceiling or floor effects. Odd-even split-half correlations boosted by the Spear- man-Brown Prophecy Formula back to the full test length were used to esti- mate test reliability. Split-half correlations averaged 0.83 for total score and 0.89 for “I don’t know” responses. In addition, the large sample size of sub- jects in the National Assessment made most results across subjects statistically significant.

Gender Differences in Total Score

As predicted, a male-female difference in total score was found. Total scores, broken down by sex, are shown in Table 11. As can be seen, males had

TABLE I1 Mean and Standard Deviations on Science Booklet Scores*

I Don’t Know Achievement n Total Score Score Score * *

Mean SD Mean SD Mean SD

Booklet 4 55 items (22 in Achievement Score) Males 1564 36.5 7.8 4.2 5.0 19.2 3.0 Females 1571 34.4 7.2 5.7 5.7 18.9 2.9 Booklet 8 59 items (19 in Achievement Score) Males 1535 29.5 7.2 4.7 5.8 15.9 2.8 Females 1534 27.0 6.4 6.1 6.2 15.6 2.7 Booklet I1 54 items (20 in Achievement Score) Males 1546 32.7 8.0 3.7 4.6 15.4 2.9 Females 1595 31.2 7.5 4.8 4.9 15.6 2.9

* All means computed with respect to possible number of items. ** Composed of those items solved correctly by 72% or more of respondents.

GENDER DIFFERENCES 27 1

higher total scores than did females on all three booklets. The average effect size was 0.27 of a standard deviation.

Furthermore, there were similar gender differences in choice of the “I don’t know” response, with females more likely to use the response (see Table I). The average effect size was 0.24 of a standard deviation, the same magni- tude as the differences in correct responses.

Achievement Score

The items for each booklet that tested material that all respondents should reasonably have covered in general science courses were combined to form an achievement score. These items were answered correctly by 70% or more of the respondents. There were no gender differences on these scores (Table 11). These were used to control for science achievement level in subsequent regres- sion analyses.

Role of Spatial Visualization

As shown in Table 111, performance on spatial visualization items did not explain gender differences in total score for booklet 4. The gender differences observed in the means formed no clear pattern relative to spatial visualization content. Also, the differences on spatial items were confounded with difficulty level; that is, the spatial items were substantially more difficult than the non- spatial items. There were too few difficult, nonspatial items and easy, spatial items to be able to discern a clear role of each dimension separately.

A considerable body of research suggests that there are no differences on spatial visualization if this dimension is taken to mean analytic reasoning about figurally presented information (e.g., Linn & Petersen, 1985; Linn & Petersen, 1986). For example, a meta-analysis of 81 effect sizes from tests of spatial visualization revealed no gender differences on this dimension (Linn & Peter- sen, 1985). The current results coincide with this trend.

Attitude towards Science

NAEP measured attitudes towards science in four categories: (a) liking of science, (b) attitude towards science teachers, (c) usefulness of science, and (d) participation in informal science activities. These science attitudes neither ac- counted for any substantial variation in total score in booklet 4 nor differenti- ated between the sexes. To show this effect we constructed regression equa- tions to predict total scores for males and females. We used the achievement score to control for prior achievement and then examined the contribution of attitude toward science. Thus, attitude was evaluated after effects for achieve- ment were accounted for. As can be seen (Table IV), attitude contributed a very small amount of variance and was equally related to males’ and females’ scores. Comparable attitudinal items were not included in the other booklets. Thus, differing attitudes toward science (as measured by items in booklet 4) were not responsible for the male-female difference in total scores.

.

TA

BL

E I11

Mea

ns a

nd S

tand

ard

Dev

iatio

ns fo

r Ite

ms

With

and

With

out S

patia

l Vis

ualiz

atio

n C

onte

nt,

Scie

nce

Boo

klet

4,

17-Y

ear-

Old

s

Tota

l Sco

re (p

erce

nt)

I D

on't

Kno

w S

core

(per

cent

)

Mal

e Fe

mal

e To

tal

Mal

e Fe

mal

e N

o. o

f Ite

ms

Mea

n%

SD

Mea

n%

SD

Mea

n%

SD

Mea

n%

SD

Mea

n%

SD

Item

s W

ith S

patia

l V

isua

lizat

ion

Con

tent

Ea

sy'

2 94

.4

4.53

89

.8

3.75

92

.1

4.13

2.

0 1.

06

4.2

.78

Mod

erat

e2

7 75

.8

10.1

7 67

.1

7.03

71

.5

8.26

7.

0 7.

85

11.6

8.

01

Diff

icul

t3

7 36

.1

8.42

29

.2

9.32

32

.7

8.01

12

.9

9.17

18

.6

10.6

6 It

ems

With

out S

patia

l V

isua

lizat

ion

Con

tent

Ea

sy'

11

92.5

2.

91

93.2

2.

91

92.8

2.

85

2.4

1.24

2.

2 1.

38

Mod

erat

e2

16

72.3

9.

59

69.8

11

.35

71.1

10

.2

6.0

3.70

7.

6 4.

36

Diff

icul

t3

8 37

.4

9.11

33

.5

6.08

35

.5

6.81

11

.5

5.44

17

.1

8.57

Eas

y ite

ms

wer

e th

ose

solv

ed c

orre

ctly

by

85%

or

mor

e of

res

pond

ents

.

Diff

icul

t ite

ms

wer

e th

ose

solv

ed c

orre

ctly

by

50%

or

few

er o

f re

spon

dent

s.

* Mod

erat

e ite

ms

wer

e th

ose

solv

ed c

orre

ctly

by

betw

een

50 a

nd 8

5% o

f res

pond

ents

.

GENDER DIFFERENCES 273

TABLE IV Summary of Regressions of Selected Variables on Total Score for

Science Booklet 4, 17-Year-Olds

Variable Entered R R2 AR2 P Order

Entered Males 1st Achievement Score‘ .83 .69 .69 .68 2nd Attitudinal ScalesZ .85 .73 .04 - 3rd I Don’t Know Score .87 .76 .04 -.23,

3

Females 1st Achievement Score’ .81 .65 .65 .64 2nd Attitudinal Scales2 .82 .68 .03 - 3rd I Don’t Know Score .87 .76 .08 - .33

I Composed of those items solved correctly by 72% or more of respondents. * Table I1 describes the Attitudinal Scales.

3

All attitudinal weights were less than 0.001.

Uncertainty

The “I don’t know” answers to the science questions were investigated to see if a gender difference in uncertainty contributed to the gender difference in total score. Figure 1 shows the distribution of “I don’t know” responses for booklet 4, which is representative of the distributions for the other booklets. Of those who neuer used the “I don’t know” response, there were more males than females. Also, females answered “I don’t know” more frequently than did males on all three booklets (Table I).

Booklet 4 was examined to see if total score could be predicted from “I don’t know” response, after the effects of achievement and attitude were sta- tistically removed using multiple regression. Replication of the regressions was attempted for booklets 8 and 11. Affective items were not included in the replication because they were not available for booklets 8 and 11 and had not contributed to the explained variance in booklet 4.

Results were mixed. In booklet 4, “I don’t know” responses accounted for twice as much variation in total score for females than for males (8 vs. 4%; Table IV). This small difference in the amount of variation in total score ac- counted for by “I don’t know” responding did not hold up in replication in booklets 8 and 11 (Tables V and VI). Although females are more uncertain as indicated by their more frequent use of “I don’t know,” this does not consist- ently contribute to their differential performance.

Physical Science Content

To determine whether gender differences in use of “I don’t know” repre- sent lack of science knowledge, we examined the relationship between physical science content and use of the response. Previous analyses revealed that fe- males know less physical science content than males, that females take fewer

214 LINN ET AL.

m 300 c 0 m

3 u)

0 200

- n

c

L m n f 2

100

0 1 2 1 4 5 6 7 8 9 10 20 30 I 1

Number of ‘I Don’t Know‘ Responses Fig. 1 . Frequency of “I Don’t Know” Responses for Science Booklet 4, 17-Year-Olds

physical science courses than males, and that females have fewer informal physical science experiences. If females lack this knowledge, then they should use “I don’t know” for physical science items.

Questions were analyzed according to whether they mentioned key physi- cal science content. Items with more than a five percent gender difference in “I don’t know” responding were identified ,and compared in content to items without such a difference. This content analysis revealed that the large male- female differences in “I don’t know” responding occurred on physical science items.

Table VII shows the correlation between item type (physical science or not) and the male-female difference in “I don’t know” responding. For con-

TABLE V Summary of Regressions of Selected Variables on Total Score for

Science Booklet 11 , 17-Year-Olds

Variable Entered R R2 AR2 P Order

Entered Males 1st Achievement Score’ .82 .68 .68 .73 2nd I Don’t Know Score .84 .70 .02 -.18

1st Achievement Score’ .84 .70 .70 .74 2nd I Don’t Know Score .85 .73 .03 -.19

Females

I Composed of those items solved correctly by 72% or more of respondents.

GENDER DIFFERENCES 275

TABLE V1 Summary of Regressions of Selected Variables on Total Score for

Science Booklet 8, 17-Year-Olds

Variable Entered R R2 AR2 P ~ ~~

Order Entered Mules

1st Achievement Score’ .78 .61 .61 .65 2nd I Don’t Know Score .81 .65 .05 - .26

Females 1 st Achievement Score’ .77 .60 .60 .65 2nd I Don’t Know Score .80 .65 .05 - .26

I Composed of those items solved correctly by 72% or more of respondents.

trast, the correlation of item type with the male-female difference in total score is also shown. For each item, the difference score was constructed by finding the difference between the male and female “I don’t know” scores or total scores. That difference score was then related to the presence or absence of physical science content for that item. The correlation coefficient describes the relationship between the male-female difference and the item type. The higher correlation coefficients indicate that the “I don’t know” difference score is related to physical science content. The correlations between total score and item type are not high, an indication that physical science content of items is contributing more to “I don’t know” responding by females than to correct answers by males. Overall, the physical science content analysis suggest that females use the “I don’t know” response to indicate lack of knowledge about physical science topics.

Discussion

Females are less successful than males on NAEP science content items, but the sexes perform similarly on inquiry items. This study offers some insight into the meaning of the greater use of “I don’t know” by females than by males when responding to NAEP science assessment items. It is also consistent with other studies showing no influence of spatial visualization on gender differ-

TABLE V1 Correlations’ Between Physical Science and Non-Physical-Science Item

Types2 Difference Scores3 for Science Assessment, 17-Year-Olds ~

Difference Scores Booklet 4 Booklet 8 Booklet 11

“I Don’t Know” (female minus male) 5 5 .20 .62 Total Score (male minus female) .36 .03 .24

I Refer to the text for a full description of the construction of this table.

3 Difference Scores = difference between male and female scores. Item Types: 1 = nonphysical science, 2 = physical science.

276 LINN ET AL.

ences and suggests that science attitudes are not central in explaining gender differences for science content items.

The most straightforward explanation for use of the “I don’t know” re- sponse is that the respondent does not, in fact, know the answer to the ques- tion. Females use the response more on difficult items and on physical science content items for which they may have less previous experience. Thus, it seems reasonable to conclude that the primary explanation for differences in use of the response is that there are differences in science content knowledge.

Another explanation, however, centers on differences in honesty. Re- search in other areas has shown that females admit negative feelings such as anxiety more readily than males do (Maccoby & Jacklin, 1974). It is possible that females are more willing to admit their lack of knowledge of science than males are.

Why would females lack knowledge of science and why would they be more willing to admit it? Females take fewer elective courses in science than do males (CBEDS, 1981). Once females are in a science class, the stereotyping of science as masculine (Vockell & Lobonc, 198 l), the psychologically trouble- some experience of being different from everyone else (Kanter, 1977; Frank & Katcher, 1975; Vockell & Lobonc, 198l), and the differential treatment and expectations of teachers for males and females (Brophy & Good, 1970) may all contribute to less effective learning. The low participation of women as scien- tists in the workforce (Vetter, 1981) and the failure of adult women to keep informed of current events related to science (Miller, Suchner, & Voelker, 1980) create a societal context in which young women are allowed to avoid participation in science, particularly physical science and engineering.

With such a cultural backdrop, it is not surprising that young women are more likely to respond “I don’t know,” especially to physical science items, but additional psychological factors may be operating. An unwillingness to take risks may also lead females to avoid giving a definite answer. Not giving an answer has the advantage that it avoids being either “right” or “wrong.” Being “right,” especially on a physical science item, is consistent with being “smart,” a status that females may avoid because of conflict with social rela- tionship goals (Stein & Bailey, 1973; Lockheed, 1975). Being “wrong” is, however, not a viable alternative since it conflicts with the socialization of females to be “good,” which includes cooperating with teachers and, by exten- sion, testers. Thus, saying “I don’t know” provides an alternative to two unpleasant choices.

For males, the “I don’t know” response has less value. First, males are expected to know about science, particularly physical science (Vockell & Lo- bonc, 1981). Second, they are expected to take risks. It is better to have guessed and been wrong than to admit uncertainty by saying “I don’t know.” Further, males are sometimes found to be less realistic than females about estimating their likelihood of success on an academic task and may overrate their confidence in their answers (Maccoby & Jacklin, 1974; Linn, 1985a,b). Thus, they may not realize that their confidence is unjustified and may in fact get the correct answer more often than those who take fewer chances. For these reasons, one would expect less 4cquent use of the “I don’t know” option by males than females.

GENDER DIFFERENCES 277

Constraints

Several constraints of this study resulted from some of the unusual features of the National Assessment. NAEP is quick to admit that the Assessment is not a test in the usual sense of the word: (a) It is not balanced with respect to content. In this case, the prevalence of physical science items biases the as- sessment against those who have not taken elective science courses, and (b) It allows the “I don’t know” response. We believe that the “I don’t know” format introduces risk-taking as a factor in performance by some students who would perhaps guess if the only alternative was to leave the item blank. Further analysis of NAEP data should take these features into account. Females, pro- pensity to guess, however, would be better investigated by observations and interviews in a variety of settings so that avoidance of risk-taking may be more clearly defined and related to academic performance.

Recommendations

Males and females differ in their knowledge of science content but not in science inquiry skills. This is reflected in more females than males responding “I don’t know” to content item, and especially physical science content items or items with masculine themes. To change the pattern, the most logical step would be to increase effective instruction for females in science courses.

Females may, however, avoid science courses for good reasons. The courses may be dull and largely irrelevant to their concerns. Precollege science courses are generally far better at educating scientists then citizens. Precollege students often have to choose between science and humanities courses. Fe- males may be taking courses in literature or social science that are more rele- vant to their concerns. Many precollege courses familiarize students with sci- ence vocabulary but not with scientific thinking. As a result, students may be proficient in NAEP science content items but do no better on science inquiry items than those who do not take the courses. We recommend, then, that a) females be encouraged to take more precollege science courses, and also that b) these courses be aimed toward the generalist audience rather than the scientist audience and that they focus on improving the scientific thinking of both males and females. Furthermore, we suggest that national assessments omit items with masculine or specialized content such as football and include items with content familiar to all respondents.

References

Brophy, J . E., & Good, T. L. (1970). Teachers’ communication of differen- tial expectations for children’s classroom performance: Some behavioral data. Journal of Educational Psychology, 61, 365-374.

CBEDS (California Basic Educational Data System). (1981). Student en- rollment by subject in California public schools for 1980-1981. Sacramento, CA: California State Department of Education.

Frank, H. H., & Katcher, A. H. (1975). Perceptions of freshwomen dental and medical students by their freshmen peers. In M. E. Lockheed (Ed.), Re-

278 LINN ET AL.

search on women’s acquisition of professional and leadership roles. Princeton, NJ: Educational Testing Service Research Bulletin.

Hueftle, S. J., Rakow, W. J., & Welch, W. W. (1983). Images of science. Minneapolis, MN: University of Minnesota.

Kanter, R. M. (1977). Men and women of the corporation. New York: Basic Books.

Linn, M. C. (1985a). Fostering equitable consequences from computer learning environments. Sex Roles, 13(3/4), 229-240.

Linn, M. C. (1985b). Gender equity in computer learning environments. Computers and the Social Sciences, 1(1), 19-27.

Linn, M. C., & Petersen, A. C. (1985). Emergence and characterization of sex differences in spatial ability: A meta-analysis. Child Development, 56,

Linn, M. C., & Petersen, A. C. (1986). Gender differences in spatial ability: Implications for mathematics and science performance. In J. Hyde & M. C. Linn (Eds.), The psychology of gender: Advances through meta-analysis. Bal- timore: Johns Hopkins University Press.

Lockheed, M. E. (1975). Female motive to avoid success: A psychological barrier or a response to deviancy? Sex Roles, 1(1), 41-50.

Maccoby, E. E., & Jacklin, C. N. (1974). The psychology of sex differ- ences. Stanford, CA: Stanford University Press.

Miller, J. D., Suchner, R., & Voelker, A. M. (1980). Citizenship in an age of science. New York: Pergamon Press.

NAEP (National Assessment of Educational Progress). (1978a). Science achievement in the schools: A summary of results from the 1976-77 national assessment of science. (Report #08-S-01). Denver, CO: Education Commis- sion of the States.

NAEP (National Assessment of Educational Progress). (1978b). Three na- tional assessments of science: Changes in achievement, 1969-77. (Report #08- S-00). Denver, CO: Education Commission of the States.

NAEP (National Assessment of Educational Progress). (1979a). Attitudes toward science: A summary of results from the 1976-77 national assessment of science. (Report #08-S-02). Denver, CO: Education Commission of the States.

NAEP (National Assessment of Educational Progress). (1979b). Three na- tional assessments of science, 1969-77: Technical summary. (Report #08-S- 21). Denver, CO: Education Commission of the States.

Stein, A. H., & Bailey, M. M. (1973). The socialization of achievement orientation in females. Psychological Bulletin, 80, 345-366.

Vetter, B. M. (1981). Degree completion by women and minorities in sci- ences. Science, 212, 35.

Vockell, E. L., & Lobonc, S. (1981). Sex-role stereotyping in high school females in science. Journal of Research in Science Teaching, 18(3), 209-219.

1479- 1498.

Manuscript accepted October 6, 1986