JOURNAL OF APPLIED BEHAVIOR ANALYSIS

ACADEMIC RESPONSES AND ATTITUDES ENGENDERED BYA PROGRAMMED COURSE IN CHILD DEVELOPMENT'

GROVER J. WHITEHURST

STATE UNIVERSITY OF NEW YORK AT STONY BROOK

Both a small course section and a large course section were taught with methods thatinvolved breaking material down into weekly units that had to be mastered by studentsas indicated by weekly tests. Attendance at lectures was voluntary and students were nottested on lecture material. In a system in which a student could receive only an A oran F for a course grade, less than 2% of the students receiving credit failed to masterall material and received Fs. Students in the large course performed better on weeklytests and received fewer Fs than their counterparts in the small course, even though theyrated their experience less favorably. Within the large course, the methods of tutorialinterview, group discussion, and written assignment were compared in terms of theireffectiveness in preparing students for weekly tests. Tutorials and written assignmentswere superior to group discussions in this regard. Students rated the effectiveness andenjoyability of these three methods in the reverse order from their actual effectivenessfor test preparation. Some consequences of lack of congruity between attitude measuresand performance measures were discussed.

Interest in the application of behavioral prin-ciples to university level teaching is burgeoning.Following Keller's (1968) lead, a variety ofcontingency management techniques have beenapplied to the teaching of experimental analysisprinciples (Ferster, 1968; Lloyd and Knutzen,1969; Malott, 1971, Johnston and Pennypacker,1971; Sheppard and MacDermot, 1970), aswell as to the teaching of more traditional con-tent (McMichael and Corey, 1969). Undoubt-edly, other instructors have applied contingencymanagement techniques to their classes withoutreporting the results of their efforts.

This report presents the results of efforts tocontrol and analyze the academic behavior ofundergraduates enrolled in a child development

'Much of the structure and content of the coursedescribed herein must be attributed to the model pro-vided by Sidney W. Bijou, who has taught a similarcourse for a number of years at the University ofIllinois. Appreciation is also expressed to MinnaGoldfarb, Gary Novak, and Ross F. Vasta who con-tributed in many ways to the progress of this re-search. Reprints may be obtained from the author,Psychology Department, State University of NewYork, Stony Brook, Long Island, New York, 11790.

course. A variety of purposes motivated the re-search. The primary technological goal was toengineer an educational environment that wouldbe maximally effective in teaching small andlarge undergraduate course sections. As a sub-section of this aim, three methods frequentlyemployed as part of the instructional format inprogrammed courses were compared. Thesemethods were tutorial interview, small groupdiscussions, and written assignments.

Effectiveness of these techniques and overalleffects of the course organization were analyzedusing two different sets of responses: test per-formance measures and verbal evaluation mea-sures. The measurement of evaluative reactionsintroduces possible outcomes that are not en-countered when only performance measures aretaken, as is the usual case with behavioral tech-nologies. The criteria of verbal evaluation andperformance may not always suggest the sameconclusions about a particular educational de-sign. What are the determinants of the two setsof response measures, and are they in any wayorthogonal? As an outgrowth of the technologi-cal goals of the present study, preliminary an-

283

1972 5,~283-291 NUMBER 3 (FALL 1972)

GROVER J. WHITEHURST

swers to these and other questions were sought.These questions go beyond technology to theissue of complex behavior, how it is to be char-acterized, and the variables of which it is afunction.

METHOD

SubjectsIn the Fall semester, 50 students enrolled in

a section of a course entitled Child Develop-ment. This was one of three sections of thecourse offered that semester. Total enrollment inall Child Development sections for that semes-ter was approximately 250. The followingSpring semester, 247 students registered in theonly section of Child Development to be offered.Students in the Fall had no way to anticipatebefore enrolling that the content or pedagogy oftheir course section would differ from that usu-ally associated with the Child Developmentcourse, or from that being offered in the othertwo sections. In the Spring, many students wereundoubtedly forewarned of the nature of thecourse, but the lack of alternative sections lim-ited any selection biases that this might have in-troduced. In both the large and small course,approximately 65%, of the students were psy-chology majors and most enrollees were juniorsor seniors.

ProcedureCourse content. The content of the course was

arranged to (1) introduce the student to thebasic principles of behavior analysis, (2) to ap-ply these principles to a survey of data and the-ory in child development, and (3) to have thestudent recognize the application of the princi-ples learned to applied problems in child-rearingand society. No single text in child developmentseemed to serve all these functions, and there-fore reading was assigned from many sources.Students were required to read all or substan-tially all of two child development texts and abook of readings by Bijou and Baer (1961,1965, 1967). They were assigned all of a pro-

grammed text by Geis, Stebbins, and Lundin(1965) and portions of a programmed text byHolland and Skinner (1961). Also requiredwere portions of books by Skinner (1953) andWatson (1965). In addition to these materials,11 journal articles were assigned and studentsalso received five unpublished handouts cover-ing topics such as schedules of reinforcement,basic and societal stages of development, andother theories of development. All this materialconstituted 811 pages of reading. Eight filmswere shown during the course of the semester.Formal lectures were delivered on seven occa-sions.

Small course organization. The content of thecourse section with the enrollment of 50 wasorganized into 13 units, each unit correspondingto one week of class time. The class meet threetimes each week in hourly meetings. Studentsreceived a course syllabus on the first class meet-ing listing their assignments by units and thedates for various presentations in class.Monday meetings of the class were devoted to

films and formal lectures. Wednesday class meet-ings were devoted to group discussions of mate-rial assigned for that week. Students who arrivedin class on Wednesday were divided into twoor three groups, depending on the number ofstudents present. One group remained in themeeting room and the other group(s) movedto public lounges in the same building. Individ-ual group discussions were led by the instructorand two graduate teaching assistants assigned tothe course. Group discussions were relativelyunstructured. Each group discussion leader pre-pared his own list of questions to use in initiatingdialogue within the group. The only instructionto the graduate student assistants was to get asmany students talking about the material as pos-sible. Group discussion leaders were free to an-swer any questions about the material raised bystudents during the course of the discussion.

Friday sessions were devoted to tests. The ex-aminations consisted of 25 multiple choice itemsdrawn solely from the reading material assignedfor each unit. The tests were handed out at the

284

PROGRAMMED DEVELOPMENTAL COURSE

beginning of class and the student was instructedto mark his answers once on an answer sheet andonce on the test. When he finished the test, hisanswer sheet was collected but he retained thetest. When everyone had finished, correct an-swers were written on the blackboard and the re-mainder of the class time was spent discussingthe examination and clarifying general misun-derstandings.

The student was required to pass the multiple-choice test with three or fewer errors. If moreerrors were made, or if the student missed thetest, he was required to attend a make-up involv-ing an alternate form of the test on the Wednes-day following the Friday test. This was con-ducted outside of regular class hours. The sameerror criterion was in effect. If the student failedtwo written tests, he was required to attend aone-to-one tutorial with one of the graduate as-sistants until the assistant was satisfied that thestudent had command of the material for thatunit.

The final unit of the course was based on thetheme of developing individuals interacting witha developing society. The usual weekly formatwas not followed. Discussions were held aboutexperimental societies, educational institutions,and other societal-individual phenomena. Eachstudent was required to submit a short paper inlieu of a test on this final unit. Papers were tobe based on an application of the principles stud-ied in the course to some societal or individualphenomenon. The papers were graded on a pass-fail basis. Unacceptable papers were returned totheir authors with instructions to re-write them.On the final meeting of the semester, a Uni-

versity sponsored questionaire on reactions to

the course was distributed and completed inclass.

Students were informed at the beginning ofthe course that they could receive either an A oran F as a course grade. The grade of A would begiven to all students passing all units by test ortutorial and submitting an acceptable paper be-fore the end of the semester. Any student whodid not complete all units and submit an accept-

able paper would receive an F. Students couldalso receive withdrawal grades (no-credit) inways consistent with university regulations.

Large course organization. The course withenrollment of 247 students was conducted in alarge lecture hall with approximately 450 seats.In addition, the university provided a large officefor use by course assistants and an additionallecture hall on Wednesdays. One graduate stu-dent assistant was added to the two assistantswho had been assigned to the small course. Tenundergraduates who had received the grade ofA in the small course volunteered to serve foracademic credit as tutors in the large course.They were told that their grade would be basedon their completion of all assigned duties andon the performance of their students.The Monday course meetings remained the

same as they had been in the small course. Lec-tures were given and films were shown. Studentswere not directly tested on these films and lec-tures and were not required to attend.

Friday meetings also remained the same. Thesame tests were employed. The same error cri-terion was in effect.

Make-up procedures were also the same, withthe following exception. If a student failed amultiple-choice make-up test, he had to submitanswers to a prepared set of questions on hisnext attempt to pass that particular unit. Theinterview with a graduate assistant was reservedfor those students who had failed a multiplechoice make-up test and who had failed the longanswer written questions.The significant changes from the procedures

used for the small course were made for theWednesday class meetings. For the small course,unstructured voluntary group discussions wereheld. For the large course, students were givenat the beginning of each unit a list of discussionquestions. There were usually eight or nine de-tailed questions. Some questions were designedto make the students integrate written materialassigned for the unit; other questions were de-signed to focus the students' study on relevantportions of the reading; and other questions

285

GROVER J. WHITEHURST

were aimed at points in the reading or on thetests that had been particularly confusing to stu-dents in the small course. The discussion ques-tions covered enough of the assignment so thatpreparation of the answers after having read thematerial should have ensured a passing mark onthe Friday test.

Students were required to participate in sixWednesday exercises based on these discussionquestions during the course of the semester.There were three different types of Wednesdayexercises available. Students could hand in writ-ten answers to the questions. Written correctionsto these answers were supplied by their under-graduate tutor and the student could pick up hiscorrected answers on Thursday morning. Thesecond type of exercise involved the student en-gaging in a one-to-one tutorial with his under-graduate tutor. These tutorials were held duringthe Wednesday class meeting. They were sched-uled to last 10 min. The tutors had previouslybeen supplied with correct written answers tothe discussion questions. The tutor would askthe student to answer each of the questions fromthe question list in turn. After the student pro-vided an answer, the tutor would correct misin-formation. If the student came to his tutorialwithout having prepared answers, the tutorswere instructed not to provide answers for thestudent and to terminate the tutorial. Finally,students could participate in a group discussionbased on the discussion questions. Four groupdiscussions were held each Wednesday, led bythe three graduate student assistants and the in-structor. Group discussions lasted for an hour.The group discussion leaders were instructed togo through the questions eliciting answers fromstudents. Misunderstandings and incorrect an-swers were to be corrected and related topicswere to be pursued as time permitted. Group dis-cussions were held simultaneously in differentlocations in the main lecture hall and thesmaller lecture hall that had been provided.

Students were required to participate in twoof each of these three types of Wednesday exer-cises, making six required exercises in all. At the

beginning of the semester, each student wasgiven information indicating those Wednesdayswhen he was required to participate, the methodof participation required on a particularWednesday, and the location in the lecture hallsto which he was to report. These requirementswere so organized that a particular student hadto participate every other week over the first12 units. The order of participation in the threetypes of Wednesday exercises was counterbal-anced across the six exercises for each student.Students were told that on the Wednesdayswhen they were not required to participate, theycould participate in the Wednesday exercise oftheir choice or they could do nothing.The result of this organization for the stu-

dents as a whole was that half the students wererequired to participate in a Wednesday exerciseone week, and the second half the next week,and so forth through the semester. In a givenweek, one-third of the students who were re-quired to participate handed in written answers;one-third went to tutorials; and one-third went togroup discussions. All possible counterbalancedorders of Wednesday exercises were representedacross students. Assignment to a particular orderwas random.

After the eighth unit of the course, these pro-cedures were changed so that attendance atgroup discussions became completely optional;attendance at tutorials and written assignmentswas still required. The nature of the group dis-cussions was also altered. Group discussions nolonger were focused on the discussion questionsbut were allowed to proceed in an unstructuredmanner, as they had in the small course.

At the final class meeting, questionairesabout reactions to the course were distributedand completed.

RESULTS

Both the small and the large course produceda large proportion of students receiving gradesof A. In the small course, 92% of the studentsappearing on the official course roster and tak-

286

PROGRAMMED DEVELOPMENTAL COURSE

ing more than two tests received the grade of A;6% of this group of students withdrew from thecourse and received no credit; 2% of these stu-dents received the grade of F. In the large course,96% of the students appearing on the officialcourse roster and taking more than two testsreceived grades of A, 3% withdrew, and 1%received Fs. Evidence related to the question ofwhether these grades were fairly earned comesfrom responses to the anonymous questionairegiven at the end of the small course. Eighty-twoper cent of the respondents indicated that theyhad done all of the required reading in the,course and the remaining 18% indicated thatthey had done most of the work.

Further comparison of the large and smallcourse can be made on the proportion of stu-dents failing their first test and having to takesome form of make-up examination. In thesmall course, an average of 14% of the studentsreceiving a final letter grade failed the first formof their unit test in a given week. This figurewas 12% for the large course.An administrative failure on the part of the

university made it impossible to use the sameattitude survey for both the large and the smallcourse. Nevertheless, some data on the compara-tive student evaluation of the two courses canbe obtained by examining responses to a similarquestion occurring on both survey forms. Stu-dents in the small course were asked to rate theirinstructor on a five-point scale, compared withall other instructors they had had at the univer-sity. The mid-point of this scale was labelled"about average". Students in the large coursewere asked to rate their course on a seven-pointscale compared with all other courses they hadhad at the university. The mid-point of this scalewas labelled "about the same". Of the 38 re-spondents in the small course, none respondedbelow the mid-point of the scale; in the largecourse, 23% of the 181 respondents rated thecourse below the mid-point of the scale. Furthercomparative data on the effectiveness of the twocourses in generating interest and positive eval-uations can be obtained by comparing attend-

ance figures for the voluntary Monday classmeetings. In the small course, attendance onMondays dropped at its lowest point to 30% ofthe enrolled students. In the large course, attend-ance at its lowest point dropped to 8% of theenrolled students and after the first two unitswas never higher than 25% of the enrolledstudents.

Within the large course format, it is possibleto compare the effectiveness of attending a tu-torial, attending a group discussion, handing inwritten answers, or doing nothing on Wednes-day in terms of unit test scores. Since the proce-dures of the course were changed after Unit 8to make group discussions voluntary, data forthe analysis were drawn from the first eight unittests only. Also, since some students throughUnit 8 had not engaged in all of the three typesof Wednesday exercises, data were entered inthe analysis from only those students who hadparticipated in all three types of Wednesdayexercises and who had done nothing on at leastone Wednesday through Unit 8. A single factorrepeated measures analysis of variance (Winer,1962) was conducted on these data (F 4.17,df =3, 612, P= <0.01), and the hypothesisthat there was no differential effect on unit testsattributable to the type of Wednesday exerciseengaged in was rejected.The mean number of incorrect answers on

multiple choice tests after preparing with writ-ten exercises was 1.56, for tutorials 1.65, forgroup preparation 1.88, and for no formal prep-aration, the mean number of incorrect answerswas 1.90. A comparison of the average of thetutorial and written means with the average ofthe group and nothing means was significantat the 0.01 level. None of the remaining orthog-onal comparisons were significant. Thus, answer-ing discussion questions in a tutorial or in awritten form resulted in significantly fewererrors on the unit tests than did attending agroup discussion or engaging in no formal activ-ity in relation to-discussion questions. The firsttwo methods did not differ significantly fromeach other, nor did the latter two methods.

287

GROVER J. WHITEHURST

Contrasted with these results obtained usinga performance criterion are the results from theattitude questionaire administered in the largecourse. Replying to a question about which typeof Wednesday exercise they found most helpfulin preparing for the Friday test, 44% of thestudents picked the group discussion method,34% picked the tutorial, and 22% picked thewritten method. Replying to a question aboutwhich type of Wednesday exercise they foundmost enjoyable disregarding preparation for theFriday tests, 62% chose the group discussion,33% chose the tutorial method, and the remain-ing 5% chose the written exercise.The effect of changing the Wednesday group

discussions from required to voluntary, whilealso changing the nature of those discussions,was assessed using the measure of class attend-ance. Class attendance per week for the three

Wednesday exercises is presented in Figure 1.Note the dramatic drop in attendance for thegroup discussions subsequent to the change ofprocedures after Unit 8 compared with the base-lines for the other exercises.

DISCUSSION

Evaluated against test performance criterion,both the large and the small course were highlysuccessful. The large course was slightly moresuccessful than the small course in this regardwith more students passing the first forms ofunit tests and more students getting As. If oneassumes that a dramatic increase in class sizeshould result in poorer student performance,then the equality or superiority of the largecourse must be attributed to the introduction of

- -* GROUP

0 60z

1 40

p20

Ez

o 10

-- TUTORIAL

.h-WRITTEN

I I I I I I I III I I Imjm1 2 3 4 5 6

UNITS7 8 9 10 11 12

Fig. 1. Number of students performing each of the three types of Wednesday exercises by course unit inthe large course. After Unit 8, attendance at the group exercise was made optional and discussions in groupsno longer focused on discussion questions.

288

PROGRAMMED DEVELOPMENTAL COURSE

study questions and associated review proceduresin the large course.While not evaluated experimentally, it is

quite probable that the high proportion of Asobtained by both classes was due in part to theprovision of only one other alternative to an A,an F. Students who would otherwise have beensatisfied with a grade lower than an A per-formed at high levels in order to avoid an F.The desirability of such a procedure might bequestioned if evaluated against standards otherthan test performance. Students working in anavoidance paradigm for an A may be less likelyto use the content of the course in new situa-tions; they may dislike the course and by ver-balizing their dislike, harm interactions betweeninstructors and other students; they may be morelikely to cheat and engage in other "grade-ori-ented" behaviors. These questions remain to beevaluated. The A-F procedure also deprives theuniversity of the numerous grades differentials itrequires in order to confer distinctions upon stu-dents. This may or may not be a negative attri-bute of the procedure.

Despite the equality or superiority of thelarge course on test performance criterion, thereis the evidence from attitude surveys and fromvoluntary class attendance that the large coursewas liked less than the small course and gener-ated less student interest and involvement. Thesedata are strongly supported by the impressionsof the instructor and graduate assistants whowere struck by the unfavorable change in thenature of interactions with students that occurredwith the change to the large class. Student-staffrelationships were often adverserial in the largecourse but there was an almost complete absenceof this type of interaction in the small course.It may well be that when class size reaches thelevel at which the instructor cannot recognizehis students when he sees them outside of classand at which most of the students feel no per-sonal contact with the student in the adjacentseat, there are changes in the normal controllingfunctions of social stimuli. Students for whomthe approval of peers or the instructor is a

powerful reinforcer may react very differentlyto these same stimuli when delivered by peersand an instructor who do not know the students.There is a strong suggestion that these effectsoccurred in the large class, notwithstanding thefact that students were learning more accordingto performance criteria. Such effects should notbe ignored by those planning the policies of in-structional institutions, for they probably influ-ence to a high degree collateral response mea-sures such as student interest and involvement,which may be more predictive of ultimate edu-cational objectives than are course grades.While the three types of Wednesday exercises

did not produce large absolute differences in per-formance on the Friday tests, the tests were sodesigned that they produced an extremely con-stricted range of scores. Thus, the differencesthat the separate preparation methods wouldproduce on a more discriminating measurewould probably be substantially larger. The rel-ative effectiveness of the three types of Wednes-day exercises in preparing students for Fridaytests was reversed from informal predictionsmade by the instructor before instituting theprocedures. It was reasoned that since the in-structor and the graduate students had muchmore information at their disposal than the un-dergraduate tutors, the students would learnmore from them than from the relatively un-sophisticated undergraduates. An explanationfor the obtained results may be as follows: astudent having to prepare for a tutorial or hav-ing to hand in written answers was forced toread the material in order to prepare his answers.Students coming to a group discussion, in con-trast, could do relatively little preparation andobtain answers to the discussion questions fromthe few well-prepared students as they volun-teered responses to the group leader. Armedwith these pilfered answers to the discussionquestions, students could do reasonably well onunit tests but could not anticipate subtleties inquestions that were obvious only after havingread the assigned material. Thus, these studentsdid relatively less well on unit tests.

289

290 GROVER J. WHITEHURST

One function of the change in procedures afterUnit 8 was to test the preceding hypothesis. Thedrop in attendance as the group leaders nolonger focused on the answers to discussion ques-tions could of course have been a result of eitherthe change in requirements or the change in thenature of the discussion section. However, thefact that 28 students came to the discussiongroup for Unit 9, even though it was completelyvoluntary, coupled with the gradual drop in at-tendance thereafter, suggests that students nolonger attended when they found that theycould not obtain full answers for their at-tendance.The disparity between the attitude measures

of the Wednesday exercises and the test per-formance measures of these exercises is interest-ing. Note that one is the mirror image of theother. Presumably, the students' evaluation ofthe enjoyableness of the three methods in-fluenced them to judge incorrectly the effective-ness of the methods in terms of test performance.There are numerous related findings from thefields of perception (Goldiamond, 1958) andsocial psychology (Kutner, Wilkins, and Yar-row, 1952) indicating that subjects' verbal re-ports are not always highly correlated with otherresponse measures. Confronted with these kindsof data, there has been a tendency, particularlyamong "operant" psychologists, to resolve thedilemma by rejecting verbalizations and attitudestatements as less-reliable indicators of environ-mental effects than non-verbal measures. Astranslated into a classroom analysis, this wouldmean that instructional effectiveness would bedefined in terms of test performance, not atti-tude statements. This may be unfortunate. Onintuitive grounds, one would predict that themore students enjoy an educational experience,the more likely they are to pursue the contentof that experience beyond the bounds of theclassroom. This is clearly desirable. On the otherhand, if students do not acquire certain skills asa result of exposure to a technology, they willbe poorly equipped to advance their educationsin an independent fashion or to make immediate

use of the material that has been presented tothem.

This analysis suggests that we need to knowmany more of the determinants of what seem tobe partially independent measures. We need todevelop technologies that produce skill acquisi-tion and student satisfaction. The practice ofderiving measures of teaching success from onlyone of these areas is likely to produce instruc-tional techniques that are much less effectivethan they could be in producing long-termlearning and involvement.

REFERENCES

Bijou, S. W. and Baer, D. M. Child development,Vol. 1. A systematic and empirical theory. NewYork: Appleton-Century-Crofts, 1961.

Bijou, S. W. and Baer, D. M. Child development,Vol. 2. Universal stage of infancy. New York:Appleton-Century-Crofts, 1965.

Bijou, S. W. and Baer, D. M. Child development:readings in experimental analysis. New York:Appleton-Century-Crofts, 1967.

Ferster, C. B. Individualized instruction in a largeintroductory psychology course. PsychologicalRecord, 1968, 18, 521-532.

Geis, G. L., Stebbins, W. C., and Lundin, R. W.Reflexes and conditioned reflexes, Vol. 1, Part1. New York: Appleton-Century-Crofts, 1965.

Goldiamond, I. Indicators of perception: I. Sublim-inal perception, subception, unconscious per-ception: analysis in terms of psycho-physical indi-cator methodology. Psychological Bulletin, 1958,55, 373-411.

Holland, J. G. and Skinner, B. F. The analysis ofbehavior. New York: McGraw-Hill, 1961.

Johnston, J. M. and Pennypacker, H. S. A behav-ioral approach to college teaching. AmericanPsychologist, 1971, 26, 219-244.

Keller, F. S. "Good-bye, teacher ...." Journal ofApplied Behavior Analysis, 1968, 1, 79-89.

Kutner, B., Wilkins, C., and Yarrow, P. R. Verbalattitudes and overt behavior involving racialprejudice. Journal of Abnormal and Social Psy-chology, 1952, 47, 647-652.

Lloyd, K. E. and Knutzen, N. J. A self-paced pro-grammed undergraduate course in the experi-mental analysis of behavior. journal of AppliedBehavior Analysis, 1969, 2, 125-133.

Malott, R. W. Contingency management in an in-troductory psychology course for 1000 students.Kalamazoo: Behaviordelia, 1971.

McMichael, J. W. and Corey, J. R. Contingencymanagement in an introductory psychology course

PROGRAMMED DEVELOPMENTAL COURSE 291

produces better learning. Journal of Applied Be-havior Analysis, 1969, 2, 79-83.

Sheppard, W. C. and MacDermot, H. G. Designand evaluation of a programmed course in intro-ductory psychology. Journal of Applied BehaviorAnalysis, 1970, 3, 5-1 1.

Skinner, B. F. Science and human behavior. NewYork: MacMillan, 1953.

Watson, R. I. Psychology of the child. New York:Wiley, 1965.

Winer, B. J. Statistical principles in experimentaldesign. New York: McGraw-Hill, 1962.

Received 24 September 1971.(Revised 10 January 1972.)