s

243

TUTORIAL CAI IN TECHNICAL AREAS:SKILL DEVELOPMENT USING AN INFORMATION NETWORK

Avron Barr, Marian Beard, and Richard C. AtkinsonInstitute for MathematicalStudies in the Social Sciences

Stanford University

Stanford,

California 94305

and

Paul Lorton, Jr.University of San Francisco

San Francisco, California 94117

As part of a general effort to explore tutorial modes of Computer Assisted Instruction (CAI), we aredeveloping a course which will serve as a model for CAI in technical areas. The course teaches computerprogramming; however, it is being designed so that it will be appropriate to training in other technical areas andapplicable in other instructional settings where the developmentof analytic and problem-solving skills is the goal.

To this end we are incorporating methods to monitor and aid the student as he works on interestingproblems from various fields. The emphasis of the course, and our research, is on exploring tutorial

CAI,

characterized by appropriate interactionwith the student when he needs assistance. The instructional programwehave developed will be used to investigate schemes for optimizingproblem presentation. Our previous experiencein the instructional and technical aspects of teaching a programming language leads us to believe that a course incomputer programming can be designed to help the student acquire concepts in personalized and efficientmanner while he is developing programming skills at increasingly advanced levels. Through out current work, weare collecting data to evaluate this proposition.

A major goal of the research project is to increase the sophistication with which the instructional programmonitors the student's work and responds to him with appropriate hints and prompts. One aspect of such workis the utilization of algorithms for checking the correctness of a student procedure (or program). Although it isunrealistic to have the computer evaluate the correctness of an arbitrary procedure, we believe that within thecontrollable context of instruction, procedure verification is a possibility. We expect to develop procedureverification techniques that will be very general in nature and applicable to the teaching of other technicalsubjects.

The BASIC Instructional Program (BIP) is presently being offered as an introductory programming courseat two nearby schools, DeAnza College in Cupertino and die University of San Francisco. The course is not yetfully developed, and the data that has been collected is being used to modify the problems and the "help"sequences, in preparation for the more controlled experimental situations we plan for the future.

Overview of Past IMSSS Research in Teaching Programming

At the Institute for Mathematical Studies in the Social Sciences (IMSSS), we have been involved in CAIprojects in computer programming and in tutorial CAI in other technical areas for some time. Our work, inteaching computer programming began in 1968-9, with a CAI Course in machine language programmingdeveloped as a high school level course (Lorton & Slimick, 1969). The project, called SIMPER, taughtprogramming via a simulated 3 register machine with variable instruction set. The project was later expanded toinclude lessons in the BASIC language.

In 1970, the Institute developed a much larger CAI curriculum for a new course to teach the AIDprogramming language at the introductory undergraduate level. This course has been used in colleges and juniorcolleges as a successful introduction to computer programming (Beard, et al., 1973). However, if is a linear,"frame-oriented" CAI program and cannot provide individualized instruction during the problem solving activity

244

itself. After working through lesson segments on syntax, expressions, etc., and then being assigned a problem tosolve with an AID program, the student must leave the instructional program, call up a separate fnterprlrperform the required programming task, and return to the instructional program with an answer. As he develophis program directly with AID, he has no access to instructional help, except for the typically semi-informat onaerror messages supplied by the interpreter. miormationai

In recent years however, developments in interactive CAI and in artificial intelligence have enabledaching programs to deal more effectively with the subject matter they purport to teach, or in effect to knowtheir subject. The generative CAI programs developedbv Carbonell and others (Carbonell, 1970 and Collins e7al. 1973) can construct and present questions c. „ . iarge data base and evaluate the answers. Courses intechnical areas (logic and integral calculus) have been developed at IMSSS which . ,pable of dealing in a ve^sophisticated way with both their subject matter and with the student. These Curses providT nstructZval'at nTth 3e feTf T*" Pr°Wem , "^by perf°"ta« °perati°

nS

S^d * *^S£evaluating the effect of such operations, and, on request, suggesting a "next step" in the solution The Logicprogram is described in Goldberg (1973) and the Calculus program is described in Kimball (1973).A Description of BIP

The core of BIP is an information network which embodies the interrelations of the concepts skillsproblems, remedial lessons available BASIC commands, and manual references. It is related to the wor reportedby Koffman (1972) and Nievergelt, et. al. (1973). With a sufficient student history, we believe that the networkcan be used successfully to control the individualized problem sequence, the frequency and typ of ass^n»given during programming, and the identification of problem areas. The skill nodes in Z Iwok arnecific eT

S

I" 7

C

°nCeP,;, n°deS °ike "°UtpUt" °r "10QPS"> and the Pfoblem "des. Ski 11 arfve"P ifi e.g. printing a string variable or incrementing a counter variable. By evaluating success on the individudhe iudrntTdTuSr3168 " **" " «"" *" and also so^err k1 u

lntj

erPreter > desiSned especially for BIP, analyzes each program line the student types andnotifies both the student and the Instructional Program (IP) of syntax errors and runtime "execut on" e'rors Ofcourse, the IP can control the availability of the BASIC commands as a pedagogical device. Interactive I buggingaids (tracing and program generatedflow charts) are being developed. "slugging

H.t,

mL* T?J° m yZm! Pr°gramS f°r SyntaX and e^cutability, it is necessary to have some facility todete mine what the program does, in the context of determining whether it is a solution to the givenproblemZon zr on se?a r y; to achr e Ms result- we wm initiaiiy com*" **» °f "**«. roZrun on test data against that of a stored model solution, but we plan to adapt the theoretical work in artificialintelligence on program verification (Manna, 1973) to the task of solution evaluationGoals of the Curriculum

Prior experience with CAI in programming languages has convinced us that many students who wish tolearn the fundamental principles and techniques of programming have limited mathematical backgrounds Moreimportant, their confidence in their own abilities to confront problems involving numeric manipulation is ve"km, InH SC°Pe TTiCUlUm ' theref°re ' iS reStrkted t0 indude ** most f-damental Prog"amZskills and a minimum of mathematical sophistication. i""Bi«uiiimng

The curriculum is designed to give the student practice and instruction in developing interactive programsThe course exposes the student to interactive uses of the computer, such as game playing, with which hfmaywel be unfamiliar. Rather than emphasize "number crunching" or programs with a one-time^ MP guides"het^eTlir T°n °f Pr°,BramS "" "Sh°W °ff '' The 6mphaSis is °n P»«">« can be used b^other people, and programs that are entertaining. The student writes programs with a hypothetical "user" inpTgdra^rnbet^L

USe

StUd6nfS Pr°B^ MS —' » Wh°m *e performance of the

245

In general, the curriculum provides useful, entertaining, practical experience in dealing with a computer forstudents who are not necessarily mathematically oriented. It gives them the opportunity to develop programmingskills while working on problems that are challenging yet not intimidating, in which the difficulties stem fromthe demands of computer programming rather than from the complexities of mathematics, or some other subjectsuch as physics or statistics.

Sources of Principles and Problems

Numerous texts were examined as sources, both for the necessary programmingprinciples to be developedin an introductory course, and for problems that illustrate those principles. We incorporated ideas from generalcomputer science textbooks (Forsythe, et al., 1969), from an introductory programming text whose focus is theALGOL language, but whose examples are easily generalized (Floyd, 1971), and from various books and notesdealing specifically with BASIC (Kameny & Kurtz, 1971;Coan, 1970; Albrecht, et. al., 1973;Nolan, 1969; andvarious publications of the People's Computer Company and the Lawrence Hall of Science). In addition, problemsets from Stanford University's introductory computer science courses were collected and examined.

Many of the programming exercises and projects that we examined demanded of the student a considerableinvestment of time, and a greater degree of sophistication than BIP was designed to teach. However,by analyzingthese problems, we developed a reliable checklist of necessary fundamental concepts that we were interested inteaching:

Stored ProgramsVariables and LiteralsExpressionsInput and OutputProgram Control-BranchingRepetition—LoopsSubroutinesData Constructs-Arrays and Strings

Organization of the Curriculum

A major goal of our research is to examine the effects of different strategies of problem selection andcurriculum branching. We are interested in comparing student performance under a student-choice mechanism,where he is allowed freedom and given sufficient information to choose the type of programming problem hewill attempt next, versus some program-choice strategy, where BIP analyzes both its own data base and thestudent's history and selects problems on the basis of the skills required, the concepts introduced, and thestudent's past performance in the relevant areas.

The curriculum is organized as a set of discrete programming problems called "tasks." There are no"question and answer" type lessons, and there is no obvious sequence of tasks. The organization is intentionallynon-linear, so that the selection of curriculum cannot fall back on a default ordering of tasks, but must dependupon an analysis of the knowledge state of each student. In a "student selection" mode, this means that thestudent is forced to evaluate his own skills and deficiencies. Tasks are not given sequential numbers or names,explicitly to prevent the student from abdicating responsibility for his own evaluation. In a "program selection"mode, the absence of a linear sequence means that BIP must use the information available to it in determiningthe next task to present. Only in the absence of sufficient data would BIP make a default choice.

A student progresses through the curriculum by writing a program that solves the problem presented on histerminal. There are virtually no limitations imposed on the amount of time he spends, the number of lines hewrites in his program, the number of errors he is allowed to make, the number of times he chooses to executethe program, the changes he makes within it, etc. When he feels that he has solved the problem, he types"MORE" and he chooses, or is given, a new problem.

246

A student is not allowed to complete a problem if he has not executed his current program successfullyBIP has available at all times information concerning the student's program, and can determine a) whelr anexecutable program exists, b) whether the student has executed that program,c) whether or not executeTerror"occurred and d) whether the student has made changes or additions since the last execution, maymove on to a new problem only if he has succeeded in an error-free execution of the most recent vrlfon of hh;Eh:::;v :ei« methods for determining whether his program * an «"*■«■Upon completion of a task, the student is given a "post task interview" in which BIP presents a "model"so u ion to the problem. (The student is encouraged to regard the model as only one of ft many po si e1 k IZeth Si* ,thlStUde!:Vf haS S°lVed ProWem ' th6n asks ' for each °f *»

*ffl,

requTrL Pb7thtask whether he feels the need for more practice involving that skill. The responses are stored and used in futureB P-generated curriculum decisions. BIP then informs the student that he has completed the task and eiftlows him to select his next task by name (from an off-line printed list of names and" problem texts), or slfcThe Manual

The BIP course currently uses Model 33 teletypes as student terminals. These devices are slow and noisvSeer ntfPenSlVe, "? ""T* dUraWe- °f Sl°W Sp6ed' lt is tedious to P».eTvoS^««°;ot et71r from the cr puter direcdy °n the terminai- we hav-h— -"Ste^ZS on he n Tm/ pnnted "nual of approximately 50 pages. The manual includesZ L P 10n °f the

C

°UrSe (signing on' dealinS with *e terminal, dealingwith BIP) ageneral introduction to computers and their capabilities, and the syntax of BlP's BASIC, complete with Sampleand suggestions for the appropriate uses of the various BASIC statementsexamples

All programming terms used in the manual and in the tasks are defined briefly in the glossary at the end ofhe manual. References to the relevant sections of the manual are included in each glossary ent y M words Jhahave precise programming meanings, different from their normal English meanings, are listed.We believe that when the student encounters another programming language with which he is not familiarh s primary resource will be the manual for that language. He is not likely to have an instructor o a CA Tourseat hand, and the principal means by which he will learn the new language will be through his ownexperimentation, guided by the explanationsand examples in the manual. Experience with BIP (withes f quentross-references to the manual) will, we hope, give the student a degree of confidence and ease in fed ing h 'wayin other situations,when the manual may be his only guide. E y

Options Available to the Student

HPtP tT ;"diately analyzes each BASIC statement the student types. If an error in the statement can bedetected at this time (e.g., an illegal variable or an incomplete expression), the line is not accepted and a vntaxnurnrbr SSage I! 'T*' *"*"*>*" execUtion error occ" ("*" an unassigned variable, or a lexis enU nenumber in a branching statement), BIP stops and prints an error message. In either of these situations th!s udent may type a questionmark and receive information specifically directed at the error he made Th s U"information is designed to assist the student in understanding the cause of the error, and proTdes examples ofcorrect and incorrect statements, as well as references to the manual. Many of the syntax ZTex cut on orIZtZl^: °ne aSS°dated Wnt St°red ' WWCh *■ StUd6nt "* °btai" »o, or thr or evenZWithin a task, when the student is writing a program intended to solve a particular problem additionaloptions are open to him. At any time, he may request the model solution. He is free to copy and mn fte moSeprogram, or to modify parts of it to fit into his own program. Also, if he wishes to leave'he Lk at any timwithout completing it, he may do so with one BIP command. That task will be presented to himZe s I Bffdoes not record his having solved the problem. '

247

BIP allows each student to save as many as four separate programs permanently. This gives him theopportunity to work on an extended programming project and, simultaneously, accumulate his work from eachsession at the terminal. His saved programs are always immediately retrievable for modifications or additions.

One feature that has been extremely useful in the developmental stage of the program is a BIP commandthat allows the student to send a message to the programmers at Stanford. This gives him a chance tocommunicate difficulties and confusions to us, and helps us to improve BlP's interaction abilities and to identifyand locate errors in the program. The tremendous convenience of simply typing a message encourages the studentto provide us with this immediate and valuable feedback.

Another feature of the program, although not strictly a student option, is the message printed each time astudent signs on to the course. This message is updated daily, and gives information about revisions to thecourse, responses to messages left by students, and notices of meetings with Stanford personnel at which studentsmaydiscuss questions too complex to be handled in short written messages.

Expansion of the Curriculum

Before the end of the school year, the number of BIP tasks will be increased from 23 to approximately 50.The additional tasks will involve all of the interpreter's capabilities in practical ways, and a number of moredemanding tasks will be written to give the student experience in dealing with a programming project of broaderscope.

Additional options within the curriculum structure will also become available. One of these is designed tosimulate the assistance that a human tutor might give to a student having difficultywith a complex task. It willallow the student to request "subtasks" of the current task, each of which will cover a part of the overallproblem with which the student is dealing. For example, a task that required the student to sort a list wouldhave available, as a subtask, a problem that required him to write a program capable of making the appropriatecomparison between elements in the list. Another subtask would require a program whose only function was toprint out the elements of a list.

When the subtask option is implemented, a comprehensive effort will be made to provide more explicitinformation to the student about his current position within the task structure. He will be told, for example,that he has completed a subtask, and is returning to the main task, and BIP will reprint the text of that maintask on request. If the student requests another task before completing the current problem, he will be remindedthat the current problem remains, and asked if he wishes to complete the old task before entering the new task,upon completionof the new task, or if he wishes to complete the old task at all.

We are in continuous communication with students using the course, whose suggestions regarding moreflexible, intelligible interaction with BIP have generated several improvements. Past experience has shown thatsuperficial problems in dealing with an instructional program become significant barriers to acquiring theconcepts and skills presented by the program, and we continue to make additions to BIP to eliminate frustratingconfrontations between the student and the uncomprehending machine.

Analysis of Current Data

We have begun evaluating the data collected on some 40 students during the Fall semester 1973. Inaddition to a complete record (stored on magnetic tape) of everything the student and BIP typed at the terminal,we have the results of various tests and inventories which were administered off line. The various measures wewill have for analysis include:

Examination scoresStudent Evaluation Form for Computer Assisted InstructionComputer Programmer Aptitude BatteryThe Orientation Inventory

248

The Adjective Check ListTest ofBehavioral Rigidity

antoun, „f time working . ' add"ioMlr^t,:^t—ti:t;i:;rcr^/rs,rt su"ey,r- d * ,Msss - **■I, no „p,„,„„, 4„, sttong di!aEreemcnt| 7 w|t „ ea;h"7, ",ea ""d«" '7 ,«d <° "><""»« *>"ng agreement,the,r own pace (Question 2), preferred no, cornnetin, Lth

,h

r nde°'

S

,ended ,0 like "-'Mng atcurriculum ,0„ easy (Qms „0 „P7) and '^Z*Z. xci«„T/n I "1?"' (Ql,eS"°° 4»' did "<" «"<> >he■n the Spring ls Questio„ I2 since ft Basic i" ,1,^,. So S(? ""I 8) ' °ne """ We wi" °bs«" closelyits ability ,„ g,,e feedback to the student l"

S

'rUC"0n'1 "^S" "as '«% ""Proved since the Fall semester inFuture Data Analysis Plans

academtSe^st^"" ""^ "^data "** **» end of the current

Perfo—^estte tj^^Z St^T <** '*" » «*etc. By correlating these indices with measured of fina. Petforma 'n, "T^°f """ Written f°r each **.and gain additional information on the more important vari" ' " '° ** pr°gram itself2- The Information Network. We have constructed an „d

hn,

■ econaderable advantage

Qf

"cted lnformat

netWQrk

rfvalidate and/or correct ad hoc formulations The student 1 ,

C

°lkCted that

Can

be used toinformation network as well as to provide "" » "**»* thewhich should allow further exploration into the possiMit nf '°me data' dted above>Certainly explorationsof responses to the adjec^^^^cai tT , have been surveyed on a wide -ty ofpresentation versus a different mo£^"S^" n^" J» one "d* * CAI curriculumtoward the BIP approach and toward . ta^StTdft rtUdentf atUtuto-tudes toward learning BASIC with their attitude^ «***

ft '""E^ P—-, be used in severa, way,It will provide some criterion information fo n^Svel^nJt " "nd Sub" task needs"for the student programs will be in the deviled "»"- But the most important useverify student programming efforts, to^^l^^*? *** «* to

Acknowledgements

"°^^^

249

from the staffs of the pilot institutions, notably Dr. Carl Grame of DeAnza College and Dr. Paul Lorton, Jr. ofthe University of San Francisco.

Table 1. Student Attitude Survey Analysis

SD QuestionMean1 . Worked as hard as in classroom2. Learned as well as in classroom3. Like working at own pace4. Prefer competingwith fellow students5. Having own tutor

3.133 2.0303.355 1.7622.613 2.186

1.9835.2583.484 1.5253.032 1.975 6. Twenty minutes enough time

7. Too easy5.097 1.4462.600 1.610 8. Exciting4.467 1.776 9. More frustrating than worthwhile

10. Take another CAI course3.161 1.80911. Too hard3.839 1.530

3.433 1.501 12. More feedback than classroom

250

References

1. R. L. Albrecht, L. Finkel, and J. R. Brown, BASIC, New York, Wiley, 1973.

2. M. H. Beard, P. Lorton, Jr., B. W. Searle, and R. C. Atkinson, "Comparison of student performance andattitude under three lesson selection strategies in computer-assisted instruction," Technical Report No. 222,

Stanford,

Calif, Institute for Mathematical Studies in the Social Sciences, Stanford University, 1973.

3. J. R. Carbonell, "AI in CAI: an artificial intelligence approach to computer-assisted instruction," lEEETransactions on Man-Machine Systems, MMS-11, 1970,pp. 190-202.

4. A. M. Collins, J. R. Carbonell, and E. H. Warnock, "Analysis and synthesis of tutorial dialogues," TechnicalReport No. 2631, Cambridge, Mass., Bolt, Beranek and Newman, 1973.

5. J. S.

Coan,

Basic BASIC, New York, Hayden Book, 1970,

6. R. W. Floyd, "Notes on programming and the ALGOL W language," Stanford, Calif., Computer ScienceDepartment, Stanford University, 1971.

7. A. I. Forsythe, T. A. Keenan, E. I. Organick, and W. Sternberg, Computer science, a first course. NewYork, Wiley, 1969.

8. A. Goldberg, "Computer-assisted instruction: The application of theorem-proving to adaptive responseanalysis," Technical Report No. 203, Stanford, Calif., Institute for Mathematical Studies in the Social

Sciences,

Stanford University, 1973.

9. John G. Kemeny and T. E. Kirtz, BASIC programming. New York, Wiley, 1971, 2nd ed.

10. R. B. Kimball, "Self-optimizing computer-assisted tutoring: theory and practice," Technical Report No.206,

Stanford,

Calif., Institute for Mathematical Studies in the Social Sciences, Stanford University, 1973.

11. E. B. Koffman, "Generative CAI tutor for computer science concepts," Spring Joint Computer Conference:1972,Proceedings, pp. 379-389.

12. J. Nievergelt, E. M. Reingold, and T. R. Wilcox, The automation of introductory computer sciencecourses," International Computing Symposium, 1973, Proceedings.

13. R. L. Nolan, Introduction to computing through the BASIC languaget New York, Holt, Rinehard andWinston, 1969.

14. P. Lorton, Jr., and J. Slimick, "Computer based instruction in computer programming-a symbolmanipulation-list processing approach," Fall Joint Computer Conference, 1969, Proceedings, pp. 535-544.

15. Z. Manna, "Program Schemas," Currents in the theory of computing, edited by A. V. Aho, Englewood

Cliffs,

N.J., Prentice Hall, 1973.