Children's Searching Behavior on Browsing and Keyword Online Catalogs: The Science Library Catalog...

Children’s Searching Behavior on Browsing and Keyword Online Catalogs: The Science Library Catalog Project

Christine L. Borgman,’ Sandra G. Hirsh, and Virginia A. Walter Graduate School of Education and Information Studies, University of California, Los Angeles, Los Angeles, CA 90024. E-mail: CBORGMAN@UCLA. EDU

Andrea L. Gallagher** Department of Psychology, University of Michigan, Ann Arbor, MI 48 709

As we seek both to improve public school education in high technology areas and to link libraries and classrooms on the “information superhighway,” we need to understand more about children’s information searching abilities. We present results of four experiments conducted on four versions of the Science Library Catalog (SLC), a Dewey decimal-based hierarchical browsing system implemented in HyperCard without a keyboard. The experiments were conducted over a 3-year period at three sites, with four databases, and with comparisons to two different keyword online catalogs. Subjects were ethnically and culturally di- verse children aged 9 through 12; with 32 to 34 children participating in each experiment. Children were provided explicit instruction and reference materials for the keyword systems but not for the SLC. The number of search topics matched was comparable across all systems and all experiments; search times were comparable, though they varied among the four SLC versions and between the two keyword online public access catalogs (OPACs). The SLC overall was robust to differences in age, sex, and computer experience. One of the keyword OPACs was subject to mi- nor effects of age and computer experience; the other was not. We found relationships between search topic and system structure, such that the most difficult topics on the SLC were those hard to locate in the hierarchy, and those most difficult on the keyword OPACs were hard to spell or required children to generate their own search terms. The SLC approach overcomes problems with several searching features that are difficult for children in typical keyword OPAC systems: typing skills, spelling, vocabulary, and Boolean logic. Results have general implications for the design of information retrieval systems for children.

* To whom all correspondence should be addressed.

** Formerly with the Division of Biology, California Institute of Technology, Pasadena, CA 91125.

Received October 5, 1994; revised January 3 1, 1995; accepted January 31, 1995.

0 1995 John Wiley&Sons, Inc.

Introduction

Automation of library processes has been underway since the 1950s first in research and university libraries, later in major public libraries, and now in libraries of all sizes and types (Griffiths & Kertis, 1994). Concurrently, schools have been adopting computers for classroom instruction, beginning in the earliest days of the personal computer in the mid-1970s. Given these parallel developments, it is surprising how little research has been done at the nexus of libraries and education, addressing questions of how to automate libraries for children in ways consonant with their learning, cognitive development, and curriculum.

We need to understand more about children’s information-seeking abilities, as we seek both to improve public school education in high technology areas through computer-based science programs and as we seek to link libraries and classrooms on the “information superhighway.” The public school curriculum, particularly in science, is shifting away from rote, textbook learning toward exploratory, hands-on, resource-based “discovery” learning.

The research reported here is an outgrowth of Project SEED (Science for Early Educational Development), be- gun in the mid- 1980s at the California Institute of Tech- nology. Project SEED is part of a broad instructional program implemented throughout the Pasadena (Cali- fornia) Unified School District and elsewhere, consisting of hands-on science projects, science simulations, an exploratory science adventure game, and the associated teacher training and curricular support. Early in the development of Project SEED, it became apparent that if children were to pursue discovery-based learning effectively, they needed the skills to search for information that would expand their knowledge beyond the specific classroom lessons. A library-centered school curric-

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE. 46(9):663-684, 1995 CCC 0002-8231/95/090663-22

ulum would be an appropriate model for true discovery learning.

As a first step, we worked with the Project SEED team to identify the scope and content of an information retrieval system for children that would be necessary for a discovery-oriented curriculum. Our goal was to understand children’s information-seeking behavior sufficiently to design retrieval systems that could be used effectively and enjoyably by children from a wide range ofbackgrounds, without prior computer experience, and without any training in the use of the system. We found that very little research had been done on children’s information-seeking behavior on which we could base our own exploration, so we went back to first principles of children’s cognitive abilities, knowledge, skills, and in- terests, as well as to basic information retrieval principles. Along the way, we worked closely with children’s librarians, developmental psychologists, and human-factors specialists to refine and develop the system iteratively. We report here on the results of four experiments conducted at three sites over a 4-year period. Some of these data and design criteria have been reported elsewhere in preliminary form (Borgman, Bower, Auth, & Krieger, 1989a; Borgman, Chignell, & Valdez, 1989b; Borgman, Gallagher, Krieger, & Bower, 1990a; Borg- man, Gallagher, Walter, & Rosenberg, 1992; Borgman, Ktieger, Gallagher, Bower, 1990b, 199 1 a; Rosenberg & Borgman, 1991, 1992; Borgman, Walter, Rosenberg, & Gallagher, 199 1 b; Walter & Borgman, 199 1); other data reported herein have not been reported anywhere. This is the first full synthesis of these experiments.

Related Research

Children’s Use qf Information Retrieval Systems

Our research focuses on library catalogs, as they are the one tool in common across all types of libraries and because a substantial amount of knowledge exists on adult behavior with automated catalogs (e.g., Borgman, 1986, Hancock-Beaulieu, 1989; Hancock-Beaulieu, 1991; Matthews, Lawrence, & Ferguson, 1983); these systems generally are called “online public access catalogs,” or OPACs.

In general, studies of the use of OPACs in elementary school libraries have found that students do indeed like using computer catalogs and prefer them to the card catalog (Armstrong & Costa, 1983; Utilization of a Micro- computer, 1983), although they have difficulties with them (Vandergrift, 1989). Children, especially young ones, reported less difficulty with the computerized catalog than with the card catalog (Armstrong & Costa, 1983); and one school found that their students used the computer catalog more than students at a control school used their card catalog (Utilization oj”a Microcomputer, 1983). In contrast, Edmonds, Moore, and Balcom (1990) found that elementary school students were less success-

ful on a touch-screen OPAC than with the card catalog and that 4th-graders in particular were unable to use either catalog without assistance. Children’s difficulties with the online catalog were attributed to the touch- screen interface which required children to move through many layers of screens and execute long sequences in order to locate information. While Solomon (1993) found that some children could use a keyword- based OPAC as early as the first grade, most children experienced breakdowns during the search process. Stan- dard information retrieval interfaces, as implemented on current commercial online catalogs, may require mas- tery of several important skills that elementary school children either do not have or are just beginning to develop around the time they complete elementary school. We discuss these skills individually.

Typing and Keyboarding. Most current OPACs require the user to enter specific commands and the keywords to be searched. Keyboarding, however, is a significant source of breakdowns for elementary school children (Solomon, 1993). Since keyboarding or typing skills are usually not offered until junior high or high school, and only rarely in elementary school (Eastman & Agostino, 1986; Erthal, 1985; Hooten, 1989), elementary students who must input commands into computer systems typically “hunt and peck” on the keyboard for the correct keys. The exception is the CARL “Kids’ Cat- alog” which has been widely implemented as a front-end interface to standard keyword OPACs, allowing children to explore subject hierarchies through a point-and-click interface and to locate selected items on a physical map of the library. These aspects of the Kids’ Catalog are sub- stantially based on design principles for the Science Li- brary Catalog, reported in earlier papers from the Sci- ence Library Catalog Project (Borgman et al., 1989a, 1989b, 1990a, 1990b, 1991a, 1991b, 1992; Busey & Doerr, 1993; Rosenberg & Borgman, 199 1, 1992; Walter & Borgman, 199 1).

Spelling. Keyword retrieval systems require accurate spelling. Few of the current systems support any sort of spell-checking (Borgman & Siegfried, 1992); many are unforgiving of word order and punctuation syntax as well. Hooten ( 1989) found that keyword systems require accuracy in spelling, spacing, and punctuation that is beyond the ability of most young users. Solomon (1993) identified spelling as a significant problem in elementary school children’s ability to find material on a keyword- based online catalog. Children’s spelling skills do not be- gin to improve until the 5th grade (age 1 l), as indicated by a decline in spelling errors in children’s essays (Taylor & Kidder, 1988). The movement in language arts instruction toward phonetic spelling in the early elementary grades (e.g., Nelson, 1989) may exacerbate the problem. While spell-correction would be extremely useful in

664 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE-October 1995

most retrieval systems, it would be especially valuable for children’s systems.

Vocabulary. Children’s vocabulary is not sufficient to understand many of the terms used as subject headings, even for books intended for their age group. Moll ( 1975) found that subject headings often were at a higher reading level than that of the book they described; only 70% of 8th-grade books and 32.1% of 6th-grade books in her study were assigned subject headings at or below the grade level of the book itself. Young children have particular difficulty with science-related words (Meyerson, Ford, Jones, & Ward, 199 1). Even the ability to select an appropriate term for a search is beyond the ability of many children, as Moore and St. George (199 1) observed in their study of 6th- and 7th-graders assigned to do a research report on birds. The students had difficulty selecting an appropriate search term and did not try alternative terms if the first attempt did not prove successful.

Studies of online catalog use by adults have shown that generating subject terminology is often the most difficult part of searching (Borgman, 1986; Markey, 1984; Matthews et al., 1983), thus it is not surprising that children have a difficult time generating appropriate terminology to match the records in a database. Even when children in grades l-6 search for topics they have selected themselves, as in Solomon’s (1993) qualitative study of children’s use of a keyword-based online catalog, they have difficulty in choosing appropriate search terms. Solomon found that children were frequently successful in completing their searches, as long as they used simple concrete search terms, such as “cats” and “dogs” which matched subject descriptors exactly. Older elementary school children, however, (e.g., grades 4-6) who have more complex information needs (e.g., topics like “ancient numerals”) were frequently unsuccessful in completing their searches. The more complex the search, the more difficulties children had in selecting appropriate search terms to match the subject descriptors.

Alphabetizing. Most keyword systems require extensive browsing of alphabetical displays. Alphabetizing, however, is one of the main problems children have with card catalogs (Edmonds et al., 1990). Most of the students tested by Edmonds et al. were “moderately skilled” or “unskilled” in alphabetizing and filing, with half of the 4th graders being “unskilled”; skills were age-related, with 6th and 8th graders having higher skills.

Boolean Logic. Boolean-based retrieval mechanisms were developed initially for power and computing efficiency (Belkin & Croft, 1987). As computing has become less expensive, easier to use retrieval mechanisms have been developed such as relevance feedback (Belkin & Croft, 1987; Cleverdon, 199 1; Hancock-Beaulieu, 1989, 199 1; Harman, 1992; Hildreth, 1987), and some

new systems offer non-Boolean methods and graphical interfaces. Boolean systems still constitute the vast majority of the installed base of online catalogs and other retrieval systems, however. These interfaces are difficult to use because they require the user to specify the exact combination of terms to be found in some set of un- known documents. Adults (Borgman, 1986; Case, Borg- man, & Meadow, 1989), high school students (Liebscher & Marchionini, 1988), and children (Siegler, 1986) have problems formulating Boolean queries.

Summary. Studies of children’s use of online catalogs and other types of automated information retrieval systems indicate that they are able to use such systems, but with varying degrees of difficulty due to system re- quirements for skills in typing, spelling, vocabulary, alphabetizing, and Boolean logic. These problems dimin- ish with age, as children’s skills increase. With few excep- tions, however, the current generation of online catalogs are designed for adults and do not meet the special needs and capabilities of children. Information retrieval systems for children must be designed on principles appropriate to their developmental level.

An Alternative Information Retrieval Model for Children

Children have a natural tendency to explore; discovery learning approaches build on children’s exploratory behavior. It follows that information retrieval systems for children similarly should encourage exploration, and should minimize the barriers of current systems outlined above. Recent research and theory is beginning to view both adult and child information seekers as active problem solvers, with an evolving information need that may be searched iteratively (e.g., Bates, 1989; Belkin, Oddy, & Brooks, 1982; Kuhlthau, 1988a, 1988b, 1991). We have identified several principles on which exploratory retrieval systems for children can be constructed, which we discuss individually.

Recognition Knowledge vs. Recall Knowledge. Chil- dren, as well as adults, find it easier to recognize information presented to them than to recall it from memory. Not surprisingly, recognition requires less cognitive load than recall in almost all tasks (Anderson, 1990; Kail, 1984), and this is particularly true for young children (Brown, 1975). The recall vs. recognition principle is ev- ident in systems that rely on menus or direct manipulation rather than commands. Systems that require entering keywords necessarily rely on recall knowledge, since the user must think of words and their synonyms from memory (Marchionini, 1987). Added to the burden of recall knowledge for children is their relatively smaller attainment of content knowledge, particularly in subject areas such as science that are not part of everyday knowledge. Siegler ( 199 1) explains that older children often are

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE-October 1995 665

able to recall more than younger children simply because they know more about the material they are trying to remember. The less the child knows about science, the more likely is the child to need recognition aids to prompt his or her memory and develop an appropriate search strategy. Similarly, children who know more in a particular domain are able to recall more concepts and integrate these concepts more readily into their cognitive structure (Chi, Hutchinson, & Robin, 1989) particularly when that information is highly structured, such as hier- archically (Chi & Koeske, 1983).

Recognition skills can be utilized in information retrieval systems by offering a choice ofcategories or terms; this approach can require little or no keyboard input, thus minimizing the need for typing skills, correct spelling, and a substantial vocabulary. The recognition approach requires sufficient vocabulary knowledge to recognize terms when offered, but less knowledge than is needed to generate terms from memory.

Browsing. Browsing is an interactive process of skimming over information and selecting choices. Browsing relies on recognition knowledge and requires less well-defined search objectives than does directed keyword searching. Children prefer browsing to keyword Boolean-based searching and can use the browsing tech- nique effectively in both OPACs (Armstrong & Costa, 1983) and full-text searching of electronic encyclopedias (Liebscher & Marchionini, 1988; Marchionini, 1987).

Hierarchies. Large databases must be organized in some rational order for search and display. In most standard keyword systems. search tasks and displays are organized by data element (usually author, title, or subject), with result sets displayed as alphabetical lists. Such display lists often are intolerably long and difficult to navigate, especially for children (Edmonds et al., 1990). Adults typically will browse only about 30 items before abandoning a search (Wiberley, Daugherty, & Danow- ski, 1990). Large result sets can be limited with Boolean operators, but these introduce their own problems.

Hierarchies are an attractive alternative for organizing databases, particularly for children. Children can utilize hierarchies for concepts they understand (Keil. 1979), and are able to make increasing use of hierarchies and categorization in the later elementary school grades. We explored children’s abilities to organize science topics into hierarchies early in the Science Library Catalog development process (Borgman et al., 1989b), and found that children could organize science topics, and that their ability to do so increased with age. These abilities, however, were highly dependent on their knowledge of the topics being organized and were too idiosyncratic to the individual child for the results to be useful in organizing the catalog database. Pejtersen, working in Denmark, did a large-scale study of children’s and adults’ categorization of fiction topics and was able to use the results to

organize an exploratory, browsing-oriented online catalog of fiction for children and adults, known as the Book House (Pejtersen, 1986, 1992). Others have shown that systems such as the Dewey Decimal Classification and the Universal Decimal Classification can provide effective browsing structures and retrieval mechanisms for online catalogs (Freeman & Atherton, 1968a, 1968b; Markey, 1987; Markey & Demeyer, 1986).

Summary. While elementary school children may not be able to use standard online catalogs effectively, it should be possible to design effective systems that build upon the capabilities and knowledge that children do have at this age. Specifically, children may be able to recognize information that they can not recall from memory, they can browse for information, and they can utilize hierarchies that organize information and provide a context for it. Following Kay’s (199 1) axiom that the measure of any computer system is whether it can be used by children, these design principles should be generally applicable to information retrieval systems.

The Science Library Catalog

System Description

The Science Library Catalog is based on the premises outlined above, that a hierarchical, browsing-oriented online catalog relying heavily on recognition knowledge is better tailored to the abilities and skills of elementary school children than are standard keyword, Boolean approaches. We have constructed a series of versions of the Science Library Catalog, making iterative improvements based on interim experimental results. We report here on the results of experiments with four versions of the SLC developed over a period of about 4 years. In all versions, the core of the system is a hierarchy of science categories drawn from the Dewey Decimal Classification, the standard classification system used in public and school libraries.

Items classified using the Dewey system are given a numerical call number that we are treating as a sequence of decimal digits, as opposed to a single numerical “address.” Each digit position within a call number repre- sents a lo-way subcategorization of the subject repre- sented by its preceding digit. The SLC versions vary by navigation features, quality (aesthetics and human factors features) of the screen displays, size of database, and depth of hierarchy. Each was tailored to the database and physical location of the experimental site.

In all versions, the Dewey-based hierarchy is presented as a bookshelf metaphor. The interface metaphor was designed to correspond to children’s mental model of a library catalog and seems to facilitate their retrieval


FIG. I Bookshelf screen, Science Library Catalog, Version 1.

strategies (Walter, in press). In Version 1 (Fig. 1) the shelf layering is shown as a list at the bottom of the screen. In all subsequent versions, which were drawn by a children’s book illustrator, each successive level of the hierarchy cascades into a bookshelf containing the subcate- gories of the previous topic (Fig. 2, from Version 3.0). The bookshelves cascade to the right, providing the sub- categories while highlighting the shelf of the category selected (Science in this example). The hierarchy is four levels deep (i.e., bookshelves at the levels of 500, 510, 5 11, 5 11.1) in the first three versions, which contained 250, 1,150, and 1,500 records respectively, and six levels deep in Version 4, which contained 8,200 records (i.e., tothelevelof511.111).

To browse, the child moves through the bookshelves, selecting categories by pointing and clicking with the mouse. The SLC is implemented without a keyboard; the mouse is the only input device. To move up the hierarchy, the child clicks on the prior bookshelf exposed to the left. At the lowest level of each branch in the hierarchy are the book records. Children select books by clicking on a book title. Figure 3 (Version 1) shows our first cluttered attempt; Figure 4 (Version 3) shows the im-

FIG. 2. Bookshelf screen, Science Library Catalog, Version 3.

We have these books on Learning about Science

They areqp the shelf at 507 Click on the name of a book to see more about it.

FIG. 3. Booklist screen, Science Library Catalog, Version I.

proved format. In Version 3 we added the numbered tabs that allow movement across the hierarchy, a more powerful navigation feature that was removed from subsequent versions, for reasons explained in the Results and Discussion sections. In Version 4, we also replaced the standard scroll bar with block arrows that were more clear to children.

Clicking on any item in the book list displays the record for the selected book. In Version 1 (Fig. 5) we presented a block-structured record; in later versions (Fig. 6, Version 3) we formatted the record to look like a book, with fields placed appropriately on the title page, verso, and following page. Version 3 added the “next book” feature to move laterally along the bookshelf; this feature also was removed from later versions. By clicking on the “see map” (Fig. 5, Version 1) or the “library map” (Fig. 6, Version 3) the child is given a map of the physical library (Fig. 7), oriented from the location of the computer, with an arrow that walks to the correct bookshelf for this item and flashes the spot. In Version 4 (Fig. 8), we placed the book description on the map screen as well, as a reminder of what book was in the flashing location.

We loaded MARC (MAchine Readable Cataloging)

This is a list of books about

mha crocodiles still wait

Dlgglng up dlnorsurs

Dinosaur bones

FIG. 4. Booklist screen, Science Library Catalog, Version 3.


You can find this book on the shelves at: 507 H Author: Herbert, Don.

r-l

ISI

DonHerbert;illostratedby Peg McKie.

LA w Edition:

Publisher Info: NewYork. RW&~~HOU~. 1980.

Size. Pictures: 96 p 01.. 28 cm. Notes: II~cI~~~s index.

I I”,

Other Authors: 1 u

FIG. 5. Book record display screen. Science Library Catalog, Venion 1.

book records directly into the system, where available, placing them in the hierarchy based on their Dewey classification. We named the bookshelves using the Dewey schedule terms matching the classification number. However, we found that many terms were inappropriate for the grade levels of our target audience (grades 4-6; ages lo- 12), such as “dicotyledons.” To select more appropriate terminology, we generated terms from the subject headings and title words in the records and clustered the keywords, selecting the terms most commonly used in the book records for each Dewey category (see Borg- man et al., 199 la; Rosenberg & Borgman, 199 1 and 1992, for more details on the clustering and record-loading processes).

The SLC is intended for use with no prior training or printed reference materials. In all versions of the system, a “bookworm” resides on all screens offering help in the use of the Science Library Catalog. If the system is inac- tive for 5 minutes (no mouse clicks, as registered by our monitoring programs), it defaults into “attract mode,” modeled on video game introductions, where the bookworm explains how to use the system. Any click restarts

Mr. Wizard’s

FIG. 6. Book record display screen, Science Library Catalog, Version 3.

FIG. 7. Library map screen. University Elementary School, Science Library Catalog. Version 3.

the system from attract mode. The attract mode also serves as a screen saver.

Research Questions

Our research questions address issues that will lead to increased understanding of children’s information seeking and information retrieval behavior and to improved systems design:

1. How do children search for topics in an automated library catalog?

2. Are children able to use a hierarchical, browsing, recognition-based system effectively? Is such searching behavior related to age, sex, or computer experience?

3. Are children able to use a keyword, Boolean retrieval system effectively? Is such searching behavior related to age, sex, or computer experience?

4. How does search behavior vary between browsing and keyword systems?

FIG. 8. Library map screen, Los Angeles Public Library, Science Li- brary Catalog. Version 4.


TABLE I. Experiments summary.

Expenment SLC Version B Site Database Size Other OPAC (# of Records)

5. How well do children like to use browsing or keyword retrieval systems?

Research Method

We have taken a formative evaluation (Patterson & Bloch, 1987; Vasek & Volger, 1984) approach to the study of children’s information-seeking behavior, as too little is known to formulate specific hypotheses. We are relying on approaches common in human-computer interaction research, to construct a prototype system based on our theories of children’s information retrieval behavior, test the children on the system, then refine both the system and our theories.

The studies reported here gathered several types of data: children’s performance on controlled experimental tasks in a field setting; interviews with children as part of field experiments; online monitoring of search tasks, both experimental and unobserved tasks in daily usage; and focus group interviews.

Research Sites

We report here on behavioral data from four versions of the Science Library Catalog and two school settings (University Elementary School and Open School) and a major public library (the Central Library of the Los An- geles Public Library) over a 3-year period. Table 1 sum- marizes the experiments by research site and systems studied.

The University Elementary School (UES) is the only laboratory school in California and is a heavily-utilized research site. The school has excellent facilities, including many classrooms equipped with computers and library resources (I 8,000 volumes) far superior to those of public elementary schools in California. We loaded 1,150 MARC records from this library for Version 1 of the SLC (Experiment 1) and 1,500 records for Version 3 (Experiment 3).

The Open School (OS) has Macintosh computers in all its classrooms, and is widely known for its participa- tion in the Apple Vivarium project (Kay, 199 1). The OS has a small library comparable to other Los Angeles Uni- fied School District (LAUSD) elementary schools, and is staffed intermittently by parent volunteers. We loaded 250 non-MARC, sparsely cataloged records from the Open School for Version 2 (Experiment 2).

Both of these schools have much greater computing resources than do typical Los Angeles public schools, although computers at UES are concentrated in the upper division (grades 4-6) classrooms. Both schools reflect the ethnic diversity of California children, with a mix of An- glo, Latino, African-American, and Asian-American students. These schools are excellent alpha sites to pursue these research questions in resource-rich environments.

The Central Library Children’s Room of the Los An- geles Public Library (LAPL) draws elementary school students from resource-poor environments. While the Central Library is intended to serve all of the children of Los Angeles, its immediate service population is approx- imately 4,000 children in nearby inner-city schools, who are 90% Latino, 8% African-American, and 2% Anglo. LAPL has an extensive collection of children’s science books; we loaded 8,200 MARC records from their database for Version 4 (Experiment 4).

The databases for each site consisted of all the science and technology records (Dewey classes 500-699) available in computer readable form. We drew location maps specific to each site. The Science Library Catalog is implemented in HyperCard on Macintosh Classic and Macintosh SE computers, which were the most widely- used Macintoshes in schools at the time of the experiments.

Samples

Samples were balanced by age and sex to the extent possible, drawing randomly from multiple classrooms in each age range. We chose to study children aged 9- 12 as this age range overlaps Piaget’s concrete-operational (age 7- 11) and formal-operational (age 11 -adulthood) developmental stages (Piaget & Inhelder, 1969), and we pre- dicted performance differences by age. UES and OS are organized into classrooms that overlap traditional grades, so we grouped subjects by age, based on the stu- dent’s birthday during the academic year studied; i.e., a child turning 10 that year was grouped in the “age 10” category, which would correspond to 4th grade, whether or not he or she had as yet reached the 10th birthday. At LAPL, children in each age range were randomly selected from visiting groups of students. The size of the samples in each experiment ranged from 32 to 34 children.

Field Experiments

Each child was taken individually from the classroom at UES or OS to the school library by an experimenter. At the Los Angeles Public Library, the child was selected from a class visiting the library. The child was seated at the computer and instructed to play with the program for a few minutes to become familiar with it. No explicit instruction was offered, although we did answer any


questions the child may have had. After about 3 minutes of practice, the child was given search topics one at a time. In Experiments 1 and 2, children searched six topics on the SLC; in Experiments 3 and 4 they searched four topics on the SLC and four topics on a keyword Boolean system for comparison. The design of Experi- ments 3 and 4 balanced the order of the questions, order of use of the two systems, and the two question sets on each of the two systems.

All search questions were drawn from teacher-com- piled lists of current science curricula in grades 4 through 6. Search topics varied slightly between the research sites due to differences in class topics and records in the databases. Topics for LAPL were adapted from those used in Experiments l-3. Topics were balanced by science (Dewey classes 500-599) vs. technology (Dewey classes 600-699); in Experiments 1 and 2, topics also were balanced by depth of hierarchy. In Experiments 1-3, all topics were defined by the experimenters. In Experiment 4, one topic in each set was selected by the child, within a science or technology topic area.

Children were allowed to search as long as they wished, being told at the beginning, “Please tell me when you’ve found a book on the topic. If you can’t find a book, tell me, and we’ll go on to the next search.” After the child had attempted all the searches, we asked questions to assess his or her understanding of the interface, opinions about the program, previous use of computers for various activities, and attitude toward using the system.

Online Monitoring

The SLC includes an online monitoring program to collect mouse-click sequences and time stamps both during execution of controlled, observed tasks, and during unsupervised use. We report monitoring results here for basic performance data only. We are using the monitoring data to study details of behavior and search patterns both during experimental sessions and at such times the system is in general library use. These data will be reported elsewhere.

Focus Groups

After the one-on-one interviews were completed in Experiment 3 at UES, we held focus group interviews with the same population of children who had participated in the one-on-one experiments.

Expert Critiques

Throughout the development process we have re- viewed the software with experts in child development, education, instructional technology, and interface design, and gathered similar critiques from school and children’s librarians and science teachers. These critiques

have been useful in interpreting the experimental results and in making interface design and experimental design changes.

Independent Variables

As indicated in the research questions, we are pursu- ing issues related to the effect of age, sex, and computer experience on the ability to use browsing and keyword catalog retrieval systems. We have also controlled for search topic type (science vs. technology), search topic set (in Experiments 3 and 4 that used two topic sets), and order effects for system use (in Experiments 3 and 4). Computer experience was measured by whether the child was in a classroom that used computers in instruction (this varied by experiment), and the response to the question, “Do you have a computer you can use outside of school?” We preferred this phrasing to “at home,” because children sometimes had regular access to a computer at another frequently visited site, such as after- school care or the home of a non-resident parent. An- swers were categorized by whether they had access to a Macintosh (the computer on which the experiment was run), another kind of computer (DOS, Apple 11, etc.), or no computer.

Dependent Variables

We studied the effect of the independent variables on search success, search time, and attitude toward the system, as defined below.

Search Success. Search success is defined as the child identifying any book record or book title from a list as matching the search topic. We accepted any item the child took to be a match, rather than requiring that our target record be identified. The alternative to a successful search is an abandoned search, where the child quits without identifying a matching record. Success was measured for one trial on each search topic; children were not allowed to return to abandoned topics after other searches.

Search Time. The time spent on a Science Library Catalog search is captured from the monitoring data and is measured from the opening of the top-level bookshelf (science vs. technology) to the opening of the last screen selected. This measure slightly underestimates the search time, since it does not include decision time to identify a match among the list of books or to choose to abandon the search. Timing of searches on the keyword system in Experiments 3 and 4 was done manually using digital watches, attempting to capture the time from first command to last screen selected; the manual method is inherently less accurate than is the automatic time stamp


TABLE 2. Access to computer outside of school. TABLE 3. Experiment I: Search success by age and sex, number of subjects, number of successful searches out of six.

Experiment Macintosh Non-Mac NO Computer NOI TOtal Computer Answered

method. Search time is captured as an indicator of time spent browsing or exploring and is not intended as a performance measure per se.

Attitude Measures. At the end of each experiment, children were asked how well they liked using the Sci- ence Library Catalog and, in Experiments 3 and 4, a comparison catalog. Children were given a three-point Likert scale with 1 = “don’t like it” and 3 = “like using it” in Experiment 1 and a five-point scale with the same end points in Experiments 2-4.

Experimental Results

We report the results of four experiments and the methods specific to each. Experiment 1 on Version 1 of the Science Library Catalog at the University Elemen- tary School established basic system features and basic performance measures on a database of 1,150 records. Version 2 implemented the features in a far more attractive interface and a smaller database (250 records) at the Open School, tested in Experiment 2 (partial results of Experiments 1 and 2 are reported in Borgman et al., 1990a). Experiment 3 at UES added non-hierarchical navigation features to the SLC (Version 3) and also tested searching the same topics on Orion, a keyword system (partial results reported in Borgman et al., 1991a). Experiment 4 at LAPL tested a much larger database (8,200 records) that necessitated a deeper hierarchy (six levels instead of four) for the Science Library Catalog (Version 4) and tested searching the same topics on LePac, a CD-ROM keyword system.

We have treated the variable sex as a nominal variable and the variables for age and search success as ordinal variables. Search times are treated as interval level variables. Most of the distributions are skewed, so we report both parametric and non-parametric statistical tests as appropriate.

Experiment I: SLC Version I at University Elementary School

System and Site. The first version of the Science Li- brary Catalog was implemented in late 1989 at UES with the 1,150 science and technology records available in

I I I I TOTAL n subjects 15 14 34 median (qu) 4 (4.5) 4.5 (4. 5) 5 (4. 5) mean (sd) 4.3 (.98) 4.6 (.75) 4.7 (.95)

Medians are presented with lower (25%) and upper (75%) quartiles, respectively, in parentheses.

MARC format (Rosenberg & Borgman, 199 1). All children had access to the SLC in the school library.

Sample. Thirty-four boys and girls aged 10, 11, and 12 participated in this experiment. All students in Exper- iment 1 were drawn from classrooms utilizing Macin- tosh computers. In Experiments 1,2, and 3, the majority of children had access to computers outside of school while children in Experiment 4 did not (Table 2).

Experimental Session. Children were given time to play on the SLC (about 3-5 minutes), then were given six search topics (three science and three technology topics), one at a time, in the same sequence. Children were given the topic only, not the designation. Questions assessing computer experience, knowledge of the system, and attitude were asked at the end of the session. The entire session lasted 25 to 30 minutes for each child.

Results. Search success. The mean success rate was 4.7 of 6

searches (sd = .95); the median was 5 of 6 searches, with quartiles of 4 and 5 searches; i.e., 75% of the subjects found matches for at least 4 of 6 topics. Table 3 provides a breakdown by age and sex. We found a significant difference in search success by age (p = 0.007, F = 5.877, df = 2, n = 34), with older children finding matches for more topics, but no difference by sex or access to a computer outside of school (Table 4). Children matched more science topics (93 of 102 searches) than technology topics (66 of 102) (p = .OOl, t = 3.7, df = 33, n = 34), as shown in Tables 4 and 5.

Search time. Data on search success and attitude were

TABLE 4. Successful searches by experiment.


TABLE 5. Success by search topic: Experiment I-UES. success Topic 1 Topic 2 Topic 3 Topic 4 Topic 6 Topic 6

(S) (T) (1) (3) (S) m

Search topics (S = science, T = technology): Topic I (S) = chemistry; topic 2 (T) = how our bodies work; topic 3 (T) = how rubber bands are made; topic 4 (S) = the planet Jupiter; topic 5 (S) = kangaroos; topic 6 (T) = fire trucks.

collected manually for all 34 subjects, but online monitoring data were captured for only 23 subjects, due to a system failure. We lost monitoring data in roughly equal proportions from each age group and sex. The results are very similar for the variables on which we have complete data, and hence we believe that the monitoring results are a valid representation of the distribution over the 34 UES subjects. Search time for all experiments is given in Table 6. Because these are skewed distributions, we report both medians and means. Children spent an average of 100.9 seconds (sd = 50.7) on each search, spending significantly more time on searches they ultimately abandoned ( 169.3 seconds, sd = 117), than on searches they completed successfully (79.9 seconds, sd = 32.5) (p = 0.002, t = -3.7, df = 16, n = 17) as shown in Table 7. We found no significant differences in search time by age, but girls took longer on successful searches (p = 0.029, t = 2.35, df = 21, n = 23). Children spent more time overall on technology topics than on science topics (p = .OOl, t = -4.0, df = 22, n = 23).

Attitude measures. Children’s median attitude toward the SLC was 3 on a scale of 3, with quartiles of 2 and 3, and with no difference by age or sex. Children’s mean attitude toward the SLC was 2.6 (sd = .66).

Experiment 2: SLC Version 2 at Open School

System and Site. The second version of the Science Library Catalog was implemented with the 250 science and technology records available in computer-readable format from the Open School. The features were sub-

TABLE 6. Search time by experiment.

TABLE 7. Successful vs. abandoned time by experiment.

stantially the same, but the interface was far more attractive and less cluttered. Based on the results of Experi- ment 1, in which children appeared not to know the term “technology,” we renamed that category as “People Us- ing Science.”

Sample. Thirty-two boys and girls aged 9, 10, and 11 participated in this experiment; an additional four children. aged 8, were excluded from these analyses, as they were too small a set to balance with the other age groups. All children were drawn from classrooms utilizing Macintosh computers; most had access to computers outside of school as well (Table 2).

Experimental Session. Children were given time to play on the SLC (about 3-5 minutes), then were given six topics, one at a time, in the same sequence. Questions assessing computer experience, knowledge of the system, and attitude were asked at the end of the session. The entire session lasted 25 to 30 minutes for each child.

Results. Search success. The mean success rate was 4.9 of 6

searches (sd = .73); the median was 5 of 6 searches, with quartiles of 4 and 5 searches; i.e., 75% of the subjects found at least 4 of 6 topics. Table 8 gives breakdowns by age and sex. We found no significant differences in search success by age, sex, access to a computer outside of school, or science vs. technology topics (renamed Peo- ple Using Science). Results by search topic are given in Table 9.

Search time. Children spent an average of 98.7 seconds (sd = 33.8) on each search, spending significantly

Time Exp. 1 - UES Exp. 2. OS k34 n=32

Exp. 3 SLC n=33

Exp 3 -0r1on “=33

OVt?A (lk23) median (qu) 66.5 (66. 126) 67.3 (76, 122) 136.2 (104, 191) 63.6 (59. 131) mean (sd) 100.9 (50.7) 96.7 (33 8) 154.6 (71 2) 92.6 (46.3)

Exp. 4 - SLC n=32

Exp 4. LePac n=32

166 (130. 247) 1306 (95, 186) 184.5 (76.9) 142.6 (61 1)

Medians are presented with lower (25%) and upper (75%) quartiles, respectively, in parentheses. Note n’s are different sizes. In Experiment I, this is due to a failure in the monitoring data collection program. In other cases, it is due to the number of subjects who performed all searches successfully or who abandoned all searches.


TABLE 8. Experiment 2: Search success by age and sex, number of subjects. number of successful searches out of six.

1 I I I I I

Medians are presented with lower (25%) and upper (75%) quartiles, respectively, in parentheses.

more time on searches they ultimately abandoned (183.7 seconds, sd = 114.6) than on searches they completed successfully (79.5 seconds, sd = 27.6) (p < .OOl, t = -5.01, df = 24, n = 25) as shown in Tables 6 and 7. We found no significant differences in search time by age, sex, computer access, or science vs. technology topics (Table 4).

Attitude measures. Children’s median attitude toward the SLC was 5 on a scale of 5, with quartiles of 4 and 5, and with no difference by age; but boys liked the SLC better than did girls (p = .034, t = 2.224, df = 30, n = 32). Children’s mean attitude toward the SLC was 4.5 (sd = .67).

Experiment 3: SLC Version 3 and Orion at UES

Systems and Site. The third version of the Science Library Catalog was implemented at UES in the summer of 1990 with an updated database of 1,500 MARC records. We added two navigation features that allowed searching laterally along bookshelves (the “next book” feature) and laterally across Dewey categories (the tabs shown in Fig. 4). They allowed continuous searching across the database, well beyond the category in which the search was entered.

Orion is the keyword, Boolean online catalog system that is in general use at UCLA, and is installed at UES on a dedicated terminal in the library. Orion is a command- driven system similar to most OPACs available at the time.

Sample. Thirty-three boys and girls aged 10 and 12 participated in this experiment. The age 10 children were drawn from classrooms that did not have computers, while the age 12 children had extensive classroom Mac- intosh computer use. All children had regular access to Orion in the school library.

Method. Making comparisons between the SLC and a keyword OPAC required several changes in form of the experiment. Although many children at UES had experience with Orion, for consistency each child was given

brief training on the system and given a reference sheet for the title and subject search commands needed for the experiment. No comparable verbal or printed instruc- tions were given for the Science Library Catalog. As in the previous experiments, the children were allowed about 3 minutes of practice on each system before being asked to perform the experimental searches. Each child was given a total of eight search tasks, four each on the Science Library Catalog and on Orion. Half the children used Orion first; half used the SLC first. Eighteen children performed Set A on the Science Library Catalog and Set B on Orion, and 15 performed Set A on Orion and Set B on the Science Library Catalog. Search topics included terms that were direct matches on Orion title or subject fields in children’s books. Single terms matched Orion records; no Boolean combinations were required. At the end of each session, children were interviewed for their opinions and knowledge about that system. The entire session lasted about 30 to 45 minutes for each child. Focus group interviews were conducted with a subset of the sample during several weeks following the experiment.

Results. Search success. Overall success rates were the same on

the Science Library Catalog and Orion, with a mean of 2.8 (sd = .99) on the Science Library Catalog and 2.8 (sd = 1.15) on Orion and median of 3 of 4 tasks successfully completed on each system and quartiles of 2 and 4 on each system (Tables 10 and 11). We found a significant difference in search success by age (p = .014, t = 2.6, df = 31, n = 33) on Orion, with older children finding matches for more topics. Children also had more difficulty with Set A than Set B on Orion (p = .022, t = 2.408, df = 3 1, n = 33), as shown in Table 12. We found no differences in search success by age, sex, computer experience, or question set on the Science Library Catalog. The order in which they searched each system and the order of the question set did not affect success on either system.

Search time. Children were significantly faster overall on Orion (median = 83.8, mean = 92.8, sd = 46.3) than on this version of the Science Library Catalog (median = 136.2, mean = 154.6, sd = 7 1.2); differences in times for successful (p = .005, t = 3.049, df = 30, n = 31) and abandoned (p = .023, t = 2.508, df = 17, n = 18) topics

TABLE 9. Success by search topic: Experiment 2-0s.

Success Topic 1 TopIe 2 Topic 3 Toplc 4 Topic 5 Topic 6 (S) C-U (T) (S) (S) U)

Search topics (S = science, T = technology): Topic 1 (S) = biology; topic 2 (T) = health; topic 3 (T) = farming; topic 4 (S) = birds; topic 5 (S) = weather; topic 6 (T) = firefighting.


TABLE 10. Experiment 3: Search success by age and sex, SLC, number of subjects, number of successful searches out of four.

I Age 10 Age 12 Overall

Medians are presented with lower (25%) and upper (75%) quartiles, respectively, in parentheses.

also were significant (Table 6). The time difference was significant between successful and abandoned searches on Orion and on the Science Library Catalog individually, as shown in Table 7.

Older children were significantly faster on overall search times for Orion (p = .002, t = -3.466, df = 31, n = 33), and on successful searches (p = .O 14, t = -2.628, df = 29, n = 31), but not on abandoned search times (Table 6). We found no other time differences on either system by sex, age, science vs. technology topics, or question set. Age and computer experience did appear to have an interaction effect, given that these IO-year-olds did not have classroom computers, but the 12-year-olds did. We found that IO-year-olds with access to Macin- tosh computers outside of school were faster on successful searches on the SLC than those with no computer or a computer other than a Macintosh (p = .008, F = 5.676, df=2,n=32).

Attitude measures. Children’s median attitude toward both systems was 4 on a scale of 5, with quartiles of 3 and 5, and with no difference by sex; but younger children gave the Science Library Catalog higher scores than did older children (p = .028, t = -2.307, df = 3 1, n = 33). Children’s mean attitude toward the SLC was 3.9 (sd = .93) and toward Orion was 3.7 (sd = 1.08).

TABLE I I. Experiment 3: Search success by age and sex, Orion, number of subjects, number of successful searches out of four.

Age 10 Age 12 OVerall

-iOTAL ” subjects median (qu) mean (sd)

17 16 33 2f1.3) 3 (3. 4) 3 (2.4) 2.3 (1 36) 3 2 (.58] 2.6 (1.15)


TABLE 12. Success by search topic: Experiment 3-UES.

Set A I Set a success :opp Topjc 2 Topic 3 Top; 4 Topic Topic 2 Topic 3 Topic 4

( ) ( ) (T) ( ) 1 U) (S) (S) CT) Abandoned 2 9 9 6 1 4 6 3 Successiul 16 9 9 12 14 11 9 12 ” 1.9 18 16 16 15 15 15 15

OrIon

Search topics (S = science. T = technology): Set A: Topic I (S) = fossils; topic 2 (S) = the planet Jupiter; topic 3 (T) = (human) feet; topic 4 (T) = how rubber bands are made. Set B: Topic 1 (T) = farming; topic 2 (S) = kangaroos; topic 3 (S) = California desert; topic 4 (T) = how your body works.

Focus groups. We held focus group interviews with the same population of children who had participated in the one-on-one experiments. These groups were designed to parallel the sample of children in the experiment and were split by age and sex, with eight children in each of four separate groups: IO-year-old girls, lo- year-old boys, 12-year-old girls, 12-year-old boys. There was more distortion than is usual in focus group data because of the children’s familiarity with each other. However, content analysis and successive data reduction did result in some findings that were consistent over the four focus group sessions, a rule-of-thumb for reliability in focus group research (Krueger, 1988).

Questioning concentrated on subjective responses to the Science Library Catalog and Orion: preferences, likes and dislikes, perceptions of difficulties using the two systems, attitudes about libraries and library information retrieval systems in general. The focus group facilitator, who also analyzed the data from the focus groups, had not been involved in the experimental testing and thus was free from bias arising from previous knowledge of the children’s performance.

Results of the focus group interviews can be summa- rized as follows:

Children prefer not to use any intermediary finding tools when they use libraries. Their first choice is to browse within a narrow range of shelves where they have successfully found books in the past; their second choice is to ask a librar- ian for help. Catalogs of any design run a poor third. The younger children, the lo-year-olds, prefer the Science Library Catalog to Orion, primarily because of its graphics and its non-reliance on correct spelling and keyboard skills. Some of these same children had difficulty manipulating the mouse, however. Most of the older children, the 12-year-olds, were quite skilled with Orion and preferred it to the


Science Library Catalog. They liked the multiple points of access on Orion-author, title, subject-and appreciated its direct route to a subject, as opposed to the multiple screens and metaphor- ical design of the Science Library Catalog.

l While age was a factor in preference for one system over another, sex was not.

Experiment 4: SLC Version 4 and LePac at LAPL

Systems and Site. The fourth version of the Science Library Catalog was implemented at the Los Angeles Public Library in the spring of 199 1 with a database of 8,200 MARC records and several changes in the interface and features. The catalog data contained large clusters of records (often 200 or more) in individual Dewey classes, due to the common historical practice of truncating class numbers for children’s books. Because the data would not sort neatly into the SLC hierarchy, we clustered the records into manageable categories within the Dewey framework, using keywords in titles and subject headings (Rosenberg & Borgman, 1992). The larger database necessitated six hierarchical layers rather than four in the previous versions. We removed the two navigation features that allowed searching laterally along bookshelves (the “next book” feature) and laterally across Dewey categories (the tabs shown in Fig. 4) that were added in Version 3, returning to the functionality of the earlier versions.

LePac is the keyword, Boolean CD-ROM catalog system that is in general use at LAPL. It is a menu and form fill-in system similar to other OPACs available at the time.

Sample. Thirty-two boys and girls ages 9 through 13 participated in this experiment. These children were drawn from two groups that were bused to the Children’s Room of the Central Library: (1) children from inner- city schools, and (2) children from an inner-city after- school program. These children did not have regular usage of computers in school and only six children had access to computers outside of school (Table 2). They were representative of the inner-city population and English was the second language for many of them. To balance our sample by age, we grouped one 9-year-old and a child just turned 11 with the lo-year-olds; one 13-year-old and a child almost 12 were grouped with the 12-year-olds.

Method. We followed the same procedures for comparing the two systems as with Experiment 3, giving children instruction on LePac but not on the Science Library Catalog, and balancing the question sets and system order. The last question in each set gave them an opportu- nity to choose a specific search topic. We also introduced two questions expected to be difficult to spell (“tyrannosaurus” for science and “veterinarians” for technology)

TABLE 13. Experiment 4: Search success by age and sex, SLC, number of subjects, number of successful searches out of four.

I Age 10 Age 12 OVerall

GIRLS n subjects median (qu) mean (sd)

16 z (2 4) 2.7 i1.03)

2 (2. 4) 2.5 (1.38) 2.5 (1.21)

I I I I I


to balance the advantages of the keyword LePac and the SLC browsing systems. Eighteen children performed Set A on the SLC and Set B on LePac, and 14 performed Set A on LePac and Set Bon the SLC. Search topics included terms that were direct matches on LePac title or subject fields in children’s books. At the end of each session, children were interviewed for their opinions and knowledge about that system. The entire session lasted about 30 to 45 minutes for each child.

Results. Search success. Search success was similar in the two

systems. Children found a mean of 2.4 (sd = 1.16) and a median of 2.5 of 4 searches on the Science Library Cata- log with quartiles of 2 and 3 (Table 13), and a mean of 2.3 (sd = 1.25) and a median of 2 of 4 searches on LePac with quartiles of 1 and 3 (Table 14). We found no significant differences in search success by age, sex, computer access, question set, or sequence of system use on either system. However, children found more of the science topics than the technology (People Using Science) topics on the Science Library Catalog (p = .003, t = 3.215, df = 31, n = 32); and more of the Set B than the Set A topics on LePac (p = .046, t = 2.08 1, df = 30, n = 32), as shown in Table 15.

Search time. Children were faster overall on LePac than on this version of the Science Library Catalog (Ta- ble 6) (p = ,005, t = 3.012, df = 3 1, n = 32) and significantly faster on abandoned searches (p = .044, t = 2.134, df = 23, n = 24), but not on successful searches. These data are computed for 3 1 subjects on the SLC, as we lost data for one subject due to a system failure. The time difference between successful and abandoned searches was significant on the Science Library Catalog but not on LePac (Table 7). On these paired tests, children with one of the values missing were eliminated. We found no significant differences in search time on either system by age, sex, science vs. technology topic, question set, order of system, or access to computers.

Attitude measures. Children’s median attitude toward the SLC was 5 on a scale of 5, with quartiles of 4 and 5, and toward LePac a median rating of 4 and quartiles of


TABLE 14. Experiment 4: Search success by age and sex. LePac, number of subjects. number of successful searches out of four.

Age 10 I Age 12 OVeEtll

GIRLS n subjects median (qu) mean (sd)

SOYS n subjects median (qu) mean (sd)

TOTAL n subjects median (qu) mea” (sd)

6 1.5 (1, 3)

i 16 (2. 41 2 (1. 3)

1.6 (1.26) 2 4 (1 51) 19 (1.44)

14 13 32 2 (1, 3) 3 (2, 3) 2(1.3) 2.1 (1.23) 2 5 (1 27) 2.3 (1.25)


4 and 5. Children’s mean attitude toward the SLC was 4.5 (sd = .92), and toward LePac was 4.2 (sd = .93). We found no difference in attitude by age or sex for either system.

Discussion

We have presented the results of four experiments conducted on four versions of the Science Library Cata- log, two of which included comparisons to two different keyword, Boolean online catalogs. The experiments were conducted at three different sites. with four databases (the database at UES was expanded from Experi- ment 1 to Experiment 3). We tailored several features of the interface to the local environment, including category labels for the database and the map of the library. Our subjects were children aged 9 through 12 (with slight category adjustments, as noted) from both computing resource-rich and resource-poor schools, with a range of prior computing experience, and representing the full range of ethnic and cultural diversity of children in Southern California. Despite the small samples in each experiment (32 to 34 subjects, split by age and sex) and small range of tasks (4 to 6 per system per child), we found a number of significant results. Taken as a whole, the set of experiments shows several interesting trends in children’s searching behavior on both the Science Li- brary Catalog and on keyword OPAC systems.

We discuss the results within the context of the research questions presented earlier.

1. How Do Children Search for Topics in an Automated Library Catalog?

Overall, children were successful in finding topics on both systems, locating book records for a median of 5 of 6 topics in Experiments 1 and 2, a median of 3 of 4 on both Orion and the Science Library Catalog in Experi- ment 3, and 2 of 4 on LePac and 2.5 of 4 on SLC in Experiment 4. A successful search was one that ended with a child declaring a match of a record to the re- quested topic; if the child gave up without finding a

match of his or her choice, we considered the search abandoned. With the small number of searches on each system and our generous measurement of success, we appear to have a ceiling effect on search performance. It will require larger numbers of search tasks to identify the true limits of searching ability in these systems. In all cases we find that children are persistent in their searching, making several tries before abandoning a search. The time differences between successful and abandoned searches is significant in 5 of the 6 trials (all four SLC experimental trials and Orion). Children spent from one and one-half to twice as long on abandoned searches as on successful searches (Table 6). The persistence may be an experimental effect, given that the children were being observed, or simply children’s determination at a task. More understanding ofchildren’s search persistence may be revealed by analysis of the monitoring data from the unobserved use of the system in these libraries and from other types of unobtrusive field studies.

Search times per topic were fairly short overall, rang- ing over four experiments on the SLC from medians of about 70 seconds to 120 seconds on successful searches, and means of about 80 to 149 seconds (Table 6). Timing differed between the two keyword systems, with a median of 67 seconds (mean 78) on Orion and a median of 95 seconds (mean 124) on LePac. The mean and median figures are much different due to the long tails on the distribution, with the slowest children spending far more time than the fastest children. Such long tails are expected, with ratios of I : 10 from fastest to slowest searching typical in computer-based information retrieval tasks (Borgman, 1989; Egan, 1988).

The structure of the search task in these experiments (“Can you find a book about XX?“) probably did not encourage browsing as much as we would have liked. Children varied in their interpretation of the task, with most identifying the first book record they thought matched the topic (our intent), some browsing several matching books before selecting one they liked best, and

TABLE 15. Success by search topic: Experiment 4-LAPL. Science Library Catalog

Search topics (S = science, T = technology): Set A: Topic 1 (T) = bridges: topic 2 (S) = the planet Jupiter; topic 3 (T) = veterinarians; topic 4 (S) = an animal you like. Set B: Topic 1 (S) = rocks; topic 2 (T) = spaceships; topic 3 (S) = tyrannosaurus; topic 4 (T) = how to cook something you like to eat.


a few choosing to follow the map to the shelf, literally “finding a book” before completing the search (in these cases time was counted only to the point of identifying the record in the database). These differences in interpretation contributed to the variation in search time. We are using “story problem” phrasing in subsequent experiments to encourage browsing. Some children appeared to learn something about the context of subjects from their browsing, judging by their interest in returning to previous topics when they encountered an item on a topic they had abandoned earlier.

2. Are Children Able to Use a Hierarchical, Browsing, Recognition-Based System Eflectively? Is Such Searching Behavior Related to Age, Sex, or Computer Experience?

Children appear able to search all versions of the Sci- ence Library Catalog effectively and quickly, with no prior training except for a few minutes of unguided practice. We do see some striking and some subtle differences in searching from one version of the Science Library Cat- alog to the next. Version 2 was a cleaner, more attractive interface than Version 1, with a smaller database (250 vs. 1,150 records, due to differences in the library collec- tions), nearly identical functionality, and with the technology category renamed People Using Science. Differ- ent sets of search topics were used in the two experiments (tailored to the database and science curriculum topics of each school). The significant differences in age and in science vs. technology topics in Experiment I (Version 1) did not appear in Experiment 2 (Version 2), yet the rate of successful searches and the search times are nearly identical between the two experiments. All children in these two experiments used Macintosh computers in their classrooms. These comparative results suggest both ceiling effects on search ability (the questions were easy enough for most of them), and that there may be some minimum search time no matter what size the database.

Children performed so well in Versions 1 and 2 that in Version 3 we added powerful navigation features that let children browse laterally through the database, both along the bookshelves (“next book”) and along the class numbers (tabs in the book list), as shown in Figures 4 and 6. These features allowed children to continue browsing beyond the boundaries of the category in which they entered the book list or book record, perturbing the hierarchical metaphor. In observing the children using this feature, they appeared to get lost in the database, which is a basic problem with hypertext systems (Conklin, 1987). The system features and database size were otherwise about the same as in the first experiment ( 1,500 records vs. 1,150), yet children required far longer to search (median 118, mean 132 on successful searches; Table 6) than on the first two experiments, without an increase in the number of successful searches (though it would be difficult to increase performance on only 4 searches).

TABLE 16. Percent of searches abandoned, all experiments.

Children may have learned more about the database in their more extensive browsing, but we did not find a sat- isfactory way to measure such learning. Thus we removed these features in subsequent versions of the SLC, returning to our more pure vision of a hierarchical searching mechanism.

Version 4 tested the limits of the database size and required an increase in the depth of the hierarchy. While the database size increased 5.5-fold (from 1,500 to 8,200 records), the number of nodes in the tree structure increased loo-fold (ten times larger for each of the two additional digits in the Dewey classification), as we went from four levels in the hierarchy to six. Only the shelves containing books actually in the database are labeled; the rest are blank, which simplifies the system considerably. Version 4 was further complicated by the uneven distribution of records. In the smaller databases with more recent and detailed cataloging, the records scattered neatly over the hierarchy, with only a few lists having more than 10 or so book records (e.g., “dinosaurs”), so we main- tained the true Dewey Decimal Classification distribution. In Version 4, with the larger database and uneven cataloging (Los Angeles Public Library has a historical collection, with records and books dating to the 19th century), records were distributed in unmanageable clusters, sometimes of 200 books or more; these books are shelved alphabetically within these large categories. By clustering the records on keywords in titles and subject headings (Rosenberg & Borgman, 1992), we made the database more manageable without the expense of recat- aloging, but it was no longer a pure Dewey hierarchy. Children found about the same number of topics in Ver- sion 4 (median 2.5 of 4) as in Version 3 (median 3 of 4) and in about the same time (Table 6). We cannot deter- mine what portion of the differences in search times in Version 4 might be due to the larger database, deeper hierarchy based on reclustered Dewey categories, slower response time (running a larger database on the same size computer), or the lesser amount of computer experience and probably lower (but untested) reading and science vocabulary skills in the subject population, the children having been drawn from less advantaged schools than those in the earlier experiments.

Where we do see some interesting trends is in success rates over the four versions of the Science Library


Catalog, as shown in Table 16 (derived from Tables 5, 9. 12, and 15). Children had comparable success rates on the first two versions of the SLC, abandoning only 22.0% and 18.2% of the 6 searches, with the significant difference between science and technology topics dis- appearing from the first to the second experiment. Children had more difficulty on Version 3 (30.3% of 4 searches abandoned), with the problematic features, and the most difficulty of all on Version 4 (39.1% of 4 searches abandoned), with the larger database. While children searched different sets of topics in each experiment, the topics were chosen by the same criteria and were comparable, except that Experiment 4 introduced two topics known to be difficult to spell and asked children to generate their own term for one topic in each set. In all cases, at least one match existed in the database. The effect of the individual search topics on search success is discussed further below under research question 4, comparing the SLC to the two keyword systems.

We found minimal effects of age, sex, or computer experience in the Science Library Catalog search results. The age difference in success rate and sex difference in time for successful searches on Version 1 disappeared with improvements in the system. In Exper- iment 3, we did not find an age difference in SLC performance between the older children who used Macintosh computers in the classroom and the younger children who had no classroom computers, yet we found that overall search time (but not other measures) was related to access to computers outside of school. The latter effect is particularly anomalous, given that children who had Macintosh computers at home were fastest, followed by children with no access to computers outside of school, and children with access to IBM-type computers slowest of all. The least computer-experienced children were those in Experi- ment 4, but it is difficult to separate computer experience from other factors.

The Science Library Catalog appears to be reasonably robust to age, sex, and computer experience factors for this age group, especially considering that children received no explicit instruction in the use of the system or the computer.

3. Are Children Able to Use a Ke-vword, Boolean Retrieval System Efectively? Is Such Searching Behavior Related to Age, Sex, or Computer Experience?

Children were able to search both Orion and LePac better than we expected, with success rates and search times comparable to the versions of the Science Library Catalog to which they were compared. In our experiments we did give the keyword OPACs several advantages over both the Science Library Catalog and the usual searching situation by providing specific instruction and a reference sheet only for the keyword systems and by

giving them searchable terms so they did not have to recall terms from memory. Giving them searchable terms also enabled them to find matches on words they could spell, whether or not they understood the concept. We also provided search topics that did not require Boolean logic by carefully selecting single topic terms from children’s books and, in Experiment 3, restricting the search to the UES subset of the database.

The format of the search task “Can you find a book about XX?” was well-suited to the keyword systems, for the child can enter a search command and term and go directly to the records. In the subset of commands used in this experiment, an Orion search consists of a command that includes the search type (FIND), a field tag (SU or TI), the keyword(s), and a limit to the UES collection (/UES). This is a non-trivial search command, but these children (many of whom had used Orion pre- viously in the library) could do it with a reference sheet. Children searched Orion on a terminal linked directly to the host computer and the response time was almost instantaneous. By limiting the search to UES, only small sets were retrieved. If the keyword matched a single record, it was immediately displayed; if it matched multiple records, a list was displayed and one more step was required to select from a short list. LePac requires that the keywords be typed in the proper field in a search form (author, title, subject); children were instructed to use title or subject fields. The system ran on a CD-ROM and was fairly slow.

Children had higher success rates on Orion (median = 3; Table 11) than on LePac (median = 2; Table 14); children abandoned 31.1% of the Orion topics and 43.0% of the LePac topics (Table 16). Some of the difference in success appears to be due to the specific topics searched, as discussed in the next section. Similarly, they were much slower on LePac, with median and mean search times about one and one-half times as long as on Orion (Table 6).

Orion usage was subject to age effects on both search success and search time (overall and successful), with older children being more successful and faster. In this experiment, the older children used Macintosh computers in their classrooms, while younger children had no classroom computers. All UES children had access to Orion in the library at all times, and while we asked about Orion use in our post-experiment surveys, we did not get results that were clear enough to compare to search performance. We learned in the focus group interviews following Experiment 3 that most of the older children were skilled on Orion, as well as having more computer experience. Thus the very fast Orion search time and the age differences probably were due at least partially to familiarity with Orion. Access to computers outside of school did not have any significant effects on Orion usage, nor were there any differences by sex. LePac usage showed no differences by age, sex, or computer experience. The slower search times and lower success rates


between Orion and LePac may be due to differences in familiarity with the systems, differences in computer experience, and differences between the systems themselves.

4. How Does Search Behavior Vary Between Browsing and Keyword Systems?

Search Success and Search Time. When systems were compared within each experiment, we found virtu- ally no difference in search success between the Science Library Catalog and either of the two keyword systems. We did find differences between the systems when comparing across experiments, however.

In comparing the SLC and keyword systems within Experiments 3 and 4, two experimental effects must be considered. One is that the keyword OPACs were given several advantages over the usual library situation, by providing the children with instruction and a reference sheet containing only the relevant subset of commands, using search topics that contained keywords known to match the database, and avoiding the need for Boolean operators. In a non-experimental situation where children have to learn independently from available materials and generate their own terms for a keyword OPAC, searching would be more difficult. Children received no training or reference sheet for the Science Library Cata- log and the search topics tested the full range of system capabilities. The other factor is the ceiling effect-with only four topics to search per system, there is insufficient variance to show differences between systems. Future studies need to test more topics over a longer period of time, and make the situations more similar, for a better comparison.

In making comparisons across the systems and experiments, we see that children excelled in the first two simple versions of the SLC (Table 16), doing less well on the later, more complex versions. Over the four experiments, children abandoned only 25.9% of their search topics on the SLC, lower than on the two keyword systems. As discussed further below, search success varied greatly by search topic, which provides both insights to the search process and guidance for the design of future studies.

Children searched both Orion and LePac significantly faster than the versions of the Science Library Catalog to which they were compared. We note, however, that Or- ion and LePac were compared to the most complex versions of the SLC-Orion to SLC Version 3 with the problematic navigation features that were later removed, and LePac to Version 4 with the much larger database, deeper hierarchy, and slower response time. Search times on Orion and LePac are comparable to search times on the first two versions of the Science Library Catalog; LePac search times appear to be longer than search times on SLC Versions 1 and 2. Thus the search times bear closer inspection.

TABLE 17. Summary of abandoned searches by science search question.

Science Exp. 1 Exp. 2 Exp. 3 Exp. 3 Questions SLC

Exp. 4 Exp. 4 SLC SLC Orion SLC LePac

(3 questns) (3 qusstnsj (2 qusstns) (2 questns) (2 questns) (2 questns)

In Experiments 1 and 2, each child searched three science topics on the SLC. In Experiment 3, each child searched two science topics on the SLC and two science topics on Orion, either from Set A or Set B. In Experiment 4, each child searched two science topics on the SLC and two science topics on LePac, either from Set A or Set B.

Children searched the first two versions of the SLC quickly and effectively, on databases of 250 and 1,150 records, with a strict hierarchical structure drawn directly from the Dewey classification numbers in the records. When we added more features to the system and expanded the database significantly, success rates re- mained fairly stable but search times increased.

The inherently different searching structures of the Science Library Catalog and the keyword systems must be considered in making comparisons of success and time. The SLC is intended to encourage browsing; no keyboard is provided and no mechanism exists to go directly to the record. Several steps (a minimum of three) are required from the opening screen to a booklist. Our goal is to provide powerful searching mechanisms that build on children’s natural tendencies to explore, while avoiding the need for typing skills, correct spelling, vocabulary knowledge, alphabetizing, Boolean logic, or the need for training. The SLC provides subject context for the search, showing where a topic fits in the overall scheme of knowledge, with the intent that children will learn more about the database content with experience, and searching will become both easier and richer. The end of the search is a map of the library, showing where the book is physically located, providing a “real life” context for the system in a graphical display attractive to children. The SLC runs on a slow (by today’s standards) Macintosh with l-2 second response time from one screen to the next. It can provide an effective exploratory environment, but not a fast, direct search.

Keyword OPAC systems enable the searcher to go directly to matching records; they do not provide context, they simply match characters. Thus the task being performed in keyword and browsing systems varies in a subtle


but important way: In the keyword system, the task requiring subject domain knowledge (i.e., identifying subject terms) is external to the system and the search task requires entering those terms with the correct spelling. In the browsing system, the task requiring subject domain knowledge is part of the search task itself, as users must apply what they know about the system to decide where to look first for records. In keyword systems one command may be sufficient to find records if the search terms match only one record; two or more commands may be necessary if selecting from a list. The end point of the search in keyword systems is the call number listed on the screen. To locate the subject context, one must use other browsing mechanisms (such as subject heading or classification displays) or go to the shelf, using information external to the system to identify the physical location. Keyword systems are faster by design, once the keywords to search are known, but do not provide the context or browsing features of the Science Library Catalog. They are also susceptible to the problems of typing skills, correct spelling, vocabulary knowledge, alphabetizing, and Boolean logic noted above, and require more training.

Search Topic Eficts. Success rates on all systems appear to be related more strongly to individual search topics than we had anticipated. We chose search topics based on the children’s current science curriculum to en- sure that these topics would be within their range of knowledge, we balanced science and technology topics, and we selected topics that could be found in the databases of each library. In Experinients 3 and 4, two sets of balanced topics were alternated between the two systems studied. Even so, we found significant differences in

TABLE 18. Summary of abandoned searches by technology search question.

In Experiments I and 2, each child searched three technology topics on the SLC. In Experiment 3, each child searched two technology topics on the SLC and two technology topics on Orion. either from Set A or Set B. In Experiment 4, each child searched two technology topics on the SLC and two technology topics on LePac. either from Set A or Set B.

search success rates between science and technology topics on the SLC in Experiments 1 and 4 and between question sets on both Orion and LePac (Table 4).

Tables 17 and 18 (derived from Tables 5, 9, 12, and 15) show the number of abandoned searches for science and technology categories, respectively. Search topics were duplicated across experiments where possible, based on science curriculum and records in the database. Only a few search tasks appeared to be easy for all children, and these followed Solomon’s (1993) finding that children could search for concrete subjects that were easy to spell; chemistry, birds, health, and farming were the only topics of 21 that consistently had 2 or fewer abandoned searches (each was searched by 14 to 34 children, depending on the experiment). Some topics were consistently difficult: weather, tyrannosaurus, how rubber bands are made, fire trucks/fire fighting, (human) feet, bridges, and veterinarians all were abandoned 6 or more times in each experiment in which they were used. Of these, only “tyrannosaurus” and “veterinarians” were specifically chosen as being difficult to spell. Results varied widely on some topics that were repeated across systems or experiments: the search topic “Jupiter,” for example, was abandoned on the SLC by 8.8%, 50%, and 27.8% of the children in Experiments 1,3, and 4; by 20% of the children on Orion; and by 57.1% of the children on LePac (Table 17). A number of them varied greatly in completion rate between the SLC and the keyword systems. The differences in success by topic appear to be due to domain knowledge (science vs. technology) and to spelling and vocabulary problems.

Science vs. Technolog~~ Topics. We found significant differences in success rates in Experiments 1 and 4 on the SLC between searching for science and technology topics. The significant science/technology difference in Experiment 1 (Table 16) disappeared in Experiment 2 after we renamed the technology category “People Using Science,” but reappeared in Experiment 4. In both cases, the difference appears to be the relative ease of searching science topics more than the difficulty with technology, especially when compared to results on the keyword OPACs (Table 16). Children study more science than technology in elementary school and thus science terminology is more familiar. These differences show up more on the browsing system, where they are searching within the context of the topic and need some knowledge of the domain, than on the keyword systems where they are simply entering an independent term; science and technology topics appear to be about equally difficult on the keyword OPACs (Table 16).

The Dewey Decimal Classification appears to function better as a hierarchy in the science domain, where it is di- vided by discipline, than in technology, where the structure is less clear. The most difficult topic to search on the SLC was “fire trucks” (Experiment 1) or “fire fighting” (Experi- ment 2), which is located under “engineering” and then


under “building for city services.” Fewer than half of all children could locate this topic in the hierarchy, despite their familiarity with fire trucks. We have used this as a test question for adults in demonstrations and find that few adults (including librarians) can easily guess where fire trucks would be classified in Dewey, This finding indicates the importance of studying the effect of classification in designing a hierarchical browsing and retrieval structure- a continuing problem in hypertext systems.

Spelling and Vocabulary. The significant differences in success rate by question set on both Orion and LePac probably are due to variations in spelling and vocabulary difficulty between questions. In Experiment 4, we delib- erately chose one topic that was difficult to spell from science (“tyrannosaurus”) and from technology (“veterinarians”). Children had far more difficulty with “tyrannosaurus” on LePac (66.7% of the children abandoned the search) than on the SLC (42.9% abandoned) (Table 17). We noted that children sometimes abandoned the difficult term given them and searched for “dinosaurs” instead, which we accepted. “Veterinarians” turned out to be an unfamiliar topic, as well as being hard to spell and located in the technology section, so children did poorly with it on both systems (abandoning the search 50% of the time on the SLC and 7 1.4% on LePac) (Table 18). Topics such as “California desert” and “rocks” appeared easier to spell than to browse, with better results on Orion and LePac, respectively, than on the SLC. Con- versely, “kangaroos” was easier to find by browsing in both SLC Versions 1 and 3 than to spell on Orion.

“Jupiter,” in which success varied so widely across systems, appeared to be susceptible to an unfamiliarity with the “astronomy” category where it was located in the SLC (though this varied across the SLC studies), and to spelling problems, though more so in LePac (search abandoned by 57.1% of the children) than in Orion (abandoned by 20% of the children). “Bridges” was difficult in the SLC (55.6% abandoned) due to its location in the engineering classification under technology, and due to spelling in LePac (64.3% abandoned)-the “dg” diphthong does not appear in Spanish and was more difficult for these children to spell than the experimenters had anticipated. Given the trend toward phonetic spelling in elementary school language arts instruction, these examples may be symptomatic of larger problems in keyword information retrieval.

In the two open-ended search topics in Experiment 4, children could choose a familiar topic that they knew how to spell, yet did better on the SLC on both topics. Here is where browsing and recognition knowledge appear to have an advantage over recalling terms from memory and spelling them correctly. All children found “an animal you like” in the SLC, but 4 could not find anything in LePac-our observations suggest that these children selected an obscure animal that was not listed by name in a title or subject heading, or entered the word

in their native language. The most notable example of the latter was the child who said aloud that he would search for “cats,” then typed in “gatos” (the Spanish equivalent). LePac does contain records on non-English language books and a few matches can be made in the title field, but not in the subject field as all records are cataloged in English. Similar vocabulary problems arose with “how to cook something you like.” Children could browse to find something familiar, but they tended to try keywords that were too specific to be searchable, such as “meatloaf’ or “tamales.”

5. How Well Do Children Like to Use Browsing or Keyword Retrieval Systems?

Children gave high marks to all four versions of the Science Library Catalog and to both keyword OPACs. The experimental results showed some minor attitude differences, with boys in Experiment 2 giving the SLC higher marks than the girls, and younger children in Ex- periment 3 giving the SLC higher marks than the older children. In the latter case, the older children had more computer experience than did the younger children. The age differences were confirmed in the focus group interviews conducted after Experiment 3. Younger children preferred the SLC to Orion because of the graphics and lack of need for spelling and keyboard skills, while older children were more skilled on Orion and preferred the familiar direct routes to topics.

Conclusions

Our goal in the design and evaluation of the Science Library Catalog has been to understand children’s information-searching behavior sufficiently to design a system with powerful searching mechanisms that build on children’s natural tendencies to explore, that can be used without prior training, and is within their range of skills and knowledge, while avoiding the need for typing skills, correct spelling, vocabulary knowledge, alphabetizing, and Boolean logic. We have made substantial progress toward that goal by studying multiple versions of the system in different environments and comparing it to two different keyword systems. Given the limited software and hardware environment in which we conducted these experiments (HyperCard versions 1.2.5 and 2.0 on Mac- intosh Classic and SE computers), the Science Library Catalog appears to be most effective in its simpler form, with a pure hierarchical searching structure based on the Dewey Decimal Classification system, with a database small enough to be searched in a four-level hierarchy. We appear to have reached the technological limits of this platform in our experiments and expect that the hierarchical browsing features shown to be effective in the Sci- ence Library Catalog can be scaled up to larger databases, given greater processing power and interface design capabilities.


The Science Library Catalog was searched effectively by children in all of our experiments, and was robust (after improvements on the first version) to differences in age, sex, and computer experience. Children were able to search these databases without prior training and without the need for technical skills required by keyword Boolean systems. Children were able to find some of the topics more easily on the SLC than on the keyword systems, particularly when the topics were difficult to spe]] or more open-ended; thus the SLC overcomes some of the known problems of keyword systems. Children generally had little difficulty navigating the hierarchical structure, finding 5 of 6 (Experiments 1 and 2) 2.5 of 4 (Experiment 3) or 3 of 4 (Experiment 4) search topics. The effectiveness of the Dewey-based hierarchy, as with any organizational structure, is dependent on the clarity of the structure. The Dewey Decimal Classification appears to provide a more effective hierarchical structure for science than for technology topics, a problem further complicated by children’s greater familiarity with science topics. Hierarchical displays in the Science Library Catalog provide context for topics, which appears to as- sist children in applying the vocabulary knowledge they do have and thus may lead to learning more about the subject domain.

Children were able to use the two keyword systems more effectively and more quickly than we anticipated. However, children were more successful and faster on Orion than on LePac, and had lower success rates on both keyword OPACs than on the simpler versions ofthe Science Library Catalog. Further, Orion was subject to differences in age, which were confounded with differences in computer experience and in use of the system. Despite having provided training and reference materials for the keyword OPACs and selecting search tasks that offered exact-match keywords, both of these systems showed performance differences by question sets. Chil- dren abandoned searches more readily on the keyword systems than on the SLC when they were difficult to spell or when they required generating an appropriate search term. We found that the keyword systems afforded more direct access to records, given terms to search, but at the expense of the subject domain context provided by the Science Library Catalog, and required more searching skills and training.

The ideal information retrieval system for children may combine the browsing features of the Science Li- brary Catalog with keyword capabilities that do not require correct spelling, searching alphabetical lists, or using Boolean logic. We are now experimenting with an advanced version of the SLC that embeds keyword searching in the hierarchical browsing system. with spelling correction facilities and rank-ordered results. That research also is studying the effects of children’s science and technology domain knowledge and their ability to manipulate hierarchies (Hirsh, 1995, in press; Hirsh & Borgman, in press). This is only a small start on a rich set

of research questions remaining to be addressed if we are to understand children’s searching behavior sufficiently to support discovery-based learning curricula and teach the next generation of students how to navigate the “information superhighways” of the future.

Acknowledgments

Initial funding for the Science Library Catalog project was provided by the UCLA Academic Senate. Funding for subsequent development of the Science Library Cat- alog and for the experiments reported here has been provided by the Alfred P. Sloan Foundation, with additional provision of equipment by the Vivarium Project of Ap- ple Computer. Project SEED is funded by Apple Com- puter, TRW, the Caltech Presidents Fund, the Pasadena Community Bank, the National Science Foundation, and the Educational Foundation of America. James Bower heads the Project SEED program in the Division of Biology, California Institute of Technology, and is our co-investigator on this project under funding from the Sloan Foundation. Additional support for analysis of the online monitoring data has been provided by the Na- tional Science Foundation and by the Australian Depart- ment of Industry, Technology, and Commerce for joint research with John Hiller of the School of Computer Sci- ence and Engineering, University of New South Wales, Sydney, Australia.

We wish to acknowledge the support and guidance of many people in this project, particularly Judith Kantor and Jenifer Abramson of the Corinne A. Seeds Univer- sity Elementary School Library at UCLA, Roberta Blatt, Principal, Open School, and Janine Goodale and Betty Gay Teoman at the Los Angeles Public Library Chil- dren’s Department.

Many people have assisted us in evaluating the interface and suggesting research directions, for which we are most grateful, including James Bower, Gary Marchi- onini, Ben Shneiderman, Aimee Dorr, Donald Case, Marcia Bates, Karen Markey Drabenstott, Patricia Greenfield, Peter Kovaric, Ann Irving, Brenda Dervin, John Hiller, Scott Kim, Amy Jo Bilson, and our many other project visitors. David Krieger programmed Ver- sion 1 and 2 of the Science Library Catalog, Jason Ro- senberg programmed Version 3, Jason Rosenberg and Paul Biron programmed Version 4, and Erika Oller drew the screen displays for Versions 2, 3, and 4. Paul Biron provided additional project assistance. Jason Rosenberg provided critical commentary on earlier drafts; all errors and interpretations remain the responsibility of the authors.

References

Anderson, J. R. (1990). C~~~nitiveps?rholr,g?,and its implications. New York: W. H. Freeman.


Armstrong, M.. & Costa, B. (1983). Computer Cat(tm) at Mountain View Elementary School. Library Hi Tech, (Winter), 47-52.

Bates, M. J. (1989). The design of browsing and berry-picking techniques for the online search interface. Online Review, 13, 407-424.

Belkin. N. J., &Croft, W. B. (1987). Retrieval techniques. In Williams, M. E. (Ed.). Annual Revietv ofInformation Science and Technology], 22, 109-131.

Belkin, N. J., Oddy, R. N., & Brooks, H. M. (1982). ASK for information retrieval: Part 1. Background and theory. Journal of Documen- tation, 38. 6 l-7 I.

Borgman, C. L. (1986). Why are online catalogs hard to use? Lessons learned from information retrieval studies. Journal ofthe American Societ.v,for Znfinmation Science. 37, 387-400.

Borgman, C. L. ( 1989). All users of information systems are not created equal: An exploration into individual differences. Information Pro- cessing und Management, 25, 237-252.

Borgman. C. L., Bower. J.. Auth, M. J.. & Krieger, D. (I 989a). From hands-on science to hands-on information retrieval. In J. Katzer, G. Newby (Eds.), Proceeding.r of the 52nd American Society for Znfor- mation Science Annual Meeting, 26. October 29-November 2, 1989, Washington. DC. (pp. 96-103) Medford, NJ: Learned Information.

Borgman, C. L., Chignell. M. H., & Valdez, F. (1989b). Designing an information retrieval interface based on children’s categorization of knowledge: A pilot study. In J. Katzer. G. Newby (Eds.), Proceedings ol’the 52nd American Societyjor Information Science Annual Meet- ing, 26. October 29-November 2, 1989, Washington, DC. (pp. 81- 95) Medford, NJ: Learned Information.

Borgman, C. L., Gallagher. A. L., Krieger. D., & Bower, J. (1990a). Children’s use of an interactive catalog of science materials. In D. Henderson (Ed.), Proceedings ofthe53rd American Society.for Znfor- mation Sc,icwe Annual Meeting, 27. November 4-8, 1990, Toronto, Canada. (pp. 55-68) Medford. NJ: Learned Information.

Borgman. C. L., Gallagher. A. L.. Walter. V. A., & Rosenberg, J. B. ( 1992). The Science Library Catalog Project: Comparison of children’s searching behavior in a direct manipulation and a keyword search system. Proceedings ofthe 54th American Societya,for Znforma- tion Scic~ncc.4nnua/ Meeting, 28. October 27-3 I, 199 1, Washington, DC. (pp. 162-169) Medford, NJ: Learned Information.

Borgman, C. L., Krieger. D.. Gallagher, A. L.. & Bower, J. (1990b). Children’s use ofan interactive science library: Exploratory research. School Library Media Quarterly, 18. 108- 1 12.

Borgman. C. L., Krieger. D., Gallagher, A. L., & Bower, J. (199la). Children’s use ofan interactive science library: Exploratory research. Apple Library U.wr.s Group Newsletter. (Reprinted from School Li- brary Media Quarterly, (1990) 18, lO8- 1 12).

Borgman. C. L., & Siegfried, S. L. (1992). Getty’s SynonamerM and its cousins: A survey of applications of personal name matching algo- rithms. Journal ofthe American Society,for Information Science, 43, 459-476.

Borgman, C. L., Walter. V. A., Rosenberg, J. B., & Gallagher, A. L. ( 199 I b). Children’s use ofa direct manipulation library catalog. SZG- CHI Bulletin. 23, 69.

Brown, A. L. ( 1975). The development of memory: Knowing, knowing about knowing. and knowing how to know. Advances in Child De- velopment and Behavior, IO, 103-l 52.

Busey, P., & Doerr, T. (1993). Kid’s Catalog: An information retrieval system for children. Journalof Youth Services in Libraries, 7, 77-84.

Case, D. O., Borgman. C. L., & Meadow. (1989). The design and evaluation of a front-end user interface for energy researchers. Journal of theilmerican Society,for Information Science, 40, 99- 109.

Chi, M. T. H., Hutchinson, J. E., & Robin, A. F. (1989). How infer- ences about novel domain-related concepts can be constrained by structured knowledge. Merrill-Palmer Quarterly, 35, 27-62.

Chi, M. T. H., & Koeske, R. D. (1983). Network representation of a child’s dinosaur knowledge. Developmental Psychology, 19. 29-39.

Cleverdon. C. W. (I 99 I). The significance of the Cranfield tests on In- dex languages. Proceedings of the 14th International ACM Special Interest Group on Information Retrieval Conference on Research and

Development in Information Retrieval, October l3- 16, 199 I, Chi- cago. (pp. 3-l 2). New York: Association for Computing Machinery.

Conklin, J. (1987, September). Hypertext: An introduction andsurvey. IEEE Computer, 17-4 1.

Eastman, S. T., & Agostino, D. E. (I 986). Commanding the computer: Functions and concepts of videotex technology for eigth-grade students. Journal of Research and Development in Education, I9 (2), 49-57.

Edmonds, L., Moore, P., & Balcom, K. M. (1990, October). The effectiveness of an online catalog. School Library Journal, 28-32.

Egan, D. E. (1988). Individual differences in human-computer interaction. In M. Helander (Ed.), Handbook ofHuman-Computer Znterac- tion (pp. 543-569). New York: Elsevier.

Erthal, M. J. (1985). The status ofkeyboarding. The JournalofBusiness Education, 60. 192-193.

Freeman, R. R., & Atherton, P. (1968a). AUDACIOUS-An experiment with an online, interactive reference retrieval system using the universal decimal classification as the index language in the field of nuclear science. In Information Transfer: Proceedings of the 31st ASZSAnnualMeefing(pp. 193-199). New York: Greenwood.

Freeman, R. R., & Atherton, P. (1968b). File organization and search strategy using the universal decimal classification in mechanized reference retrieval systems. In Kjell Samuelson (Ed.), Mechanized Zn- formation Storage, Retrieval, and Dissemination: Proceedings of the FID/ZFIP Joint Cor$zrence (pp. l22- 152). Amsterdam: North-Hol- land Publishing.

Griffiths, J.-M., & Kertis, K. (1994). Automated system marketplace. Library Journal. 119, 50-59.

Hancock-Beaulieu, M. M. (1989). Online catalogues: A case for the user. The Online Catalogue: Developments and directions. (pp. 25- 46). London: Library Association,

Hancock-Beaulieu. M. M. (I 99 I). User friendliness and human-computer interaction in online library catalogs. Program, 26,29-37.

Harman, D. (1992). User-friendly systems instead of user-friendly front-ends. Journal ofthe American Society for Information Science, 43, 164-174.

Hildreth. C. R. (I 987). Beyond Boolean: Designing the next generation of online catalogs. Library Trends, 35.647-667.

Hirsh, S. G. (1995). The effect of domain knowledge on children’s search behavior on an information retrieval system: The Science Li- brary Catalog. SZGCHZ ‘95 Conference Companion, May 7-l I, 1995, Denver. Colorado (pp. 55-56). Annual Conference Series, ACM SIGCHI, New York: Assocation for Computing Machinery. Editors: Irvin Katz, Robert Mack, Linn Marks.

Hirsh, S. G. (in press). The effect of domain knowledge on children’s search behavior on an information retrieval system: The science library catalog. CHZ ‘95 Conference Companion.

Hirsh, S. G., & Borgman, C. L. (in press). Comparing children’s use of browsing and keyword searching on the Science Library Catalog. Proceedings of the 58th American Society for Information Science AnnualMeeting, 32. Medford, NJ: Learned Information.

Hooten, P. A. (1989). Online catalogs: Will they improve children’s access? Journal of Youth Services in Libraries, 2, 267-272.

Kail, R. (1984). The development of memory in children, 2nd ed. New York: W. H. Freeman.

Kay, A. C. (1991). Computers, networks and education. ScientiJic American, September, 139- 145.

Keil, F. ( 1979). Semantic and conceptual development: An ontological perspective. Cambridge, MA: Harvard University Press.

Krueger, R. A. (1988). Focus groups: A practical guide for applied research. Newbury Park, CA: Sage.

Kuhlthau, C. C. (1988a). Developing a model of the library search process: Cognitive and affective aspects. RQ, 28, 232-242.

Kuhlthau, C. C. (1988b). Meeting the information needs of children and young adults: Basing library media programs on developmental stages. Journal of Youth Services in Libraries, Z, (Fall), 5 I-57.

Kuhlthau, C. C. (199 I). Inside the search process: Information seeking


from the user’s perspective. Journal ofthe American Sucietll,for In- formation Science, 42, 36 l-37 1.

Liebscher, P., & Marchionini, G. (I 988). Browse and analytical search strategies in a full-text CD-ROM encyclopedia. School Library Me- dia Quarterly, 16 (Summer), 223-233.

Marchionini. G. (1987). An invitation to browse: Designing full-text systems for novice users. The Canadian Journal ofInformation Sci- ence, 12, 69-79.

Markey, K. (1984). Subject searching in library catalogs: BeSore and after the introduction ofonline catalogs. Dublin, OH: OCLC.

Markey, K. (1987). Searching and browsing the Dewey decimal classification in an online catalog. Cataloging & C/as$cation Qttartcrl.v, 7, 37-68.

Markey, K., & Demeyer, A. N. (1986). Dewey decimal classification online project.. Evaluation ef a library schedule and index integrated into the subject searching capabilities ofan online catalog.. Final report to the Council on Lihrary Resources. Dublin, OH: OCLC Online Computer Library Center, Inc.

Matthews, J., Lawrence, G., & Ferguson, D. (Eds.) (1983). Using On- line Catalogs: A Nations-ide Survey: New York: Neal-Schuman.

Meyerson, M. J., Ford, M. S.. Jones, W. P., & Ward. M. A. (1991). Science vocabulary knowledge of third and fifth grade students. Sci- ence Education. 75.4 19-428.

Mall, J. K. ( 1975). Children’,s access to i&rmation in print:An analysis of the vocabulag, (reading) levels of subject headings und their appli- cation to childrcnS books. Unpublished doctoral dissertation, Rutg- ers University.

Moore. P. A.. & St. George, A. (199 I). Children as information seekers: The cognitive demands of books and library systems. School Librar! Media Quarterly, 19, 16 l-l 68.

Nelson. L. (I 989). The influence ofphonies instruction on spellingprog- ress. (ERIC Document Reproduction Service. No. ED3 17953).

Patterson, A. C., & Bloch, B. (I 987). Formative evaluation: A process required in computer-assisted instruction. Educational Technologv, 27,26-30.

Pejtersen. A. M. (1986). Design and test of a database for fiction based on an analysis of children’s search behaviour. In P. Ingwersen, L. Kajberg. & A. M. Pejtersen (Eds.), Infixmation technologv and inli,r- mation use: ToH,ards a lrnified vie\48 qfinformation and information technology(pp. 125-146). London: Taylor Graham.

Pejtersen, A. M. (1992). New model for multimedia interfacesto online public access catalogues. The Electronic Library, 10, 359-366.

Piaget, J.. & Inhelder, B. (1969). Theps.scholog.vqfthechild. New York: Basic Books.

Rosenberg, J. B., & Borgman. C. L. (199 I). A report on the loading of MARC format bibliographic records into HyperCard. Information TechnologJs and Libraries, IO. 292-297.

Rosenberg, J. B., & Borgman, C. L. (1992). Extending the Dewey decimal classification via keyword clustering: The science library catalog project. Proceedings of the 54th American Society for Information Science Annual Meeling, 29. October 26-29, 1992. Pittsburgh. (pp. I7 I- 184) Medford, NJ: Learned Information.

Siegler. R. S. ( 1986). Children’s Thinking. Englewood Cliffs, NJ: Pren- tice Hall.

Siegler. R. S. (1991). Children’s Thinking, 2nd ed. Englewood Cliffs, NJ: Prentice Hall.

Solomon, P. (1993). Children’s information retrieval behavior: A case analysis of an OPAC. Journal ofthe dmerican Society.for IGforma- tion Science 44. 245-264.

Taylor, K. K.. & Kidder, E. B. (1988). The development of spelling skills: From the first grade through eighth grade. Written Communi- cation, 5. 222-244.

LJtilization of a microcomp~rter in an c1ementar.v school learning resource centre, (1983). Edmonton, Alberta: Alberta Department of Education, Edmonton. Planning Services Branch. (ERIC Document Reproduction Service. No. ED239-601).

Vandergrift. K. E. ( 1989). Are children and teenagers second-class users? Lihrar!, Resolmw and Technical Services, 33, 393-399.

Vasek, J. R.. & Volger, D. E. (1984). Applying formative evaluation to microcomputer assisted instruction: A case study. Journal qf Voca- tional Education Research, 9. 38-50.

Walter, V. A. (in press). Memory and metaphor: Designing the science library catalog. In S. Velluci (Ed.), Catalogsjbr kids: Organization. acceu, andintcr/&e. Medford. NJ: Learned Information.

Walter. V. A.. & Borgman. C. L. (1991). The science library catalog: A prototype information retrieval system for children. Journalof You/h Services in Libraries, 4, 159-166.

Wiberley, S. E.. Daugherty. R. A., & Danowski, J. A. (1990). User persistence in scanning postings of a computer-driven information system: LCS. Lihraq~and Information Science Research, 12, 341-353.


Children's Searching Behavior on Browsing and Keyword Online Catalogs: The Science Library Catalog...

Documents

Transcript of Children's Searching Behavior on Browsing and Keyword Online Catalogs: The Science Library Catalog...