may bear little relevance to the information they are seeking(Lucas & Topi, 2002; Spink, Wolfram, Jansen, & Saracevic,2001). Searchers rarely use Boolean operators and evenmore rarely use them correctly, negating any possible benefitthey might achieve. Search features labeled as “advanced”by search engines are seldom used, and “hints and tips” areusually ignored, partly due to an (often misguided) assump-tion that they are intended for advanced users (Pollock &Hockley, 1997). In addition, advanced search features areless accessible and therefore more likely to be overlookedthan the simple search box offered as the primary searchinterface. In fact, many usability experts explicitly adviseagainst implementing a search interface that supportsBoolean queries as the primary interface (Nielsen, 1997;Petersen, 2000).

The purpose of the study reported here is to investigatethe effects of training on users’ Web search performance. Itis not the search technology that is under investigation herebut rather the users’ interactions with that technology. Weconsider the effects of both training in Boolean logic andtraining in the specific characteristics of the search inter-face on user performance and attitudes. Study participantssought answers to information requests by searching aknown set of documents. All links had been removedfrom the documents, forcing users to formulate and refor-mulate queries rather than navigate from one page to an-other in search of answers. This study explicitly excludesissues related to broader search strategies (Brajnik,Mizzaro, Tasso, & Venuti, 2002) or paradigms (Dennis,Bruza, & McArthur, 2002).

Two of the experimental treatments were based on thetype of training the users received. The first of these treat-ments was the presence or absence of training on the princi-ples of the use of Boolean operators, henceforth referred toas logic training. The second was the presence or absence oftraining that focused on the specific characteristics of theassisted search interface, which we will refer to as interfacetraining. The third treatment was the type of interface usedby the participants, namely, simple or assisted. The impactsof logic and interface training and the type of search inter-face used on correctness, time, satisfaction, and confidencewere measured and analyzed.

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY, 55(13):1183–1198, 2004

Received June 11, 2003; revised September 15, 2003; accepted January 22,2004

© 2004 Wiley Periodicals, Inc. • Published online 13 July 2004 in WileyInterScience (www.interscience.wiley.com). DOI: 10.1002/asi.20074

Training for Web Search: Will It Get You in Shape?

Wendy Lucas and Heikki TopiDepartment of Computer Information Systems, Bentley College, 175 Forest Street, Waltham, MA 02452.E-mail: wlucas@bentley.edu

Given that time is money, Web searching can be a veryexpensive proposition. Even with the best search tech-nology, the usefulness of search results depends on thesearcher’s ability to use that technology effectively. In aneffort to improve this ability, our research investigatesthe effects of logic training, interface training, and thetype of search interface on the search process. In a studywith 145 participants, we found that even limited trainingin basic Boolean logic improved performance with a sim-ple search interface. Surprisingly, for users of an inter-face that assisted them in forming syntactically correctBoolean queries, performance was negatively affectedby logic training and unaffected by interface training.Use of the assisted interface itself, however, resulted instrong improvements in performance over use of thesimple interface. In addition to being useful for searchengine providers, these findings are important for allcompanies that rely heavily on search for critical aspectsof their operations, in that they demonstrate simplemeans by which the search experience can be improvedfor their employees and customers.

Introduction

Being able to conduct effective searches has becomemore and more of a requirement for many of us, with com-panies relying on e-commerce to market and sell their prod-ucts, and e-services rather than human operators to providesupport. Companies also depend on the searching skills oftheir employees for finding information from knowledgemanagement systems, corporate intranets, and the Web.While the effectiveness of the search technologies employedin these applications is critical, those technologies can onlybe effective if people are able to use them. To that end,searchers must understand how to express their informationneeds in a form that is both understood by the search engineand adequately conveys the intent of their search.

Yet, users’ queries are often inadequate for expressingtheir search criteria, containing only a few search terms that

1184 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—November 2004

The next section of this article reviews work related to theresearch presented here. This is followed by our researchmodel and hypotheses, study methodology, and results. Adiscussion of those results, directions for future work, andconclusions are then presented.

Related Work

The majority of studies related to Web searching focus onthe content of users’ queries or the characteristics of searchengines rather than factors affecting the search experienceand the impact of those factors on search performance. Twoexceptions are Topi and Lucas (2003) and Te’eni andFeldman (2001). In the former study, participants were pre-sented with one of two interfaces: a simple search tool likethose used by the majority of search engines or an interfacedesigned to assist the users in formulating correct Booleanqueries. Half of the participants received training onBoolean operators, while half did not. For those receiving notraining, the quality of the search results (as measured by thenumber of correct responses to information requests) wassignificantly higher for users of the assisted search tool thanfor users of the simple one. This difference in correctnessbetween interface treatments did not exist for users whoreceived logic training. In all cases, the use of the assistedinterface had a positive impact on both user satisfaction andconfidence. The latter study evaluated user performance butconcentrated on searching by browsing rather than by query-ing. They found that user performance, measured by timeand accuracy, improved when the user’s view of the Web sitewas adapted to the search task. User satisfaction with theadapted sites, however, did not increase, despite their im-proved functionality. The authors postulate that users preferWeb sites with consistent appearances.

The selection of terms used in a query has been shown tohave a significant effect on search performance (Lucas &Topi, 2002). Relevance feedback, which refers to the modifi-cation of queries by adding new terms and reweighting exist-ing ones based on user feedback, is an accepted means for im-proving search results in traditional information retrieval(IR) systems (Buckley, Salton, Allan, & Singhal, 1995;Harman, 1992; Salton & Buckley, 1990). Yet Web searchersare loath to take advantage of such features. A transaction loganalysis of 51,473 queries from 18,113 users of the Excitesearch engine found that only about 5% of users’ queriesmade use of the provided relevance feedback mechanism(Jansen, Spink, & Saracevic, 2000). The queries used an av-erage of 2.35 terms, and less then 10% included any Booleanoperators. These findings are consistent with those of similarstudies (Abdulla, Liu, & Fox, 1998; Silverstein, Henzinger,Marais, & Moricz, 1999; Spink et al., 2001), which foundthat users typically submit short queries containing two orthree terms. Although users do tend to change some termswhen modifying their queries, the total number of terms oftenremains unchanged (Spink, Jansen, Wolfram, & Saracevic,2002). In addition, logical operators are used infrequently(and used correctly even less frequently), and only the first

page of search results are typically viewed (Jansen et al.,2000; Silverstein et al., 1999).

Expert Web searchers do use more terms and advancedsearch operators than the average user (Hölscher & Strube,2000), but far fewer terms than is typically found in queriesto traditional IR systems, which are three to seven timeslonger than an average Web query (Jansen et al., 2000).Given the difficulty users have in formulating queries, nat-ural language processing (NLP) technology, in which theuser provides a description of the information need ratherthan keywords connected by logical operators, would seemto hold great promise for Web searches. In the IR commu-nity, it is generally accepted that users achieve better resultswith natural language interfaces than with explicit Booleansearches (Borgman, 1986; Anick et al., 1990; Turtle, 1994).Empirical evidence, however, is contradictory, with Hersh,Turpin, Price, and Kraemer (2001) finding that Booleanqueries can be at least as successful as natural languagesearches. The application of NLP technology to Web search-ing is in its infancy, with commercial search engines provid-ing limited capabilities in this area (Lucas, 2001). For exam-ple, although the AskJeeves search engine permits users toenter natural language queries, it does not attempt to inter-pret them. Rather, it returns a list of matching questions fromits database. The user then chooses from this list and is takento the Web page with an answer to the selected question.

Kwok, Etzioni, and Weld (2001) built a question-answering system called MULDER on top of the Googlesearch engine. MULDER parses the user’s natural languagequery, building several different keyword queries that areautomatically submitted to Google. Answers are extractedfrom the returned results using a voting procedure to selectthe best candidates. In comparisons with AskJeeves andGoogle, MULDER outperformed both in terms of requireduser effort at every level of recall. User effort was signifi-cantly less than that required with Google, and 34% of ques-tions had the correct answer as their top-ranked results (ascompared to 1% for Google). MULDER also demonstrateda threefold advantage in recall when compared toAskJeeves.

The IntelliZap System of Finkelstein et al. (2002) imple-ments a context-based approach to searching. Users canhighlight text in a document about which they want moreinformation. The context around that text is captured usingNLP techniques, including semantic keyword extraction andclustering, and new augmented queries are automaticallygenerated. These queries are submitted to general anddomain-specific search engines, and the results are rerankedbased on their semantic proximity to the original context.The level of performance achieved by IntelliZap, whichrequires very limited user involvement, was comparable tothat achieved by participants using major search engines.

Research efforts have also been devoted to providingusers with graphical representations of their queries, with theexpectation that these will be easier to understand than theunderlying textual queries. Anick et al. (1990) developedan interface that translates users’ NLP queries into a visual

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—November 2004 1185

representation of their Boolean equivalent. Users can thenmanipulate this representation to improve its expressiveness.This approach was not tested with any actual users. Youngand Shneiderman (1993) developed an interface that usesthe metaphor of water flowing through filters to representBoolean queries. In a comparison with a text-only SQL inter-face, 20 study participants performed better in terms of querycorrectness when using the visual interface. These findingsare not, however, directly applicable to Web search becausethe study was implemented in a structured database context.

The study described in this article is not about user inter-face design; our assisted search interface provides standardassistance with Boolean operators, using dropdown lists likethose found in the typical advanced search pages of com-mercial search engines. Rather, our focus is on the user’sability to conduct searches using standard search featuresand how that ability can be affected by training. For Websearches to be effective, it is critical that the queries enteredby the user accurately captures her information needs. In thisarticle, we explore the impacts of logic training, interfacetraining, and the search interface on the user’s ability to suc-cessfully translate those needs into executable queries.

Theoretical Framework

The success of a search is commonly defined in terms of itsperformance (correctness or accuracy, time) and the searcher’sattitudes (confidence and satisfaction) (Chan, Tan, & Wei,1999; Te’eni & Feldman, 2001). The theoretical frameworkfor our research focuses on those factors that ultimately affectthe successfulness of the search process via their effects onperformance and attitudes, as represented in Figure 1.

First and foremost, a searcher must understand the infor-mation request to initiate a successful search. This under-standing will affect the entire search process, from the qualityof the searcher’s conceptual query to his ability to understandquery results.

The conceptual Boolean query is also affected by thesearcher’s understanding of Boolean constructs. As an exam-ple, consider the case where the searcher is asked to findinformation about coffee beans that do not come fromColombia or Brazil. In conceptualizing this query, thesearcher must recognize that “coffee beans,” “Colom-bia,” and “Brazil” are the key terms or phrases that definethe context for this search. The searcher must also under-stand the nature of Boolean constructs that allow search cri-teria to be associated with search terms and phrases. That is,coffee beans must be included in the search results, butneither Colombia nor Brazil should be. The conceptualBoolean query can therefore be expressed as: “coffeebeans” AND NOT (Colombia OR Brazil).

The searcher must then translate this conceptual queryinto a form that can be parsed by the search engine to whichit is being submitted. Understanding the syntax rules gov-erning proper query formation will therefore impact thequality of the query implementation. If submitting toGoogle, for example, the use of “AND” or “�” is not re-quired, because all terms in the query must appear in everyretrieved document by default. This is not always the defaultinterpretation for other search engines. In Lucas and Topi(2002), it was shown that errors involving query operators,caused by either their incorrect usage or their absence, had asignificant negative effect on search results.

The success of a query, as defined by performance andattitudes, ultimately rests on the quality of the query imple-mentation and the ability of the searcher to understand and in-terpret the results. In addition, there are numerous individualand contextual characteristics that play a role in all aspects ofthe search process. These include the searcher’s experienceand domain knowledge (Hölscher & Strube, 2000), cognitiveabilities and style (Allen, 2000; Ford, Miller, & Moss, 2001),problem-solving style (Kim & Allen, 2002), self-efficacy(Ford et al., 2001), maturity, and ability to concentrate,among other factors.

Conceptual Understanding of the Information Request

Understanding ofBoolean Constructors

Quality of ConceptualBoolean Query

Quality of QueryImplementation

Ability to Understandand Interpret Results

Performance andAttitudes

Ability to UseSearch Interface

SearchInterface

Understanding of SearchEngine Characteristics

Additional Factors: Individualand Contextual Characteristics

FIG. 1. Model of factors affecting user’s search experience.

1186 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—November 2004

Research Model

The theoretical framework for our research lays thegroundwork for evaluating the effects of the users’ ability toformulate correct Boolean queries and to transform a concep-tual Boolean expression into a syntactically correct query.Topi and Lucas (2003) found that the use of their assistedsearch tool had a significant positive effect on the quality ofsearch results over the use of their simple search tool. Booleantraining, however, removed the assisted search tool advantageby compensating for the lack of direction and assistance in thesimple search environment. This study extends that work byconsidering the effects of assisted interface training, in addi-tion to those of logic training and the search interface, on thesearcher’s performance and attitudes, as shown in Figure 2.Performance is defined here in terms of the correctness of theresponse to an information request and the elapsed searchtime, while attitudes include satisfaction and confidence.

In accordance with the model of Figure 1, training inBoolean logic should improve the searcher’s understandingof Boolean operators, which should have a positive effect onthe quality of the conceptual query. We do not expect thistraining to have any impact on the selection of search terms,which also affects quality. Standard experimental proce-dures should ensure that errors related to search term selec-tion are evenly distributed amongst the study participants,limiting their effects on our results.

The choice of the search interface affects the quality of thequery implementation, and can be influenced by training inthe use of that interface. Because the simple interface is onewith which all search engine users are expected to be familiar,and an understanding of how to form Boolean queries is suf-ficient for its successful use, no additional training is needed.For the assisted interface, searchers should benefit from train-ing specifically aimed at understanding its functionality.

Hypotheses

Our first four hypotheses concern the users of the assistedinterface. Based on the results of prior related research(Topi & Lucas, 2003), we do not expect logic training to

affect the performance of these users. We do anticipate,however, that their performance can be improved by trainingthem on the specific characteristics of that interface. Weposit that interface training improves the participant’s abilityto translate the conceptual idea of a Boolean query into theimplemented query that is ultimately submitted to the searchengine. The search results retrieved by the search engineshould be more relevant to the information request becauseof the improvement in the users’ ability to formulate queries.Increased relevancy of search results leads to an increase inthe likelihood that the participant will find the correct an-swer to the information request. In addition, the time re-quired for finding that answer should be reduced becausetraining is expected to improve the speed of query formula-tion as well as the quality of the retrieved documents.

We also predict that interface training will improve theparticipants’ confidence in their results because they willhave a better understanding of the search interface. Havingbetter command of the search tool will reduce the effort theyneed to expend, thereby improving their satisfaction with thesearch environment. These predictions lead to the followinghypotheses:

H1: For the users of the assisted interface, interface trainingwill lead to improved correctness of search results.

H2: For the users of the assisted interface, interface trainingwill lead to reduced time required to complete the searchtask.

H3: For the users of the assisted interface, interface trainingwill lead to improved confidence in the search results.

H4: For the users of the assisted interface, interface trainingwill lead to improved satisfaction with the searchenvironment.

As previously mentioned, logic training on Boolean con-structors was shown to lead to improved search perfor-mance, as measured by the correctness of answers to infor-mation requests, for participants using a simple, nonassistedsearch interface (Topi & Lucas, 2003). This earlier researchalso implied that logic training is not effective if the partici-pants use an interface that supports the user in constructing

Operator Usage

Term Usage

Search Interface

• Simple• Assisted

Training Type

• Logic • Assisted Interface

User Support Success Measures

Correctness

Time

Satisfaction

Confidence

FIG. 2. Model of study treatments.

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—November 2004 1187

Boolean queries. In the absence of training, users of theassisted interface performed at a higher level than users ofthe simple interface. We expect to replicate these findings inthis research with simple interface users and assisted inter-face users who received interface training.

H5a: For the users of the simple interface, logic training willlead to improved correctness of the search results.

H5b: The users of the assisted interface will perform at ahigher level of correctness than the users of the simple in-terface who did not receive logic training.

Results of that earlier research also suggest that logictraining will not have an impact on the time it takes to per-form the search tasks. This is understandable because a rela-tively brief training session is not extensive enough for par-ticipants to be able to automate any of the routines requiredfor producing correct Boolean queries. Thus, training may,in fact, increase the required search time.

We do predict, however, that users of the simple interfacewho receive logic training will be more confident about theirresults than those who do not receive this training. This isbecause logic training will improve both the participants’understanding of the process of query writing and the qual-ity of the search result, leading to an increase in the partici-pants’ perception of the quality of their work. These samefactors should also cause an improvement in the partici-pants’ satisfaction with the search process.

H6: For the users of the simple interface, logic training willlead to improved confidence in the search results.

H7: For the users of the simple interface, logic training willlead to improved satisfaction with the search environment.

Methodology

Experimental Search Environment

The experimental environment used in this study was de-veloped for an ongoing research program on Web searching.It permits the integration of survey instruments and searchinterfaces to measure the effects of a variety of factors onuser performance and satisfaction in conducting Websearches. The modular design of this three-tiered architec-ture enables components to be swapped in and out, depend-ing on the specific needs of the experiment being conducted.

In a typical session using this research environment, astudy participant is first guided through a number of surveyand test instruments. The former are used to obtain informa-tion on computer skills, educational background, and vari-ous attitudes, while the latter test the participants’ under-standing of such concepts as Boolean operators or the use ofa search interface.

The participant then attempts to find the answers to sev-eral information requests through a repetitive process offorming queries, submitting them, and viewing the retrievedpages. Searches are conducted over a predefined searchspace containing a snapshot of 21,890 documents from a

university intranet Web site. All of the hypertext links havebeen removed from those documents so that participantsmust search, rather than browse, for information.

Two search interfaces have been integrated into the ex-perimental search environment to date. Each participant isassigned to one of these interfaces. The simple search inter-face is a text box with a submit button. An assisted search in-terface is also integrated into this environment. Dropdownmenus allow the participant to select the appropriateBoolean operators. Multiple terms entered to a search boxare treated as phrases, so that quotation marks are not re-quired. Parenthesis may be specified by clicking the buttonsappearing to the left and right of the search boxes. Figures 3and 4 show the simple and the assisted interface, respec-tively (please see the Appendix for a description of the usageof the assisted interface).

A query submitted from either interface is processed andforwarded to the Microsoft® Index Server search engine.Search results are returned in groups of 10, with a title and abrief abstract displayed for each of the pages included in theanswer set. The participant can click on links to view the fullpages listed in the set of retrieved results but cannot navigatefrom any of those pages to another page. If the participant doesnot find the answer being searched for, then new queries can beentered until either the answer is found or the maximum timeallotted for each information request has been reached.

Responses to surveys, keystrokes entered to the search in-terfaces, answers to information requests, and correspondingtime stamps are all stored in a database for later analysis.

Participants and Setting

The participants for this study were recruited fromstudents enrolled in an introductory information systemscourse at a small university located in the northeasternUnited States. Table 1 contains background information andInternet-related experience for the 145 participants. The ex-periment was conducted as an in-class search exercise, andparticipation was voluntary. As an incentive, $50 prizes wereoffered to the two best performing participants based on theaccuracy of their responses to the information requests andtheir total search times.

FIG. 3. Simple search interface.

1188 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—November 2004

TABLE 1. Demographic characteristics of study participants.

Mean Min Max

Age: 18.90 18 29

Male Female

Gender: 48.28% 51.72%

Freshman Sophomore Junior Senior

Class status: 87.59% 11.03% 0.69% 0.69%

�6 mos 6–12 mos 1–3 yrs 4–5 yrs >5 yrs

Online search exp: 4.83% 1.38% 36.55% 35.86% 21.38%Internet exp: 3.45% 2.76% 35.17% 37.93% 20.69%

interface, with those in cell II receiving logic training.Participants in cells III through VI used the assisted searchinterface. Cell III participants received no training, cell IVparticipants received only logic training, cell V participantsreceived only assisted interface training, and cell VI partici-pants received both logic and assisted interface training. It isimportant to note this is not a complete 2 � 2 � 2 design, be-cause there is no meaningful interface training treatment forthe users of the simple interface. This will, naturally, betaken into account in the statistical analysis of the data.

Procedure and Task

The experimenter provided all of the treatment groupswith a general introduction to the experimental stages to becompleted. Then the participants were directed to the URLof the first stage. As participants completed a stage, theywere individually directed to the URL for the subsequentstage. This process continued until each participant hadcompleted all stages of the experiment.

Participants receiving both types of training completedfour stages in the following order: (1) interactive Web-basedtraining in Boolean logic, (2) interactive Web-based assistedinterface training, (3) experimental search system overviewusing PowerPoint slides, and (4) experimental task. Partici-pants in the other treatment groups skipped stage 1, stage 2,or both, depending on the training they received. All partici-pants completed stages 3 and 4.

Boolean logic training. The logic training teaches participantsabout the basic Boolean operators (i.e., AND, OR, and ANDNOT), how to group query terms into phrases, and how toform query clauses using parentheses. Participants completeseveral exercises in which they must form a Boolean querycorresponding to a particular information request. Please seethe Appendix for a description of this training.

Assisted interface training. The interface training familiar-izes participants with the technical components of the inter-face and how they work, and demonstrates how to formqueries using the interface. The participants then complete

FIG. 4. Assisted search interface.

Design

Figure 5 shows the segmentation of study participants bysearch interface, assisted interface training, and logic train-ing. The participants in cells I and II used the simple search

No Logic Training Logic Training

No Assisted Search Training

I. Simple Interface II. Simple Interface

III. Assisted Interface IV. Assisted Interface

VI. Assisted InterfaceV. Assisted Interface

No Assisted Search Training

Assisted Search Training

FIG. 5. Research design.

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—November 2004 1189

several exercises in which they are provided with textualqueries that they must represent using the assisted interface.The Appendix also contains a description of the interfacetraining process. Please note that the interface training isclearly different from the logic training because it focusessolely on the characteristics of the technical interface,whereas the logic training emphasizes how to express a con-ceptual idea of a query using Boolean logic.

Experimental task. The experimental task used the experi-mental search environment previously described. Partici-pants first completed a background questionnaire, followedby a manipulation check on Boolean operators for use in de-termining the effectiveness of the logic training. Those par-ticipants who would be using the assisted interface thencompleted another manipulation check measuring the effec-tiveness of interface training.

All participants completed several more questionnaireson computer experience and attitudes before moving on tothe information requests. Each participant was presentedwith six questions in random order, and was allotted a maxi-mum of five minutes per question in which to search for andenter an answer. The queries required for each informationrequest fall into three levels of complexity, based on theirstructural characteristics. These complexity categorizationswere validated in a pilot study and have been used in anearlier study (Topi & Lucas, 2003). Those queries in the firstcategory were the simplest, requiring the use of the conjunc-tive AND operator between two single search terms. The sec-ond category required AND operators for joining three termsin the first case and two terms and a phrase in the secondcase. The two queries in the third category were the mostcomplex, requiring four or five terms or phrases and the useof the negation operator, NOT. In one of these cases, a morecompact query could be formed by applying the NOT opera-tor to a group of two terms joined with a disjunctive OR. Thesix information requests used in the experimental task, theoptimal Boolean queries for finding answers to those request,and the complexity of those queries appear in Table 2.

After entering an answer for an information request, eachparticipant entered answers to four instrument items on con-fidence levels before being directed to the next informationrequest. Those who reached the five-minute time limit weretaken directly to the next request. When all six requests hadbeen completed, participants filled in some final question-naires, after which they were dismissed.

Independent Variables

Logic training. The first independent variable is logic train-ing, that is, whether or not the participant received Web-basedtraining on the use of fundamental Boolean constructors.Operators AND, OR, and NOT were covered in the training;in addition, the use of quotation marks to construct phrasesand parentheses to group query structures together werereviewed. Logic training is described at a more detailed levelin the earlier section on Procedure and Task.

Interface training. The second independent variable, inter-face training, was also implemented on the Web as a seriesof screens that provided detailed step-by-step instructions onthe use of the assisted search interface. Interface training isalso described at a more detailed level in the Procedure andTask section.

Search interface. Finally, the third independent variable isthe search interface. As described above, participants usedeither the simple interface, consisting of only a text box, or anassisted interface containing user interface elements intendedto help participants convert their conceptual query into a syn-tactically correct query implementation (see Figure 4).

Dependent Variables

Correctness. The most fundamental performance variable iscorrectness, i.e., the searcher’s ability to provide a correctanswer to an information request, which is used in this studyas a measure of the quality of search results. All of the infor-mation requests were designed so that they had only one cor-rect answer, and the results were evaluated to be correct orincorrect based on the match between the participant’s an-swer and the correct one. Answers containing an error thatclearly was typographical were evaluated as correct. A mea-sure that aggregates individual correctness scores at the par-ticipant level is called score, which varies from zero to six(that is, the total number of information requests).

Time. The system included automatic mechanisms for mea-suring the time the participants used for formulating thequeries, viewing the results, and entering their chosen answer.

TABLE 2. Information requests, queries, and complexity.

Information request and query Comp.

R: Who is the chair of the philosophy department?Q: chair AND philosophyR: What former chair who established Bentley’s English

program died in Weymouth? 1Q: chair AND Weymouth

R: What faculty member of the management departmentwhose first name is John attended Syracuse University?

Q: John AND management AND“Syracuse University”

R: From what institution did Professor Davis of theManagement Department receive his BSEE? 2

Q: Davis AND Management AND BSEE

R: What faculty member has an MA from Indian University,is not in the philosophy or management department, andis interested in politics?

Q: “Indiana University” AND MA AND NOT(philosophy OR management) AND politics 3

R: What faculty member of the management departmentwhose first name is Joseph but whose last name is notByrnes taught and worked in the Middle East?

Q: Joseph AND management AND NOT ByrnesAND “Middle East”

1190 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—November 2004

Each user action was time stamped, so that very detailed in-formation about user behavior is available. In this analysis,the performance variable time is the sum of the times the par-ticipant used from first seeing the information request to en-tering an answer to that request. As noted earlier, the systemimposed a five-minute time limit, gave a warning 30 secondsbefore that time was up, and moved the user automatically tothe next question if the five-minute limit was exceeded.

Satisfaction. An ease-of-use measure adapted from Venkatesh(2000) was used to guage participants’ satisfaction with thesearch environment. This scale demonstrated acceptable psy-chometric characteristics in a reliability analysis (Cronbach’salpha � 0.937).

Confidence. The participants’ confidence in their answerswas measured with a one-item scale after each question. Theitem is the same at that used in the latest Text REtrievalConference (TREC) studies (see http://www-nlpir.nist.gov/projects/t9i/qforms.html).

Results

Manipulation Checks

Two 2 � 2 ANOVA analyses were used to perform themanipulation checks, the purpose of which was to measurethe effectiveness of the interface training and logic trainingmanipulations. In the first analysis with participants usingthe assisted interface (because there was no interface train-ing for the users of the simple interface), the independentvariables were logic training (training vs. no training) andinterface training (training vs. no training). The dependentvariable was interface test, a five-item instrument developedfor this research program that measured the participants’ability to translate Boolean constructs into Web queriesusing the assisted interface. This analysis reveals a verystrong main effect for interface training (F(1,93) � 71.039,p � 0.001) and a weaker main effect for logic training(F(1,93) � 4.293, p � 0.041). The main effect for logictraining is not surprising because this training also exposesthe participants to issues related to Boolean operators andincreases the participants’ familiarity with them (the differ-

ence between the means was relatively modest, with a meanof 3.43 out of a maximum of 5 for those with training and2.89 for those without it). The interface training effect is,however, significantly stronger and indicates that the inter-face training was effective in improving the participants’performance in the use of the assisted interface (the meanswere 4.27 for the interface trained group and 2.06 for thegroup without interface training).

The other manipulation check was also a 2 � 2 ANOVAtest, with logic training and interface as independent vari-ables and the pretest, which is a five-item instrument that hasbeen designed for this research program to measure the par-ticipants’ ability to use Boolean logic constructs, as the de-pendent variable. In this test, as in later hypotheses testing,the analysis included users of the simple interface andthose users of the assisted interface who received interfacetraining. The results indicate a strong main effect for logictraining (F(1,92) � 31.55, p � 0.001).

Hypothesis Testing

Whenever possible in the hypothesis testing, statisticalanalysis was performed at the individual information requestlevel with the complexity category associated with each in-formation request used as an additional independent variable.As will be seen, the explanatory role of complexity is essen-tial because it improves the quality of the analyses eventhough it is not one of the independent variables used in thehypotheses (for a more detailed description of the complexitycategories, please refer to the prior section on experimentaltask). The sample sizes and the means and standard devia-tions for the dependent variables score, time, ease-of-use(satisfaction), and confidence are shown in Table 3.

Hypothesis 1 suggested that the interface training willimprove the participants’ performance in the search tasks,as measured by the number of correct answers to the infor-mation requests. To test this hypothesis, a binomial logisticregression analysis was performed in which only the partici-pants that utilized the assisted interface were included(because there was no interface training manipulation for theusers of the simple interface, as noted earlier). This analysiswas conducted at the individual information request levelwith the two training manipulations (Logic Training and

TABLE 3. Numbers of participants, means, and standard deviations of dependent variables per treatment.

Score Time Ease-of-Use Confidence

Assisted interface N Mean SD Mean SD Mean SD Mean SD

No logic tr. No interface tr. 27 3.85 1.58 14.30 4.37 4.62 1.78 4.16 .89No logic tr. Interface tr. 21 4.05 1.32 14.27 3.19 4.55 1.83 4.29 .72Logic tr. No interface tr. 22 3.14 1.91 12.28 4.05 4.38 1.70 3.76 1.09Logic tr. Interface tr. 27 3.37 1.60 14.94 4.12 4.55 1.13 4.14 .71

Simple interface

No logic tr. No interface tr. 25 2.92 1.411 17.17 2.95 3.85 1.54 3.75 1.03Logic tr. No interface tr. 23 3.66 1.69 12.36 4.89 4.64 1.40 4.33 .64

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—November 2004 1191

Interface Training) and the complexity category of the infor-mation request (Complexity), along with all possible interac-tions between them, used as independent variables. Correct-ness was the dependent variable with binary values of Corrector Incorrect. The hierarchical analysis was started by enteringcomplexity as a predictor. The resulting model was highlysignificant (x2 � 105.92, 1 df, p � 0.001) suggesting thatComplexity is a strong predictor of Correctness. In the nextstep, Logic Training was added to the model and improved itspredictive capability, with a change in x2 of 8.757 (1 df, p �0.003). A closer evaluation of cell frequencies suggests thatComplexity and Logic Training each had a negative impacton Correctness. The third step added Interface Training, whichresulted in ax2 change of 1.232 (1 df, p � .267), showing thatInterface Training did not influence the participants’ ability tofind correct answers. In the last two steps, all interactions be-tween logic training, interface training, and complexity wereadded; neither one of these steps improved the model’s fit.Based on these results, Hypothesis 1 was not supported.

Hypothesis 2 predicted that interface training will reducethe time required by participants using the assisted interfaceto complete the information search task. This hypothesiswas tested using a 3 � 2 � 2 ANOVA analysis, with Com-plexity, Logic Training, and Interface Training as indepen-dent variables and Time as the dependent variable. The maineffect for Interface Training was only marginally significant(F(1,568) � 2.771, p � 0.097) and thus Hypothesis 2 wasnot supported. The rest of the analysis reveals that com-plexity had a strong main effect on Time (F(2,568) � 50.698,p � 0.000). There were no other significant effects.

Hypothesis 3 predicted that participants using the assistedinterface who have interface training will be more confidentabout their results than those without interface training. Theresults indicate that this was, indeed, the case. Interface Train-ing had a main effect on Confidence in an ANOVA analysiswith the same independent variables as above (F(1,504) �3.970, p � 0.047), and an inspection of the means reveals thatthis effect was positive. Hypothesis 3 was therefore sup-ported. In addition, both Complexity (F(2,504) � 24.128,p � 0.001) and Logic Training (F(1,504) � 9.096, p � 0.003)had main effects on Confidence, and both of these effects arenegative (i.e., Logic Training and increased Complexity led tolower Confidence). The results concerning Logic Training aresurprising and will be examined further in the Discussionsection. There were no significant interaction effects.

Hypothesis 4, the last one related to interface training,predicted that users of the assisted interface who receive in-terface training will be more satisfied with the search envi-ronment than those who do not. This hypothesis was testedusing a 2 � 2 ANOVA, with Logic Training and InterfaceTraining as independent variables and Ease-of-Use as thedependent variable. There were no significant differencesand thus, Hypothesis 4 was not supported.

Hypothesis 5a posited that logic training will lead to im-proved correctness of the search results for the users of thesimple interface. Hypothesis 5b suggested that, in the absenceof logic training, users of the assisted interface, both with and

without logic training, will perform at a higher level ofcorrectness than users of the simple interface who have nothad logic training. These hypotheses were tested using data atthe individual information request level, thereby requiringregression analysis. Furthermore, because the dependent vari-able, Correctness, is either right or wrong, analysis was per-formed using binomial logistic regression. The independentvariables were Complexity, Logic Training, and Interface.

The hierarchical analysis was started by entering Com-plexity as an independent variable. The resulting model washighly significant (x2 � 101.40, 2 df, p � 0.001), providingfurther support for the role of Complexity as a strong predic-tor of correctness. In the next step, Logic Training and Inter-face were added to the model. Given that the change inx2 was3.224 (2 df, p � 0.1995), these variables did not add to thepredictive capability of the model. The third step added twointeractions, Logic Training by Complexity and Interface byComplexity. The resulting change in x2 of 2.206 (4 df, p �0.697) indicates that these interaction effects were not signif-icant. The fourth step added the most important interaction,i.e., Logic Training by Interface, because the hypotheses test-ing results will be based on it: If this interaction is significant,it suggests that the effects of logic training are different de-pending on the interface that is used. The x2 change was9.576 (1 df, p � 0.002) and thus, the interaction is highly sig-nificant. An inspection of the cell counts reveals that logictraining benefited those who used the simple interface andwas harmful for those who used the assisted interface. Inspec-tion of the average results for the four Logic Training by In-terface quadrants shows that the users of the assisted interfaceperformed at the highest level but only if they did not receivelogic training. With logic training, the correctness of the usersof the simple interface is relatively close to the correctnessof the assisted interface users who did not receive logic train-ing. Therefore, Hypotheses 5a and 5b were supported, al-though the negative effect of logic training on participantsusing the assisted interface was not initially predicted.

As a last step, the three-way interaction among LogicTraining, Interface, and Complexity was added to the modeland improved its fit significantly (x2 change was 6.661, df 2,p � 0.0358). This result signifies that there is further varia-tion in the nature of the interaction between Logic Trainingand Interface based on the complexity category.

Hypothesis 6 proposed that the users of the simple interfacewill feel more confident about their results if they have LogicTraining before performing the tasks. This hypothesis wastested with a 3 � 2 � 2 ANOVA at the information requestlevel with Complexity, Logic Training, and Interface as inde-pendent variables, and Confidence as the dependent variable.The results of this analysis show a significant Complexitymain effect (F(2,492) � 23.030, p � 0.001) and a significantLogic Training by Interface interaction effect (F(1,492) �16.127, p � 0.001). An evaluation of the cell means revealsthat logic training has a positive effect on the confidence ofthe participants using the simple interface and a negativeeffect on the participants using the assisted interface. There-fore, Hypothesis 6 was supported.

1192 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—November 2004

Finally, Hypothesis 7 suggested that the users of the sim-ple interface will be more satisfied with the search environ-ment if they receive logic training. This hypothesis wastested at the participant level with a 2 � 2 ANOVA withLogic Training and Interface as independent variables andEase-of-Use as the dependent variable. Although an inspec-tion of the cell means suggests a strong interaction effect,this is not statistically significant, and thus Hypothesis 7 isnot supported.

Summary of the Results

One of the hypotheses related to interface training wassupported: As Hypothesis 3 suggested, users of the assistedinterface had more confidence in their search results if theyreceived training in the use of the search interface. Hypothe-ses 5 and 6 were also supported, i.e., logic training had apositive effect on the search result correctness and confi-dence in search results among the users of the simple inter-face. An analysis of the experimental data also indicates that,for users of the assisted interface, logic training had a nega-tive effect on correctness and confidence. Complexity wasalso found to have a negative effect on correctness, time, andconfidence for all users, regardless of interface and training.

Discussion

First, it is important to point out that the results of thisstudy support prior research (Topi & Lucas, 2003) suggestingthat even relatively limited training in formal logic can im-prove human performance in information search tasks forusers of an interface that does not provide assistance in the useof Boolean logic constructs. This is an important and highlyrelevant finding because most Web searchers rarely use ad-vanced search features and limit their queries to a few searchterms and even fewer operators (Spink et al., 2002). In addi-tion, participants using the simple interface who had Booleantraining were justifiably more confident in their searchresults. Although the characteristics of the interface were notthe main focus of this study, our findings also support earlierevidence that a user interface providing assistance in formu-lating Boolean logic can lead to the same performance im-provements as Boolean logic training.

Our results suggest that relatively small, well-designedinterventions (either user interface manipulations or training)can significantly improve the effectiveness of interfaces thatrequire the direct use of Boolean constructs by end users. Thisis a much simpler alternative to the development of interfacesthat offer increased support via NLP or graphical representa-tions of query constructs. Although the correct use of Booleanoperators may help searchers gain better results, many sites arefollowing advice by Nielsen (1997), Petersen (2000), and oth-ers who suggest that primary search tools should not includeBoolean search. Our results suggest that inexperienced users’Boolean searches can be effectively supported with simpletools that do not necessarily require the use of natural language(Anick et al., 1990) or a visual representation of the Boolean

structure (Young & Shneiderman, 1993). We do not have em-pirical evidence about the differences in effectiveness be-tween these approaches; this has to be left for future research.

We believe that our results provide justification for furtherresearch on the effectiveness of both search interface manip-ulations and training approaches and methods. Much of therecent research on Web search has focused on search strate-gies and other aspects of the broader search process (Brajniket. al, 2002; Cothey, 2002; Dennis, Bruza, & McArthur, 2002;Sutcliffe, Ennis, & Watkinson, 2000). While this research ishighly useful and has greatly improved our understanding ofthe factors affecting the ultimate success of the process offulfilling information needs, we should not ignore the morenarrow search process.

Effects of training were not, however, the same in all con-texts in this study. An unexpected result was that logic train-ing had a clearly negative impact on the performance ofusers of the assisted interface; that is, those participantsusing the assisted interface and receiving logic training per-formed significantly worse than those using the same inter-face but not receiving the same training. While we can onlyspeculate about the reasons underlying this phenomenon,there are several possibilities. The combination of the as-sisted interface and logic training could have led to concep-tual confusion even though these manipulations have differ-ent goals: The logic training is intended to support theformulation of conceptual query constructs, whereas thepurpose of the assisted interface is to help users translate aconceptual construct into the correct query implementation.A related possibility is that the combination of Booleantraining and assisted interface usage caused informationoverload, i.e., too many cues within a short period of time. Inaddition, logic training is likely to have had a better fit withthe simple interface than with the assisted one because theformer allows participants to enter Boolean queries in ex-actly the same way as they appeared in the logic training,whereas the latter requires the use of multiple text boxesseparated by interface-provided operators. Giving the logictraining immediately before the task may have prompted theparticipants to use the assisted interface in a less effectiveway (e.g., by entering combinations of terms and operatorsas phrases). The negative impact of logic training on bothsatisfaction with the search process and on user confidencefor assisted interface users, as opposed to the positive impactof logic training on confidence for simple interface users,provides some support for these suppositions.

A major goal of this study was to investigate the effec-tiveness of and the need for training that focuses on the char-acteristics of the assisted interface. The purpose of interfacetraining was to help the participants close the gap betweenthe conceptual query and the practical query implementationby pointing out available mechanisms for building querieswith the assisted interface. Our results suggest that, at least inthis study, the ability to use the assisted search interface wasnot the roadblock that prevented users from getting correctresults. This is indicated by the fact that interface training,which was shown to be successful by the manipulation

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—November 2004 1193

check, did not have any impact on correctness, time, or satis-faction with the search environment. Implicitly, this alsosuggests that the main benefit of the assisted interface wasthat it reminded the participants of the existence of theBoolean constructs and provided hints about the correctstructure. Once they became actively aware of them becauseof the presence of the interface, they appeared to have beenable to express their intent equally well (or poorly) with orwithout the interface training. Although interface trainingdid have a positive impact on the level of the users’ confi-dence in the search results, the benefits of this perceptualchange are questionable because they are not linked to achange in actual search performance.

It is therefore necessary to search for other sources un-derlying the less than perfect observed search performance.First of all, as a background fact, it is important to rememberthat query complexity was a very important determinant ofCorrectness—82% of the simplest queries, 60% of the mod-erately complex queries, and only 33% of the complexqueries were correct. The effects of all the factors discussedhere were significantly moderated by query complexity. Asindicated in our research model, human performance in Websearch tasks is affected by the selection of query terms, theselection of query operators and their correct syntactic use,the understanding of the characteristics of the search engine,the searchers’ ability to understand the initial informationrequest, and the ability to find a correct answer to an infor-mation request using the documents retrieved in response tosubmitted queries. In addition, results are affected by indi-vidual and contextual factors, including experience anddomain knowledge (Hölscher & Strube, 2000).

Implications

This study and the factors that were manipulated focusedprimarily on the selection and correct syntactic use of queryoperators. Our results and prior research indicate that train-ing in Boolean logic can help users of simple query inter-faces improve their search results. This concurs with re-search supporting instruction in Boolean logic for helpingusers conduct effective searches (Bellardo, 1985). The roleof the search engine, however, cannot be overlooked. Al-though Johnson and Szabo (1998) also found that training inboth keyword selection and Boolean logic led to the correctusage of keywords and operators in constructing advancedsearch statements, they did not find a corresponding increasein searcher success when participants submitted queries tothe Excite search engine. This can be attributed to the factthat Web search services seek to accommodate the typicalsearch behaviors of their users, who rarely use Boolean op-erators. Adding advanced operators may therefore have littleeffect on query results (Jansen, 2000; Lucas & Topi, 2002),depending on how the search engines interpret queries.

It is also important to note that in this study even those par-ticipants in the best-performing treatment groups were able tofind only about 70% of the correct answers on average, andthe worst groups performed below the 50% level. This finding

is supported by a survey of online end users at the Ohio StateUniversity business library, which found the success rates forend-user searches to be quite low (Ankeny, 1991) and by a re-cent study that found the overall performance of medical stu-dents performing MEDLINE searches to be poor (Sutcliffe etal., 2000). Based on observations that will be investigatedmore formally in future research, we believe that our partici-pants did not find the right answers because of one or more ofthe following factors: (1) incorrect or suboptimal search termusage, (2) inability to find answers that were available in theresulting document set, (3) misunderstanding of the informa-tion request, and (4) misunderstanding of the entire task, asdemonstrated in a few cases by participants who entered aBoolean query rather than a fact into the answer box. Choos-ing the correct terms, patiently evaluating the resulting docu-ment set beyond the first few documents, knowing how tosearch for keywords within returned documents, and carefullyevaluating the search task and its requirements are all essentialfor successful search performance.

Limitations

Our study participants were college students enrolled in arequired introductory computer course and were thereforelikely to have more experience, exposure to search engines,and comfort with searching than the general public. In addi-tion, they represented a limited range of ages. Although thehomogeneity of the participant population is a limitationfrom the perspective of external validity, it should improveinternal validity, which was, as in almost all laboratory stud-ies, the main concern of this research.

The logic and assisted interface training participants re-ceived was of relatively short duration. Although our manip-ulation checks did show strong effects of training on testperformance, it is possible that the amount of time devotedto the self-paced Web-based instructions was not sufficientfor fully absorbing the information presented and that longertraining could have had a stronger effect on performance.

We would also like to point out that the main focus of thisresearch was not on evaluating the user interface character-istics of a specific interface, but on evaluating how twodifferent types of training in conjunction with the availabil-ity of certain user interface features affect the participants’performance and attitudes. Therefore, a detailed evaluationof the assisted interface using methods of usability researchwas outside the scope of this study.

Finally, to focus solely on Web queries, we created an ar-tificial environment that prohibited users from the commoninformation-gathering approach of combining searchingwith browsing (Catledge & Pitkow, 1995). This undoubtedlyled to increased user frustration with both search interfaces.

Conclusions and Future Work

This study confirms that even a limited amount of logictraining can lead to improved correctness of search resultswhen a simple interface is used. The fact that an assisted

1194 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—November 2004

interface can achieve an even higher level of correctnesswithout any training is a significant result for search serviceproviders. Also of interest are the findings that interfacetraining with the assisted interface had no effect on perfor-mance and logic training actually hurt performance, perhapsby confusing the search participants. Although confidence insearch results increased with assisted interface training, thisis not a desired outcome because it was unaccompanied by acorresponding improvement in performance.

Training manipulations and the choice of search interfacewere not the only determinants of performance. The com-plexity of the queries required to retrieve the desired docu-ments had a strongly significant negative effect on perfor-mance and confidence. It was also postulated, and furtherresearch is expected to confirm, that search effectiveness de-pends on the searcher’s ability to understand the informationrequest, identify terms and phrases relevant to that request,understand the search task, and correctly interpret the resultsreturned by the search engine. We therefore believe that fu-ture research on Web search must pay a great deal of attentionto issues related to assisting the searcher in term selection,providing relevance feedback mechanisms, imparting mean-ing through the presentation of search results, and correctingfor or preventing simple mistakes in elementary tasks.

Acknowledgments

This study was supported in part by a Bentley CollegeFaculty Affairs Committee grant. We gratefully acknowl-edge this support. An earlier version of this article was in-cluded in the proceedings of the 2003 IEEE InternationalConference on Information Technology: Coding and Com-puting in Las Vegas, Nevada. We are thankful for the in-sightful comments by the editors and reviewers of all ver-sions of this article.

References

Abdulla, G., Liu, B., & Fox, E.A. (1998). Searching the World-Wide Web:Implications from studying different user behavior. Paper presented atthe WebNet98 Conference, Orlando, FL.

Allen, B. (2000). Individual differences and the conundrums of user-centered design: Two experiments. Journal of the American Society forInformation Science, 51(6), 508–520.

Anick, P.G., Brennan, J.D., Flynn, R.A., Hanssen, D.R., Alvey, B., &Robbins, J.M. (1990). A direct manipulation interface for Boolean infor-mation retrieval via natural language query. Paper presented at the 13thInternational ACM SIGIR Conference on Research and Development inInformation Retrieval, Brussels, Belgium.

Ankeny, M.L. (1991). Evaluating end-user services. Journal of AcademicLibrarianship, 16(6), 352–356.

Bellardo, T. (1985). An investigation of online searcher traits and their rela-tionship to search outcome. Journal of the American Society for Infor-mation Science, 36, 241–250.

Borgman, C.L. (1986). Why are online catalogs hard to use? Lessonslearned from information retrieval studies. Journal of the American Soci-ety for Information Science, 37(6), 387–400.

Brajnik, G., Mizzaro, S., Tasso, C., & Venuti, F. (2002). Strategic help inuser interfaces for information retrieval. Journal of the American Societyfor Information Science and Technology, 53(5), 343–358.

Buckley, C., Salton, G., Allan, J., & Singhal, A. (1995). Automaticquery expansion using SMART: TREC 3. Paper presented at the NISTSpecial Publication 500–225: The Third Text REtrieval Conference(TREC-3).

Catledge, L.D., & Pitkow, J.E. (1995). Characterizing browsing strategiesin the World Wide Web. Paper presented at the Third International WorldWide Web Conference, Darmstadt, Germany.

Chan, H.C., Tan, B.C.Y., & Wei, K.K. (1999). Three important determinantsof user performance for database retrieval. International Journal ofHuman-Computer Studies, 51(5), 895–918.

Cothey, V. (2002). A longitudinal study of World Wide Web users’information-searching behavior. Journal of the American Society forInformation Science and Technology, 53(2), 67–78.

Dennis, S., Bruza, P., & McArthur, R. (2002). Web searching: A process-oriented experimental study of three interactive search paradigms.Journal of the American Society for Information Science and Technol-ogy, 53(2), 120–133.

Finkelstein, L., Gabrilovich, E., Matias, Y., Rivlin, E., Solan, Z., Wolfman,G., & Ruppin, E. (2002). Placing search in context: The concept revis-ited. ACM Transactions on Information Systems, 20(1), 116–131.

Ford, N., Miller, D., & Moss, N. (2001). The role of individual differences inInternet searching: An empirical study. Journal of the American Societyfor Information Science and Technology, 52(12), 1049–1066.

Harman, D. (1992). Relevance feedback revisited. Paper presented at the15th Annual International ACM SIGIR Conference on Research andDevelopment in Information Retrieval.

Hersh, W., Turpin, A., Price, S., & Kraemer, D. (2001). Challenging con-ventional assumptions of automated information retrieval with real users:Boolean searching and batch retrieval evaluations. Information Process-ing & Management, 37(3), 383–402.

Hölscher, C., & Strube, G. (2000). Web search behavior of Internet expertsand newbies. The International Journal of Computer and Telecommuni-cations Networking, 33(1–6), 337–346.

Jansen, B.J. (2000). The effect of query complexity on Web searchingresults. Information Research, 6(1). Retrieved from http://InformationR.net/ir/6-1/paper87.html

Jansen, B.J., Spink, A., & Saracevic, T. (2000). Real life, real users, and realneeds: A study and analysis of user queries on the Web. InformationProcessing & Management, 36(2), 207–227.

Johnson, B.L., & Szabo, M. (1998, June 20–25). Instruction, gender, searchsuccess, and search satisfaction on the World Wide Web: A researchstudy. Paper presented at the World Conference on Multimedia andHypermedia Edmedia/Edtelecom, Freiburg, Germany.

Kim, K.-S., & Allen, B. (2002). Cognitive and task influences on Websearching behavior. Journal of the American Society for InformationScience and Technology, 53(2), 109–119.

Kwok, C., Etzioni, O., & Weld, D.S. (2001). Scaling question answering tothe Web. ACM Transactions on Information Systems, 19(3).

Lucas, W. (2001). Search engines, relevancy, and the World Wide Web. InA.G. Chin (Ed.), Text Databases and Document Management: Theoryand Practice (pp. 22–51). Hershey, PA: Idea Group Publishing.

Lucas, W., & Topi, H. (2002). Form and function: The impact of query termand operator usage on Web search results. Journal of the American Soci-ety for Information Science and Technology, 53(2), 95–108.

Nielsen, J. (1997). Search and you may find. Retrieved June 6, 2003, fromhttp://www.useit.com/alertbox/9707b.html

Petersen, C. (2000). Simplify & sort for better searches. Retrieved June 6,2003, from http://www.smartisans.com/articles/Web_search.aspx

Pollock, A., & Hockley, A. (1997, March). What’s wrong with Internetsearching. D-lib Magazine.

Salton, G., & Buckley, C. (1990). Improving retrieval performance by rele-vance feedback. Journal of the American Society for Information Sci-ence, 41(4), 288–297.

Silverstein, C., Henzinger, M., Marais, J., & Moricz, M. (1999). Analysisof a very large Web search engine query log. SIGIR Forum, 33(1),6–12.

Spink, A., Jansen, B.J., Wolfram, D., & Saracevic, T. (2002). From e-sex toe-commerce: Web search changes. IEEE Computer, 35(3), 107–109.

Spink, A., Wolfram, D., Jansen, B.J., & Saracevic, T. (2001). Searching theWeb: The public and their queries. Journal of the American Society forInformation Science, 53(2), 226–234.

Sutcliffe, A., Ennis, M., & Watkinson, S.J. (2000). Empirical studies of end-user information searching. Journal of the American Society for Informa-tion Science, 51(13), 1211–1231.

Te’eni, D., & Feldman, R. (2001). Performance and satisfaction in adaptiveWeb sites: A laboratory experiment on search tasks within a task-adaptedWeb site. Journal of AIS, 2(3).

Topi, H., & Lucas, W. (2003). Searching the Web: Operator assistance re-quired. Information Processing and Management (forthcoming; availableelectronically in final form from ScienceDirect).

Turtle, H. (1994). Natural language vs. Boolean query evaluation: A com-parison of retrieval performance. Paper presented at the SIGIR ’94.

Venkatesh, V. (2000). Determinants of perceived ease of use: Integratingcontrol, intrinsic motivation, and emotion into the technology acceptancemodel. Information Systems Research, 11(4), 342–365.

Young, D., & Shneiderman, B. (1993). A graphical filter/flow representationof Boolean queries: A prototype implementation and evaluation. Journalof the American Society for Information Science, 44(6), 327–339.

Appendix: Logic and Interface Training

Logic Training

The purpose of the logic training is to familiarize thestudy participants with the basic Boolean operators; the con-cepts of conjunction, disjunction, and negation that relate totheir use; the use of quotation marks for forming phrases;and the use parentheses to group query clauses. The traineeis first guided through several exercises focusing on the

principles of Boolean logic. In these exercises, the trainee isshown a sample information request and the correspondingBoolean query. Then, the trainee is prompted to form a queryfor another information request that is similar in structure tothe sample one (see Figures A1 and A3). After pressing thesubmit button, the correct Boolean query is shown alongwith the one entered by the trainee (see Figures A2 and A4).

After completing the practice exercises on the AND, OR,and AND NOT operators, as well as on the usage of paren-theses and quotation marks within queries, the trainee is pre-sented with a series of information requests and prompted toenter Boolean queries for each. These final queries serve asa review of all the information presented in the training ses-sions. As in the individual exercises, the trainee is shown theideal search string and the string that was entered for eachinformation request.

Interface Training

The purpose of the interface training is to familiarize thestudy participants with the look and feel of the interface andhelp them understand the technical functionality of the inter-face. The participants are first shown screen shots of the in-terface, which highlight the fields and buttons for enteringterms and phrases, selecting operators, adding parenthesis,and submitting a query. The screen shots shown during train-ing that describe these operations have been incorporatedinto the one view shown in Figure A5. Please note that the

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—November 2004 1195

FIG. A1. Training in the use of the AND operator.

1196 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—November 2004

FIG. A3. Training in the use of parentheses.

FIG. A4. Solution to parentheses operator practice exercise.

FIG. A2. Solution to AND operator practice exercise.

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—November 2004 1197

actual training consists of multiple slides, with each focus-ing on one interface characteristic.

The next step in the training is a review of the basic rulesconcerning interface usage. Figure A6 contains the informa-tion from the slides shown during training about how to formphrases and choose the AND operator. Figure A7 describeshow to select the OR and AND NOT operators, as well asparenthesis for applying conditions to multiple terms andphrases.

The preceding example would send this query to thesearch engine:

“Southwest Airlines”AND“Herbert K. Kelleher”AND retirement AND year

The preceding example would send this query to thesearch engine:

airline AND NOT (Northeast OR Transatlantic)

In the last phase of the training, the participant interactswith the assisted interface to form queries matching the

FIG. A5. Overview of assisted interface components.

FIG. A6. Summary of default usage rules.

1198 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—November 2004

textual examples shown in a total of five exercises. One ofthese exercises, which is representative of the other four, isshown in Figure A8. The participant is shown a textualquery and asked to form that query using the assisted inter-face. After pressing the submit button, the participant cansee if the formed query, as translated by the interface,matches the original textual one. If it does not, the partici-pant can reform and resubmit the query until it has beenentered correctly.

FIG. A7. Summary of operator and parenthesis usage rules.

FIG. A8. Sample training exercise.