W3

382 The Jou

Using the Wisconsin–Ohio Reference EvaluationProgram (WOREP) to Improve Training andReference Services

by Eric Novotny and Emily Rimland

Available online 12 March 2007

This article discusses a service quality studyconducted in the Pennsylvania State University

Libraries. The Wisconsin–Ohio ReferenceEvaluation Program survey was selected as a

valid, standardized instrument. We present ourresults, highlighting the impact on reference

training. A second survey a year laterdemonstrated that focusing on behavioralaspects of reference can improve service

quality ratings.

Eric Novotny is Humanities Librarian,Arts and Humanities Library, University Libraries,

The Pennsylvania State University,University Park, PA 16802-1803, USA

becn1@psu.eduN;Emily Rimland is Reference Librarian,

30 Pattee Library, Laboratory, University Libraries,The Pennsylvania State University,

University Park, PA 16802-1803, USAbelf113@psu.eduN.

rnal of Academic Librarianship, Volume 33, Number 3, pages 382–39

INTRODUCTION

In the late 1990’s the Pattee Library at the University Parkcampus of Pennsylvania State University underwent a radicaltransformation. Along with extensive physical renovations, thelibrary collections and services were re-distributed into newlycreated subject and format libraries. These ‘‘libraries within alibrary’’ were designed to focus on assigned areas. It wasexpected that by concentrating on specific disciplines the subjectlibraries would be able to provide enhanced levels of service fortheir users. This article discusses the use of the Wisconsin–OhioReference Evaluation Program (WOREP) survey instrument tocollect data on reference quality at one of the newly createdservice desks. We focus on how the Arts and Humanities subjectlibrary modified service desk training based on the results of theWOREP study, and its impact on the delivery of referenceservice. A second WOREP study was conducted a year later toassess the effectiveness of the changes we made. The secondstudy showed vast improvements in how patrons rated ourreference services. As a result of these experiences the authorssuggest best practices for quality service training.

The authors work in one of the newly created subjectlibraries, the George and Sherry Middlemas Arts andHumanities Library. The Arts and Humanities Library pro-vides collections and services in a variety of subject areasincluding: area studies, classics, dance, film, history, lan-guages, literature, music, philosophy, religious studies, the-ater, and the visual arts. To serve users in these intellectuallydiverse subject areas, the Arts and Humanities Library staffstwo service desks. The Arts and Humanities Reference Deskis housed near the print reference works, and is adjacent to anentry point to the circulating collection of approximately onemillion volumes. The Music and Media Center (MMC) deskprovides circulation services for the closed-stack collection ofapproximately 20,000 sound and 5000 video recordings, acourse-reserve service for media, and reference service formusic and media. During the study period, both service deskswere staffed every hour the library was open, a total of 102hours per week each. Staff included six faculty librarians, fivefull-time staff assistants, and 2.5 FTE part-time referenceassistants (students). Both desks participated in the study.

From the onset it was clear that Arts and Humanitiesreference staff faced a daunting challenge. They were expected

2

to be knowledgeable about the core print and electronicresources in a staggering range of disciplines, everything fromArt and Music, to History and Asian Studies. To address this, anextensive training program was devised to ensure that staff wereaware of key resources. New hires received a lengthyorientation manual which was reviewed with the Coordinatorsof Reference Services over many hours of individual sessions.This initial orientation was supplemented by staff trainingsessions conducted throughout the semester. Led by subjectspecialists, these sessions typically covered resources in a singlediscipline in depth, or reviewed course assignments expected togenerate significant amounts of questions at the service desks.In spite of these elaborate (and time-intensive) efforts, there wasa sense that our service offerings fell short of the idealenvisioned by the subject library model. Informal observationsand conversations revealed instances of less than satisfactoryservice, but we lacked evidence beyond the anecdotal. PennState participates in the LIBQual+program used to collect dataon patron impressions of service quality, but the instrument isdesigned to measure service in general, and does not specifi-cally relate to reference desk transactions. Also, users are askedto respond about the library’s services as a whole, making itimpossible to pinpoint the performance of a specific serviceunit. The study discussed in this article was conducted to collectquantitative and qualitative data on patron’s perceptions ofreference service in the Arts and Humanities Library.

LITERATURE REVIEW: ASSESSMENT AND TRAINING

Evaluating reference quality has been a concern of librarianssince the genesis of the profession, and literature on this topicis copious. The Reference Assessment Manual (RAM)published in 1994 by the Reference and Adult ServicesDivision of ALA has served as a resource for librariansinterested in reference evaluation methodology.1 Since itspublication, librarians have continued to debate and conductresearch on assessing reference service quality.

One way to break down the mountain of research in this areais by looking at two main methodologies for analyzing thereference encounter—obtrusive and unobtrusive studies. Unob-trusive studies are evaluations in which reference staff areunaware that a study is taking place. Recognized unobtrusivestudies were introduced in the 1960s by Crowley and Childers.2

The seminal article that reported this type of study was Hernonand McClure’s ‘‘Unobtrusive Reference Testing: The 55Percent Rule,’’ which introduced the disheartening 55 percentaccuracy benchmark to the profession in 1986.3 Althoughpopular, the methodology used by many unobtrusive evalua-tions in the past only distinguishes between right or wronganswers and does not account for the complexities that comprisemost reference questions. Because of these black and whitelimitations, newer studies have been developed to account forthe nuances and behaviors involved in answering referencequestions. For example, Joan Durrance’s ‘‘Willingness toReturn’’ study introduced new measures into unobtrusivestudies by measuring the willingness of a patron to return tothe same library staff member for help with another question.4

Dewdney and Ross were able to use Durrance’s measures totarget how particular aspects of the reference transactions (suchas how the interaction is terminated) affect reference success.5,6

The unobtrusive evaluation has been used most recently totarget particular aspects of reference transactions rather than theoverall success or failure of answering a question correctly.

The most common manifestation of the obtrusive study –where participants know they are being evaluated – is the usersurvey. Although reputed for their valuable insights, usersurveys are often plagued by response bias on the part of theuser, who, rather than responding with their honest feelings,responds with answers they feel are expected of them. Inaddition to response bias, another caveat of employing usersurveys is the lack of differentiation between measures ofsatisfaction and measures of success. While similar and closelyrelated, satisfaction and success are, in fact, different measures.Satisfaction can be simply defined as the overall outcome ofthe encounter, while success contributes (usually quite largely)to that overall experience. For example, a patron may not findexactly what he or she wanted, but can still feel generallysatisfied if they experienced friendly service at the library. Forthis reason, it is important to measure satisfaction and successseparately and provide a finer level of granularity when usingsurveys to evaluate reference services.7

To address survey response bias, and to separate satisfactionfrom success, Marjorie Murfin, Gary Gugelchuk, and CharlesBunge developed a new instrument in the mid 1980s, firstknown as the Reference Transaction Assessment Instrument(RTAI), and known today as the Wisconsin–Ohio ReferenceEvaluation Program or WOREP.8 After each reference trans-action, the staff member and patron complete separate surveys,rating different aspects of the reference encounter. At theconclusion of the survey period, the survey pairs are reunitedfor processing using a unique identifier. The result is arelatively complete picture of each reference transactionfurnished to the participating library. Transactions are onlyconsidered successful if the patrons indicate that they foundexactly what they wanted, were fully satisfied with thetransaction, and do not indicate any reasons for discontent.9

A few features of the WOREP set it apart from other usersurveys. The main benefit is its decoupling of service ratingsfrom success ratings, which provides unambiguous measure-ments in these areas and increases the legitimacy of the test.Another advantage is the use of standardized, expertly designedquestionnaires. Due to the standardization of the WOREPsurvey, statistical analysis of completed WOREP surveys can beexpedited at a processing center. Participating institutions areable to benchmark results with other libraries using a shareddatabase.10 These benefits have made the WOREP a popularchoice for reference evaluation because libraries can easilyrecognize their areas of strength and target areas that needimprovement. For example, Brandeis University was able to putits new tiered reference model to the test and found through aWOREP study that the amount of time spent on questionscontributed to its high reference success rates.11 Other studieshave used the WOREP to examine how techniques from thebusiness world can help to improve reference service.12

Although newer techniques for evaluating reference effec-tiveness have been introduced, there is by no means agreementas to which method is best. However, studies that reflect the truecomplexities of reference transactions are a welcome addition toearlier studies that only view success in black and white.

WOREP STUDY AT PENN STATE

In Fall 2001 the Arts and Humanities Library agreed toparticipate in the Music Library Association’s WOREP study.Music libraries at colleges and universities are frequently

May 2007 383

decentralized (branch) libraries that tend to be staffed bystudents and paraprofessionals a higher percentage of time thancentralized library service points such as general referencedesks. Yet music reference service is a specialty requiringknowledge of music and music formats, particular rules fordescription and access in library catalogs, and an array ofreference sources and techniques. Recognizing this, membersof the Reference Performance Subcommittee of the MusicLibrary Association designed a research project to evaluatewhich library and staffing characteristics have a significanteffect on the overall question-answering success of referencestaff in the decentralized music library environment. The MusicLibrary Association chose the WOREP as its evaluationinstrument because it was ‘‘a valid and reliable tool formeasuring. . .effectiveness in answering reference ques-tions.’’13 The study utilized a slightly modified version of thestandard WOREP survey instrument (see Appendix A).14 Thisform was used at both the Arts and Humanities Reference Deskand the Music and Media Center desk.

To prepare library staff, the WOREP guidelines werediscussed at staff meetings before the survey began. The surveyform was reviewed to ensure that staff were familiar with thequestions and interpreted them the same way. As recommendedby WOREP, a sampling test period was designated during adnormalT time of year. All patrons asking a question (other thanDirectional) in the Arts and Humanities Library were asked toparticipate in the survey. Surveys were distributed until therequired minimum of 100 completed surveys were obtainedfrom each service desk which took a total of twenty days.During this period survey data were collected for 201 referencequestions asked at the Arts and Humanities Reference Desk and100 reference questions at the Music and Media Center desk.An additional 1173 and 946 directional questions were recordedat the Arts and Humanities and Music and Media Center desksrespectively, but patrons with these types of questions were notasked to participate in the survey.

When compared to the aggregate WOREP population theArts and Humanities sample included more juniors and fewersophomores. This reflects student enrollments at the UniversityPark campus. Juniors are the largest population at UniversityPark (around 23 percent) due to transfers from other Penn Statecampuses offering two year degrees. Graduate students makeup around 14 percent of the student population at UniversityPark Table 1.

Table 1WOREP Respondents by Status

2001 Arts andHumanitiesLibrary

2003 Arts andHumanitiesLibrary

All WOREPLibraries(2003)

Freshman 17.2 16.5 16.0

Sophomores 9.9 8.6 15.0

Juniors 23.9 28.3 17.6

Seniors 21.9 23.7 21.5

Graduates 12.5 15.8 15.6

Faculty 4.2 3.3 3.6

Other (staff, alumni) 10.4 3.8 10.7

384 The Journal of Academic Librarianship

The primary success measure we focused on was thepercentage of users who reported that they ‘‘Found exactly orapproximately what they wanted AND were satisfied.’’ At theArts and Humanities Reference Desk 64 percent of participantsdescribed the interaction this way for the ‘‘complex’’ referencequestions (those that did not involve looking for a knownitem). This was six percentage points below the average on thismeasure for all WOREP libraries (70 percent). Alarmingly,fewer than half of the 2001 respondents (47 percent), reportedfinding EXACTLY what was wanted and were satisfied at theArts and Humanities Reference Desk. Again, this was wellbelow the WOREP average (58 percent). While there wereexceptions, the Arts and Humanities Reference Desk consis-tently performed below the WOREP average on mostmeasures.

Music and Media Center employees performed better: forthe more complex reference questions that comprised two-thirds of the question pool, users reported they found exactly orapproximately what they wanted, and were satisfied, 72 percentof the time (the national average for all libraries was 70percent). Their performance with specific- or known-itemquestions (one-third of the sample) was also within 2percentage points of the national average: users found exactlywhat they wanted 58 percent of the time compared with thenational average of 60 percent.

Given our desire to provide excellent service, the results ofthe initial WOREP survey were disappointing overall, andgenerated substantial discussion and analysis. The remainderof this article describes the steps taken to improve servicequality, and the results achieved in the Arts and HumanitiesLibrary.

CLUES TO LESS THAN SUCCESSFUL PERFORMANCE

An important feature of the WOREP is that it not only recordsuser satisfaction, but it also gathers data on aspects of theencounter that may contribute to the success (or failure) of thereference interaction. Users report on factors such as thequantity of information provided, the amount of attention theyreceived, and the perceived knowledge of the staff personassisting them. For the same encounter library staff recordinformation such as the perceived depth of the collection on thetopic, and the level of activity at the desk. These data can helplibraries identify specific environmental and behavioral factorsthat influence patron outcomes.

After analyzing the data from the survey (see Table 2 for asummary of selected data), we identified several areas that wefelt could be improved. These included:

Time – Arts and Humanities reference desk staff reportedspending less than three minutes on almost half (45 percent)of the more complex questions (those not involving knownitems). The average for all WOREP participants was 35percent. This indicated we were concluding the referenceencounter more quickly than our peers. Although this couldbe an indicator of staff efficiency, the relatively lowsatisfaction ratings suggested we were not spending enoughtime with our users.

Sources – We reported using one source at the Arts andHumanities Reference Desk 47 percent of the time, versus42 percent for all WOREP participants. The single sourcewas usually the library catalog. Often we were not alerting

Table 2Selected 2001 WOREP Study Results

Survey Measure

Arts andHumanities

Reference DeskAll WOREPlibraries

Patrons who ‘‘found exactly or

approximately what was

wanted’’ and satisfied

64.5% 69.9%

Employee reports information

‘‘found’’

72% 68.5%

Patron reports ‘‘found exactly’’

and was satisfied

47.1% 58.2%

Patrons finding exactly what was

wanted when library staff

‘‘helped with the search’’

53.9% 66.0%

Patrons finding exactly what was

wanted when library staff

‘‘directed or suggested only’’

36.7% 54.0%

Library staff use only one source 47.4% 42.0%

Library staff answers question in

less than three minutes

45.7% 35.4%

Patron reports ‘‘not enough

explanation or help’’

12.3% 7.3%

Patron reports ‘‘library employees

only partly or did not

understand what was wanted’’

10.1% 5.1%

Patron reports ‘‘not enough’’

information

15.2% 12.5%

Staff reports ‘‘books off shelf’’ 13.0% 4.3%

our users to other potentially useful resources such as printreference works or library subject guides.

The impact of the two factors above can be seen by ourusers reporting finding ‘‘Not Enough explanation or Help’’ 12percent of the time (well above the WOREP average). Thisindicated to us that our staff needed to concentrate on spendingmore time providing additional resource options to ensure usersfound what they were looking for.

The physical Environment was identified as another keyfactor. In 13 percent of the interactions respondents at theArts and Humanities Reference Desk indicated that‘‘Books off shelf’’ was a problem. This is more than threetimes the WOREP average of 4 percent. This was notunexpected. The Arts and Humanities print collections aredispersed over seven floors making it difficult for referencedesk staff to assist users in locating materials found in theonline catalog. The distance of the service desk from mostof the circulating collection also limits the ability of staffto help users find alternative sources if their first choicesare not on the shelf.

Communication – Library employees and patrons reporteddiffering success rates. Library staff were far more likely

than our patrons to conclude that the encounter had gonewell. Employees reported that the desired information was‘‘found’’ 72 percent of the time. Patrons at the Arts andHumanities Reference Desk indicated that they ‘‘foundexactly’’ what they wanted 47 percent of the time. Thistwenty-five point gap indicated to us that there was abreakdown in communications at some point in thereference process.

Mode of interaction – For each question library employeesreported whether they ‘‘Directed and suggested only’’ or‘‘Helped with or made search.’’ Patrons were more likelyto report finding what they wanted if they had helpsearching. Fifty-four percent reported ‘‘Found Exactly’’when helped with the search, versus 37 percent who onlygot ‘‘Directions and Suggestions.’’ These results wereconsistent with the WOREP averages across all libraries,which reveal that fewer respondents report ‘‘FoundExactly’’ when library staff direct or suggest only,compared to when the staff helped with the search (54percent to 66 percent, respectively).

In addition to the factors cited above, another measure wasinvestigated as a cause of our below average satisfactionratings. This was the use of employees other than professionallibrarians at service desks. The WOREP instrument collectsdata about the staff completing the survey. Categories areProfessional Librarian, Library Assistant, and Student/OtherEmployee. This allows for analysis of the data by the level ofstaff involved in answering the question. Although differenceswere present in the 2001 study, they were not emphasized insubsequent training conversations. More important to us wasthat all staff levels performed below average. As a group, eventhe subject specialists at the Arts and Humanities ReferenceDesk performed below the WOREP average on the ‘‘Foundexactly or approximately what they wanted AND weresatisfied’’ measure. Rather than highlighting staff differences,we wanted to make clear that the data pertained to everyone inthe Arts and Humanities Library, and we all had to commit to asolution.

RESPONDING TO THE STUDY RESULTS

Based on the results and analysis described above, it was clearthat there was room for improvement in service performance.Staff meetings were used to report the results, raise awarenessof the issues, and brainstorm ideas. The critical issue was to getstaff to ‘‘buy-in’’ to the process. At this juncture thestandardized nature of the WOREP proved invaluable. The(not unexpected) reaction of some staff upon hearing theresults was to poke holes in the methodology—what could becalled the ‘‘Yeah, but...’’ response. ‘‘Yeah our results weremediocre, but that’s because our patrons don’t know what theywant.’’ A benefit of the WOREP is that it has been utilizednationally, allowing for comparisons with peers. Thosedismissing the results had to provide a credible explanationfor why our patrons differed so dramatically from otherinstitutions. Following the meeting at which the data wereshared, two brainstorming sessions were held in whicheveryone’s responses and opinions were heard and validated,followed by a third group discussion inviting everyone to offer

May 2007 385

ideas for improvements and solutions. Once staff realized theirideas would become part of the solution, they bought into theprocess.

After the results were disseminated and accepted, theentire staff discussed best practices with an emphasis oncommunication and behavioral aspects, areas that theWOREP had identified as critical. This represented amarked change in approach. As noted earlier, prior trainingefforts were heavily resource-based, usually led by subjectspecialists who went into great detail about the manyspecialized tools in their disciplines. While these sessionswere essential for building up a core knowledge base amongpublic service staff, the WOREP results indicated thatfundamental behavioral techniques were also key to promot-ing reference success. Of course, this is not news to anyonefamiliar with the reference assessment literature,15 but theWOREP results provided concrete data supporting theimportance of the behavioral component of the referenceencounter.

To build upon the WOREP results, summer staff meetingswere devoted to discussing selected reference articles andtraining videos.16 We found it helpful to provide excerpts of thearticles to maximize staff time and avoid portions of anotherwise outstanding article that employed a tone that mightbe viewed as denigrating to non-professional staff. The videos,while entertaining, proved less helpful than the readings. Wewere unable to identify a recent general video on how toconduct a quality reference interview. The ones we did obtainwere outdated. They probably generated more discussion aboutthe antiquated clothing styles than effective communicationtechniques.

Based on the group conversations and discussion of theliterature on reference success, we were able to agree on a listof ideal desk behaviors emphasizing areas identified by theWOREP results. A representative selection is provided here togive a sense of how the study results were translated intorecommended reference practices.

1. Get out from behind the desk as often as possible and assistusers with their searches (those who had ‘‘Help with theSearch’’ were more likely to rate the outcome successful).

2. Always think of, and mention to the user, at least twosources (we had reported using only one source more oftenthan the WOREP average, a factor that was seen ascontributing to less than successful encounters).

3. Do not assume the online catalog is always the first place tolook (related to #2).

4. Offer alternatives in case the requested volumes are not onthe shelf; interlibrary loan, others books on the same topic,articles on the topic, etc.. . . (to address issues with thephysical environment and ‘‘Books Not on Shelf’’ discussedearlier).

See Appendix B for the complete list of ideal desk behaviors.Though reference performance at the Music and Media

Center desk was of less concern, due probably to its compact,closed-stack collection, and more focused subject coverage(music and media), we nevertheless provided refresher train-ing for music reference, emphasizing use of the library

386 The Journal of Academic Librarianship

catalog in more advanced ways, including uniform titles andsubject headings, and helping staff to think more creativelyabout alternative sources when requested items were not onthe shelf.

Our training techniques, and the list of best practices wedeveloped, are probably familiar to anyone who providespublic service. We certainly could have had similarconversations without conducting a formal study. The valueof participating in the WOREP was that it provided a‘‘wake-up’’ call. At the first meeting there was much gaspingas the numbers were reported in rather stark, unambiguousterms. The WOREP data greatly facilitated staff buy-in forthe subsequent ‘‘back to basics’’ reference training that wasdeveloped. Without the study results as a catalyst, it isdifficult to imagine experienced reference staff engaging sowillingly in a series of discussions about how to spend moretime with users, or the importance of consulting more thanone resource when answering a question. Most of usprobably assume we are already doing these things satisfac-torily. The WOREP study provided an invaluable refresher toeveryone, the authors included, of the importance of thesehabits and techniques in the success of the referenceencounter.

DID THE TRAINING WORK?

In Fall 2003, the Arts and Humanities Library conducted afollow-up study to determine the effectiveness of the trainingconversations. Procedures were the same as described for the2001 study. During the sampling period 152 completedforms were collected at the Arts and Humanities Desk.Upper-level undergraduates and graduates made up a higherproportion of respondents when compared to the earlierstudy (see Table 1).

Desk staffing remained nearly constant from the 2001 study.Only two-part time student workers were new out of a totalstaff of 13.5 FTE. The reference environment was alsofundamentally unchanged. As will be seen below, access tothe circulating collections continued to present obstacles inhelping patrons locate exactly what was desired. While thesefactors remained essentially constant, the 2003 WOREP surveyresults improved in almost every category. A few examplesinclude:

More Time – After emphasizing this in our discussions, wereported answering a question in less than three minutes farless often. This figure declined from 45 percent in 2001 to10 percent in 2003 (see Fig. 1). This suggests we werespending more time listening, communicating, and betterhelping our users.

More sources – In the 2001 study we used only onesource almost half the time. Training sessions emphasizedmentioning other useful sources to ensure that patronswere aware of the range of options available to them(library Web guides, article indexes, print referenceresources, etc.) In 2003 we reduced the times we usedonly one source by more than half (see Fig. 2) Likely asa result, patrons reporting that they learned more thanone new reference source increased 9 percentage points(see Fig. 3). This suggests that library employees weregoing beyond the library catalog and showing our usersadditional resources. Most importantly, the percentage of

Figure 1Percentage of Complex Reference Questions

Answered in Less than Three Minutes

Figure 3Percentage of Patrons who Learned of at Least

Two New Reference Rources

patrons reporting ‘‘not enough’’ helps declined from 12percent to 3 percent (see Fig. 4). Patrons reporting ‘‘notenough’’ information also declined from 15 percent in2001 to 5 percent in 2003.

More Communication – The extra time spent with patronsand the extra sources consulted may explain the con-vergence of patron and staff reports of success. In 2001library employees reported that they found the requestedinformation 72 percent of the time. Patrons reported thatthey found exactly what they were looking for only 47percent of the time, a gap of 25 percentage points. In the2003 study this gap was narrowed to only 8 percentagepoints (see Fig. 5). This suggests that library staff weredoing a better job discovering and responding to thepatron’s ‘‘real’’ question.

Figure 2Percentage of Complex Reference Questions

Where Only One Source was Used

Same Physical Environment – The Arts and HumanitiesDesk at Penn State continued to experience a higher thanaverage percentage of questions where books were reportedas not on the shelf (see Fig. 6). To counteract this, webrainstormed ways of helping patrons find substitutes forwhat they were seeking. Although the percentage of booksoff the shelf essentially remained the same, the increase insatisfaction from 2001 to 2003 suggests that we weresuccessful in finding alternatives for our patrons (see Fig. 7).

More Success – Ultimately the improvements reportedabove are of little intrinsic value, unless they also lead toincreased reference accuracy and patron satisfaction. In theArts and Humanities Library we found that making thebehavior modifications outlined above correlated withbetter service ratings. Patrons with complex questions in2003 reported they found ‘‘Exactly or approximately whatwas wanted and were satisfied’’ 83 percent of the time.This was an increase of 19 percent from the earlier survey.The same service point, with essentially the same staff, andongoing facility issues, went from performing 6 percentbelow the WOREP average to 13 percent above theWOREP average on this measure of success (see Fig. 7).

Figure 4Percentage of Patrons Reporting ‘‘Not Enough Help’’

May 2007 387

Figure 5Librarian and Patron Reports of Success On

Complex Reference Questions

Figure 7Percentage of Patrons who ‘‘Found Exactly or

Approximately What They Wanted andWere Satisfied’’ on Complex Reference Questions

At the Music and Media Desk, reference performance alsoimproved. During the 2003 study, users reported that theyfound exactly the specific- or known-items they were seeking62 percent of the time, up 4 percentage points from 2001 (witha national average of 60 percent). For the more complexreference questions, users found exactly or approximately whatthey wanted, and reported they were satisfied, 83 percent of thetime, up 11 percentage points from 2001 (national average forlibraries: 70 percent).

CONCLUSION

Based on our experiences, the authors believe that theWOREP, developed in the pre-Internet 1980s, is still relevantin today’s reference environment. The WOREP results helpedidentify factors within the Arts and Humanities Library thathindered reference success. Analysis of user feedback allowedus to focus our training efforts, and ultimately, improve ourperformance.

Figure 6Percentage of Questions Where‘‘Books not on Shelf’’ Reported

388 The Journal of Academic Librarianship

‘‘Based on our experiences, the authors believethat the WOREP, developed in the pre-Internet

1980s, is still relevant in today’s referenceenvironment.’’

While our experience was positive, libraries considering theWOREP should be aware of limits to our study. Although weused the same survey instrument, the two studies wereconducted at different times, with different patrons and aslightly different staff. Given these and other variables, wecannot ‘‘prove’’ that our increased success rates are attributableto modifications we made in training. It must be conceded thatthe data provided are not sufficient to allow statisticallysignificant comparisons based on a few percentage points ofdifference. We are reassured by the uniformly positive move-ment of the numbers in the 2nd study, suggesting that thechanges were not a mere random fluctuation in the data. Otherpossible factors include that staff may have taken the studymore seriously the second time around. While this may explainsome of the improvement, it does not diminish the value of theWOREP. Even if some staff consciously tried harder during thesecond study, we still view the results as a useful reminder ofbest practices. The second WOREP study results dramaticallydemonstrated the positive effects that could be achieved byfocusing on best reference practices.

It is hoped that Arts and Humanities service staff haveinternalized these best practices as a result of participation inthe study. Among the research questions to explore in anysubsequent study, is whether the positive effects observeddiminish over time. Have library employees absorbed thelessons learned and incorporated them into their long-termhabits? The authors hope a future WOREP survey in the Artsand Humanities Library will help explore these and otherquestions.

APPENDIX AWOREP SURVEY AS USED BY THE MUSIC LIBRARY ASSOCIATION

Survey Part 1: Completed by the Patron

May 2007 389

Part 2: Completed by Library Staff

APPENDIX BARTS AND HUMANITIES ‘‘BEST PRACTICES’’ DOCUMENT

B.1. Reference Service Goals

Spend more time with each reference question. (Referencequestions answered in less than three minutes are more likelyto result in unsatisfied users. We reported spending less thanthree minutes on the majority of our reference questions.)

Refer users to more than one resource. (Users referred to onesource were less likely to be satisfied than those given more

than one option.)

390 The Journal of Academic Librarianship

Provide hands-on help. Walk the user through the searchprocess, rather than merely suggesting sources. (Users whoare involved in the process are more likely to rate theencounter positively, and are more likely to be satisfied withthe results.)

Follow-up. Find out if the patron really found what theywere looking for. (Users often reported not finding what theyneeded even when staff felt they answered the question.Always double-check to make sure you have fully met thepatron’s needs.)

B.2. Behaviors for Successful Reference Encounters

B.2.1. Be Approachable

1. Avoid appearing immersed in work/desk activities. Scan thearea periodically to see if anyone nearby appears in need ofassistance.

2. Make eye contact with everyone who walks by or near thedesk.

3. Ask ‘‘May I help you?’’ as patrons approach.

B.2.2. Explore What the Patron Needs Before Answering

Ask clarifying questions:

1. What types of sources are you looking for? (articles, books,encyclopedia, Internet, etc.. . .)

2. Are you working on a class project? May I see theassignment?

3. How much information do you need? (How many pages isyour paper?)

4. Where have you looked so far? What searches have youalready tried?

5. Listen to the entire question. Do not jump to conclusions.

6. Repeat and rephrase to ensure that you understand. ‘‘So youneed to find books on the Russian Revolution, right?’’

B.2.3. Include the Patron in the Process

1. Be certain the patron understands what you are doing forhim/her. Narrate as you think and type, explaining what youare doing and why.

2. Assist patrons at the public workstations rather thanremaining behind the desk. Encourage the patron to logon to the workstation themselves. Have them do the typing,while you guide them.

B.2.4. Consult Multiple Sources

1. Brainstorm a variety of possible sources. Consider howarticle databases, subject encyclopedias, subject guides, andWeb sites may be helpful. Write sources down as you thinkof them. Convey the range of options to the patron, and helpthem get started. Provide a written list of resources to thepatron for future consultation.

2. Remember the subject guides on the Web, and resources inother subject libraries. Explore resources in libraries outsidePenn State (e.g. WorldCAT).

B.2.5. Navigating the Floor/building

1. Get up from behind the desk and walk with patrons, ratherthan pointing out locations.

2. Direct patrons to the side stairs of the stacks, rather thanthe central ‘‘core’’ stairs, if their call number warrantsthis.

B.2.6. Referrals

1. Inform patrons of the availability of subject specialists.Assist patrons needing additional expertise in sending an e-mail to the appropriate subject specialist.

2. Avoid hasty referrals to other locations. Call ahead to theother location to make certain they have what is needed andwill be expecting the patron.

3. Note interesting or difficult questions in the topic column ofthe Reference Statistics sheet, or in the desk notebook.

B.2.7. Follow-up With the Patrons

Use questions/comments such as:

1. Does this answer your question? Does this look like whatyou need?

2. Please be sure to let me know if you do not find what youneed. I will be here at this desk.

3. Please come back if you don’t find what you need, and wewill try other sources.

4. If this turns out not to be what you were looking for, let meknow. We have many other resources and librarians who areexperts in this area. I can get help from them if needed.

5. When possible, approach patrons as they leave. Ask if theyhave found everything they need.

NOTES AND REFERENCES

1. ‘‘The Reference Assessment Manual,’’ Ed. American LibraryAssociation, Evaluation of Reference and Adult ServicesCommittee. Ann Arbor, MI: Pierian Press, 1995.

2. Terence Crowley and Thomas Childers, Information Service inPublic Libraries; Two Studies. Scarecrow Press, Metuchen, NJ,1971.

3. Peter Hernon and Charles R. McClure, ‘‘Unobtrusive ReferenceTesting: The 55 Percent Rule,’’ Library Journal 111.7 (1986):37–42.

4. Joan C. Durrance, ‘‘Reference Success: Does the 55 Percent RuleTell the Whole Story?,’’ Library Journal 114.7 (1989): 31–36.

5. Patricia Dewdney and Catherine S. Ross, ‘‘Flying a Light Aircraft:Reference Service Evaluation from a User Viewpoint,’’ RQ 34.2(1994): 217–231.

6. Catherine S. Ross and Patricia Dewdney, ‘‘Negative Closure,’’Reference and User Services Quarterly 38.2 (1998): 151–164.

7. Marjorie Murfin, ‘‘Assessing Library Services’’ in TheReference Assessment Manual. Ed. Evaluation of Referenceand Adult Services Committee. (Ann Arbor, MI: PierianPress, 1995), 11–12.

8. Marjorie E. Murfin and Gary M. Gugelchuk, ‘‘Development andTesting of a Reference Transaction Assessment Instrument,’’College & Research Libraries 48.4 (1987): 314–338.

9. Ibid.10. For additional details see the WOREP Web Site, http://worep.

library.kent.edu/index.html.

May 2007 391

11. John C. Stalker and Marjorie E. Murfin, ‘‘Quality ReferenceService: A Preliminary Case Study,’’ Journal of AcademicLibrarianship 22.6 (1996): 423–429.

12. Janet D. Brown, ‘‘Using Quality Concepts to Improve Refer-ence Services,’’ College & Research Libraries 55.3 (1994):211–219.

13. Beth Christensen, Mary Du Mont, Alan Green, ‘‘TakingNote: Assessing the Performance of Reference Service in Aca-demic Music Libraries A Progress Report.’’ Notes 58.1 (2001):39–54.

14. Ibid, Footnote – see Appendix A for a copy of the versionused. The original WOREP survey instrument and documen-tation of its testing for validity and reliability were publishedin Marjorie E. Murfin and Gary M. Gugelchuk, ‘‘Develop-ment and Testing of a Reference Transaction AssessmentInstrument,’’ College & Research Libraries 48.4 (July 1987):314–338.

15. For examples see: Gatten, Jeffrey N. and Carolyn J. Radcliff,‘‘Assessing Reference Behaviors with Unobtrusive Testing,’’ InLibrary Evaluation: A Casebook and Can-Do Guide. Ed. Danny P.

392 The Journal of Academic Librarianship

Wallace and Connie Van Fleet. (Englewood, CO: LibrariesUnlimited, Inc., 2001), 105–115;Virtual Reference Desk. ‘‘Facets of Quality for Digital ReferenceServices, Version 5.’’ June 2003. Available: http://www.vrd.org/facets-06-03.shtml;Gers, Ralph and Lillie J. Seward. ‘‘Improving Reference Perform-ance: Results of a Statewide Study.’’ Library Journal, 110.18(1985): 32–35; Whitlatch, Jo Bell. ‘‘Reference Service Effective-ness.’’ RQ, Winter 1990. 205–220;Ross, Catherine Sheldrick and Patricia Dewdney. ‘‘NegativeClosure: Strategies and Counter-Strategies in the ReferenceTransaction.’’ Reference & User Services Quarterly, 38.2 (1998):151–163.

16. Texts discussed were: Dewdney and Ross, Flying a Light Aircraft;Durrance, Reference Success;Gers and Seward, Improving Reference Performance; andReference and User Services Association, Guidelines for Behav-ioral Performance of Reference and Information Services Profes-sionals, http://www.ala.org/ala/rusa/rusaprotools/referenceguide/,2004.