Evaluating the Contribution of Library Instruction to the Quality ...

Engaging Conversation 545

545

Engaging Conversation: Evaluatingthe Contribution of Library Instructionto the Quality of Student Research

Mark Emmons and Wanda Martin

Mark Emmons is an Assistant Professor and Head of Instruction Services at the General Library, Univer-sity of New Mexico; e-mail: [email protected]. Wanda Martin is Associate Professor of English and anAssociate Dean in the College of Arts & Sciences at the University of New Mexico; e-mail:[email protected]. The authors would like to thank Elizabeth Malloy, graduate statistics consultant, forassisting them with their statistical analysis.

The study discussed in this article compared research papers beforeand after implementation of an inquiry-based library instruction programto assess the program’s effectiveness and consider its future develop-ment. The new program appears to have made a small difference in thetypes of materials students chose and how they found them. Little changewas seen in how students used their sources. The evaluation has fos-tered increased collaboration between the library and the English de-partment, and suggests that the programs can be further improved byproviding more training for instructors and placing greater emphasis onthe rhetorical approach to research.

his study was designed to ex-amine the impact of a new li-brary instruction program onthe research processes of stu-

dents in a first-year writing course. Threeyears after replacing a tool-based libraryinstruction program with one based oninquiry, the authors ask what differencethe new program makes and how the nextsteps can be designed to help students de-velop information literacy.

In 1998, the library instruction programfor first-year students at the Universityof New Mexico consisted of a series ofworkshops designed to teach users howto use research tools such as the onlinecatalog, the most-used online indexes,and the Web. The program was a remnantof the early days of publicly available in-formation technology when many stu-

dents, staff, and faculty, familiar with thecard catalog and printed indexes, wereeager to learn how to use the new elec-tronic versions of these tools. However,interest ebbed and these tool-based work-shops were sparsely attended. Studentswho came to the university already famil-iar with computers assumed they couldfigure out how to work the systems ontheir own. Except where faculty requiredattendance, the library’s certificate ofcompletion had little appeal. And becausemore students used the Internet to do re-search before they came to the university,they saw little need for the tools the li-brary could offer. These students came tocampus highly computer literate, not re-alizing they were information illiterate.

Across the nation, librarians had no-ticed similar trends. In 1989, the ALA’s

546 College & Research Libraries November 2002

Presidential Committee on InformationLiteracy issued a report explaining theimportance of information literacy to de-mocracy in an information age.1 The re-port defined information-literate peopleas “those who have learned how to learn.[T]hey know how knowledge is orga-nized, how to find information, and howto use information in such a way that oth-ers can learn from them.” Librarians be-gan to reduce emphasis on research toolsand focus more on research processes andcritical thinking, eventually publishingthe Information Literacy CompetencyStandards for Higher Education in Janu-ary of 2000.2

It was in this context, with the avail-ability of information resources outpac-ing the students’ ability to sort and evalu-ate them, that the library approached theEnglish department with a proposal tohave each English 102 class formally par-ticipate in a library instruction program.Although only a few hundred feet of openplaza separate the library and the Englishdepartment, initiating this discussion wasa bold move.

The University of New Mexico is apublic, doctoral-extensive university ofabout 25,000 students. The freshman classnumbers about 2,500 each year, mostlytraditional-age students recently gradu-ated from high schools located in the state.Every undergraduate must complete En-glish 102, Analysis and Argument, to earna degree. This course is the second in atwo-course composition sequence de-signed to help students develop skills forreading and writing in academic contexts.It requires a researched essay in whichstudents make and support an argumentusing sources other than the assignedcourse readings.

At the time the library made its pro-posal, some instructors still required theirstudents to attend the library workshops,

but English 102 faculty, mostly graduateinstructors, found the workshops to be ofminimal value. The department’s Fresh-man English Committee, already in theprocess of redesigning the English 102curriculum and adopting new texts,readily agreed to help rethink the libraryinstruction program as part of an effortto make the course more effective.

The committee and the library facultyagreed that students needed improvedresearch skills. Students were coming tothe university with minimal library re-search experience, often having writtenonly a single research or term paper inhigh school English. They did not knowhow to locate materials in a large libraryand typically lacked skill in evaluating thecredibility and biases of sources and incomparing and synthesizing materialfrom more than one point of view. Al-though most of the English 102 instruc-tors were fairly adept at performing re-search in their own fields of literature orrhetoric, they were not necessarilyequipped to teach library research strate-gies for the broad range of topics typicalof freshman writing.

What freshman English needed was alibrary program that would support itsemphasis on writing as an inquiry pro-cess. Students should begin their researchwith a question or a thesis. In addition,they should use periodicals as well asbooks and other resources to focus theirquestions more productively and to dis-cover possible answers. Moreover, theyshould take a critical stance towardsources, regarding authors as rhetoricallysituated contributors to the conversationabout an issue (i.e., people with an inter-est and a point of view) rather than as dis-interested sources of facts to be reported.They should understand and use sourcesrhetorically to explore new ideas and toinform and support their own claims.

Working from early versions of infor-mation literacy goals and a 1993 articleby Barbara Fister, the library facultyshared this perspective on research,which encourages students to become apart of the formal conversation being con-

What freshman English needed wasa library program that wouldsupport its emphasis on writing asan inquiry process.


ducted in the literature about a particu-lar issue.3 An instruction librarian with abackground in literature, Fister noted thatmuch library instruction emphasizes “in-formation retrieval as the purpose of re-search,” when, in fact, “most college-levelassignments demand an idea that is de-veloped, argued, and supported with evi-dence.” She argued for a “rhetorical”stance toward sources in terms almostidentical to those used in English depart-ment documents. The new program de-signed to meet these needs was launchedin the spring of 1999.

The library lesson plan for English 102puts library tools in the context of inquiryby organizing the library instructionaround the questions students bring fromtheir classrooms. The lesson builds on aframework proposed by Carol C.Kuhlthau, which described “intellectualaccess to information and ideas, and theprocess of seeking meaning.”4 InKuhlthau’s six-stage model, students ini-tiate the research process by exploringpossible topics and developing a questionthey would like to answer. They becomebetter informed about the field of studyby gathering broad background informa-tion from materials such as encyclopedias.As they formulate more-focused ques-tions, they seek more-specific sources ofinformation, developing more-sophisti-cated search strategies to find what theyneed. Thus, rather than learn to use theonline catalog and an index simply for thesake of becoming familiar with thosetools, students do so in the service of areal quest for information to increase theirunderstanding. The lesson extendsKuhlthau’s model by addressing distinc-tions such as the one between popularmagazines and scholarly journals, invit-ing students to consider where the con-versation is taking place and the kinds ofexpertise that are involved. An outline ofthe lesson is shown in figure 1.

Of course, students also need to learnthe practical skills that will enable themto find the information they need to un-derstand the conversation and participatein it. These skills include the nuts and

bolts of searching the online catalog andthe EBSCOhost Academic Search Elite in-dex (hereafter, EBSCO). The lesson in-cludes a section on simple Boolean logic.Although students have only brief oppor-tunities for hands-on work with their owntopics during the session, the library of-fers an optional second lab in which stu-dents can spend the entire class periodresearching their topics with the assis-tance of a librarian. This enables studentsto practice what they have learned withsomeone who can help them deal withuncertainty, refine their search strategies,and evaluate what they find. Both ses-sions still teach the use of library researchtools but focus attention on the researchcontext.

In the spring of 2001, the authors setout to evaluate the new program’s effec-tiveness by comparing research paperswritten before and after it began. Despiteearlier assessments in which it wasfound that English 102 instructors andstudents overwhelmingly valued the li-brary instruction sessions, the authorswere not satisfied because they had nottested student skills. A survey of the lit-erature by librarians Lois M. Pausch andMary Pagliero Popp5 revealed that theauthors were not alone. In their survey,Pausch and Pagliero found that few stud-ies measure student learning and thatmost assessment of library instruction isrelatively informal, often consisting ofsurveys of student satisfaction. Theyfound that although “the emphasis ofmany programs has changed from toolsto concepts … few programs haveadopted methods aimed at assessingwhether students gained the cognitiveskills for analysis, synthesis, and evalu-ation of information that form the basisfor much of the assessment of highereducation.” Although formal assessmentfor program improvement is increas-ingly prevalent in higher education, andincreasing attention is paid to the “as-sessment culture” of libraries, little re-search has been published that uses stu-dent research papers in a nonlibrarycourse in a rigorous way to measure the


effectiveness of library instruction. Theauthors thought it would be useful toadapt the freshman writing program’sexisting outcomes assessment protocolso as to conduct a careful assessment notonly of what students heard in their li-brary instruction sessions, but also ofhow they applied that learning.

MethodsThe PortfoliosStudents in English 102 complete theirsemester’s work by submitting a portfo-lio that includes two revised papers. In-structors emphasize the revision process,encouraging students to submit the verybest work they can do. In most sections,

FIGURE 1Outline of English 102 Lesson Plan

ASSIGNMENTStudents in English 102 write a researched essay in which they make and supportan argument using sources other than the assigned course readings. In additionto helping students develop skills for reading and writing academically, the pa-per is designed to involve students in the discussion taking place in the scholarlyliterature, understanding authors’ contributions as rhetorically situated and sub-ject to evaluation.

While introducing basic research process and tools, the library becomes a part ofthe rhetorical process in which students make their paper part of formal conver-sation going on in books and journals. An example based on the reading takesstudents through the research process. There is time for students to begin theirresearch and a second optional lab session where they can continue researching.The lesson includes the following:

RESEARCH PROCESS• Choose a topic.• Develop a research question or a working thesis.• Gather background information, books, and articles.• Evaluate what you find.

FINDING TOOLS• Reference sources• Library catalog• Multidisciplinary index (EBSCO)

KEY CONCEPTS• Keyword versus subject searching• Boolean logic• Scholarly journals versus popular magazines

NAVIGATION• How to access• How to locate• Services available, including help with research

SAMPLE RESEARCH QUESTIONHow does the image of women in the media affect what women believe aboutthemselves?


this portfolio accounts for half the semes-ter grade, so students generally revisewith care. Portfolios are evaluatedcollaboratively by small groups of instruc-tors. Since the fall of 1996, the freshmanEnglish program has conducted an out-comes assessment project designed to dis-cover how effectively instructors teachcollege composition skills and to helpthem make the program more effective.

During each semester’s grading pro-cess, the authors randomly selected un-marked portfolios from each gradinggroup and photocopied them, winnow-ing the sample to twenty-five from eachcourse (about two percent of the studentstaking the course each year). Thus, in theten semesters from the fall 1996 throughthe spring of 2001, 250 portfolios weregathered for analysis from students com-pleting English 102.

For the purpose of outcomes assess-ment in freshman English, a group of in-structors reads the sample portfoliosagainst an analytical rubric that asks, withrespect to each of the objectives of English102, “What evidence do I see that the au-thor of this paper controls this particularskill?” The features assessed include thepresence of an arguable thesis, the use ofsource texts to support that thesis, theoverall organization of the argument, thedevelopment and coherence of para-graphs and sentences, and the control ofconventional usage and mechanics.

This kind of rating scheme, using arubric to describe the qualities sought inthe object, is well established in the field.The best-known uses of such rubrics arein scoring free responses on standardizedtests such as the SAT II or AdvancedPlacement Examination. In those contexts,where the purpose is to rank students’work for college admission or credit, re-sponses are scored relative to one anotherand rated holistically for overall qualityin terms of several key characteristics. Incontrast, an analytic reading looks at eachobject separately, not as a relative perfor-mance. And rather than give a single scorefor “overall quality,” it assigns a value tothe performance of each characteristic

rated. Thus, instead of asking how goodone is compared to another, the readerfocuses on specific qualities: “On whatskills do students do well? On which onesare they weaker?” After six readers haverated each paper, some time is spent de-briefing, comparing notes on what hasbeen seen, and considering how practicesmight be shifted to make the programwork better for students. This program isdiscussed more fully in Martin.6

The 250 portfolios on file, equally di-vided between those completed beforeand after the library instruction programwas developed, provided a rich source ofdata about the students’ information lit-eracy and how the library instruction pro-gram may be influencing it.

The RubricThe first step was to design a rubric thatwould allow examination of studentwork for the qualities the authors wantedto study and to assign them numericalvalues so that performances could becompared over time. The authors beganby reading together samples from the col-lected portfolios, deciding to work withonly the researched essay from each port-folio that provided the most consistentand unambiguous examples of studentsapplying their library research skills.

Next, the authors sought to identify thefeatures of interest by exploring parallelsand divergences between the ACRL Infor-mation Literacy Content Standards for HigherEducation7 and the course objectives thatdrive English 102. The authors looked atthe analytic rubric for English 102 out-comes and corresponded with DennisIsbell, a presenter at an ACRL workshopon developing rubrics for library research.8

They also looked at several examples ofrubrics for assessment of research and bib-liographies posted on the Web, including

If the study was to be of interest totheir colleagues in the library andthe English department, the authorsalso needed to look at the uses towhich students put their sources.


the Lowell Research Cycle Rubric, scoringcriteria developed by Kim Ranger for awriting course, and a rubric designed byKeith Wetzel to analyze literature re-views.9–11 The authors first decided to de-scribe the students’ bibliographies in termsof number of citations and variety in for-mat, source and time frame of the infor-mation, and accuracy of the citations, andthen to evaluate the writers’ use of sources.

It was hypothesized that students whohad taken the library instruction programwould use more varied sources and, giventhe emphasis on entering an ongoing con-versation, perhaps more current sources.Because the distinction between popularmagazines and professional or scholarlyjournals is an important part of the pro-gram, the authors expected to see morescholarly and professional sources inpostprogram bibliographies. And becausestudents can write on many subjects, theprogram featured EBSCO, which is amultidisciplinary full-text magazine andjournal article index. It was thought thatthe frequency of sources indexed inEBSCO might provide an index of the ef-fect of the instruction program on stu-dents’ research practices.

However, student research is not justsearching for sources. If the study was tobe of interest to their colleagues in the li-brary and the English department, theauthors also needed to look at the uses towhich students put their sources. Thiswould mean reading the essays carefullyto assess how judiciously students chosesources to support their research pur-poses, how well they understood thesources’ arguments and limitations, andhow effectively they engaged the sourcesto construct their own meaning. Tackingback and forth between the ACRL Stan-dards and the English 102 objectives, aswell as mining their own experienceswith student research (including theirown college research experiences), theauthors drafted a matrix-style rubric thatdivided each of these categories, whichthey called Relevance, Credibility, andEngagement, into three qualities theyhoped to see in successful papers and as-

signed a numerical value to each of fourdegrees of success they thought it wouldbe possible to identify.

To test this rubric, the authors indepen-dently scored five sample essays chosento represent the range of final gradesearned in English 102. Then, in a long andlively discussion, they compared the as-sessments in order to reach agreement onwhat marked each level of success on afeature, to flesh out the descriptions ofeach feature and level, and to sort outoverlapping categories. The final versionof the rubric is shown in figure 2.

The Evaluation ProcessThe authors wanted to describe quanti-tatively all the bibliographies at their dis-posal to develop the most detailed under-standing of how the library instructionprogram may have influenced students’research practices and to provide abaseline of what the students are doingnow for comparison as the programs con-tinue to evolve. But it would not be fea-sible to conduct the more-detailed read-ings for Relevance, Credibility, andEngagement on all 250 samples, so the au-thors randomly selected thirty sampleseach from the pre- and postlibrary in-struction pools. A research assistant useda random number table to select the sixtysamples for close analysis, giving themreference numbers that disguised theirplace in the portfolio sequence so thatreaders would not be biased by knowingwhether they were “before” or “after.”

Three senior teaching assistants in theFreshman Writing Program were chosento assist with the scoring. Each reader wasan advanced doctoral student in Englishwho had taught English 101 and 102 forseveral different semesters and partici-pated in the outcomes assessment read-ings. This made the task more manage-able and gave the authors the opportu-nity to discuss with three knowledgeablereaders their perceptions of the studentwork, greatly enriching the analysis.Moreover, this advanced the program’sagenda of continuous assessment and im-provement because the readers, after ex-


From Library:Did the class come to the library?

¨ YES ¨ NODid the class take the second optionallab session? ¨ YES ¨ NO

FIGURE 2The Rubric

English 102 Portfolios�Research Paper Rubric READER _________Fall Semester, 2001Portfolio Number: __________English 102 section: __________Fall Spring SummerCourse Grade __________DESCRIPTIONFormatTotal Textbook Reference Book Scholarly Popular Web Other

or Reading Journal Magazine

List total number of citations followed by total number of each format.Journals and MagazinesEBSCO index EBSCO full text Total journals/magazines

List number indexed in EBSCO and number available in full text. List total journals andmagazines cited.Time Frame

3 2 1

Within five years Five to fifteen years old More than fifteen years oldList number published in each time frame.Accuracy

3 2 1All references in Includes identifying information, Insufficient or incorrect

standard MLA style with errors in format information, frequent errorsSee following pages for relevance, credibility, and engagement scores.

Content Currency Level TotalRelevance

Authority Summary Purpose TotalCredibility

Evidence Challenge Meaning TotalEngagement


AuthorityRecognizesauthor�sauthoritySummarySummarizesmain idea

PurposeAcknowledgesauthor�s biasand purpose

Identifies eachauthor�scredentials � andthey are relevantSummarizesmain idea of eachsource and makesexplicit connec-tion with theargumentExplicitlyacknowledgesthe purpose orbias of eachsource

Identifiesauthor�scredentials

Summarizesmain idea ofmost sources

Implicitlyacknowledgesthe purpose orbias of eachsource

Provides limitedinformationabout authors

Summarizesmain idea ofsome sources

Acknowledgesthe purpose orbias of somesources

Provides noinformationabout authors

Does notsummarize mainidea of sources

Does notacknowledge thepurpose or biasof sources

CREDIBILITYStudent cites authority of author, summarizes main ideas, and recognizes bias.

4/Excellent 3/Adequate 2/Limited 1/Poor

ContentSource relevantto topic

CurrencyTime frameappropriateto topic

LevelLevel andvariety appropri-ate to purpose

All sourcesclearly related totopic.

All sourcespublished inappropriate timeframe

All sourceswritten at levelappropriate topurpose

Most sourcesclearly related totopic.

Most sourcespublished inappropriate timeframe

Most sourceswritten at levelappropriate topurpose

Some sourcesclearly related totopic.

Some sourcespublished inappropriate timeframe

Some sourceswritten at levelappropriate topurpose

Sources un-related to topicor relevanceunclear.No sourcespublished inappropriate timeframe; Noattention at all totimeNo sourceswritten at levelappropriate topurpose

FIGURE 2The Rubric (continued)

RELEVANCE/SIGNIFICANCEStudent selects information that addresses the thesis or question and that helps analyze aproblem or propose a solution

4/Excellent 3/Adequate 2/Limited 1/Poor


FIGURE 2The Rubric (continued)

ENGAGEMENTStudent constructs knowledge and makes meaning of information by entering into aconversation and arguing with sources.

4/Excellent 3/Adequate 2/Limited 1/PoorEvidenceSupportsarguments withevidence

ChallengeChallenges ideasand recognizesmore than oneside to an issue

MeaningConstructsmeaning orpresents originalidea

Elaborates andextends bysupportingarguments withevidencethroughoutPaper as a wholeacknowledgesmore than oneperspective;criticizes ownperspectiveAnalyzes andsynthesizes;provides newinterpretationsof old ideas;draws connec-tions

Does notquestion thevalidity ofsources.

Acknowledgesmore than oneperspective.

Interprets andevaluates;compares andcontrasts ideas;assesses valueof ideas

Quotes sourceswithoutcomment andwithout criticalunderstanding orevaluation.Acknowledgesonly ownperspective anddenies validityof otherperspectives.Understands,interprets, anddiscusses ideas

Makes unsup-ported assertionsthroughout withlittle or no effortto cite sources

Does notacknowledgemore than oneperspective

Presents nooriginal ideas;shows poorunderstanding

amining the students’ work through thisdifferent lens, had the opportunity toshare their observations with colleaguesin the program’s next teaching orienta-tion.

Working as a team, the authors con-ducted readings to generate data. First,they completed the Description portionof the rubric for each of the 250 essays.Next, they met for four hours on each offour days to complete the Relevance,Credibility, and Engagement analyses.They began this reading by scoring asample from outside the pool and dis-cussing it to arrive at a reasonable con-sensus on how to apply the rubric. Eachessay then was scored independently bythree readers. After the first of thesereading sessions, the authors discussedhow well the rubric facilitated the scor-ing and agreed to two minor adjustments

in wording. The first, in the Relevancecategory, was to add “level” to the char-acteristic “variety,” recognizing that in astrong paper they were looking not onlyfor an appropriately expert and detailedtreatment of the topic, but also for arange of appropriate sources. In theCredibility category, under the charac-teristic “Purpose,” the authors changed“recognizes author’s bias” to “acknowl-edges author’s bias,” so as to assert thatthey sought an overt statement of evalu-ation.

ResultsThe rubric poses a number of questionsabout the quality of the essays, and withthe help of a consultant, the authors sub-jected most of the questions to statisticalanalysis. However, only a handful ofthese generated statistically significant


findings (as discussed below). The au-thors did not, for example, try to relatestudents’ grades in English 102 to theoutcomes of library instruction becausethe course grade includes many addi-tional factors. And because the optionalsecond “lab” visit is a recent innovation,it was too soon to measure its impact onmore than the most informal terms. Asmuch as anything, the authors sought toestablish a method and a body of baselinedata that would help them evaluate theprogram as it develops and changes.

Part I: Description of theBibliographyThe first part of the rubric described thebibliography of each paper in terms of thenumber of each format (textbook or classreading, reference, book, scholarly jour-nal, popular magazine, Web, or other),whether journal citations came fromEBSCO Academic Search Elite, the timeframe covered, and the accuracy of the ci-tations.

The pool of 250 yielded 223 usable pa-pers; the remaining twenty-seven portfo-lios included something other than a re-searched essay, such as an annotated bib-liography. Although there was no reasonto believe that the distribution would sat-isfy the normality assumption of a stan-dard symmetrical bell curve, the samplewas large enough to use a standard t-test.

Are There More Citations?The authors did not expect more citationsbecause most English 102 instructors limitthe total number of works that studentscan cite in their papers. The test deter-mined that there were no significant dif-ferences in total number of citations. (Seetable 1.)

Are There More Citations to Any Particu-lar Format?Because of the program’s emphasis onusing scholarly literature, it was antici-pated that students would cite morescholarly journal articles. And because ofthe instructor-imposed ceilings on thetotal number of citations, the authors be-lieved that students would cite fewerbooks as a result. The authors were righton the first count, but not the second. (Seetable 2.)

The increase in the number of journalarticles cited was the most significant dif-ference in the entire analysis. As the p-value for t-tests reveals, the increase incitations to scholarly journals is the onlystatistically significant change in format.(See table 3.) Students cited exactly thesame number of books before and after;books were, in fact, the most cited formatboth before and after implementation ofthe library instruction program. Becausethe total number of citations was not sig-nificantly larger, the authors wonderedwhether students were citing fewerworks in other formats. (Totals are shownin table 2.) Slightly fewer citations to theclassroom textbook and to referencebooks were found, but the only, even bor-derline, significant difference was unfor-tunately a reduction in the category“other.” Further analysis will be needed

TABLE 1Mean Number of Citations

Number Mean St. DevBefore 108 5.17 1.94After 115 5.17 1.76

TABLE 2Total Number of Citations by Format

Text Ref Book Journal Magazine Web Other TotalBefore 71 13 202 57 69 66 80 558After 55 11 202 113 87 83 49 600Total 126 24 404 170 156 149 129 1,158


to see what kinds of materials areincluded in the “other” category,but the informal impression fromthe readings is that many arenewspaper articles. In fact, aMANOVA test (multivariateanalysis of variance) to see whatcombination of formats had themost impact indicated that theincrease in scholarly journal ar-ticles cited, along with the reduc-tion in citations to textbooks and“other,” accounted for the largestdifference. (See table 4.)

These results suggest that thelibrary instruction program did bringscholarly journals to the attention of stu-dents. However, the persistence of booksas the most-cited category begs the ques-tion of what factors led students to makethat choice. A scan of the essays suggests

that topic may be one factor. Students writ-ing on historical and literary topics, for ex-ample, seem to have used more bookswhereas those with more contemporarytopics, such as body image or environmen-tal pollution, generally cited more journalarticles. This suggests, at least, furtherstudy to explore how a student’s topicchoice may influence his or her researchstrategies and choice of sources.

Are Articles More Recent?Because of the program’s emphasis onstudents entering an ongoing conversa-tion, the authors thought that students

might cite more recent articles than thoseused by their preprogram counterpartsbut found no significant difference in thetime frame of their sources. (See table 5.)

By far, the most-cited works were writ-ten within five years of when the studentwrote the paper. (See table 6.) Perhaps amore fine-grained analysis, using shortertime frames, will reveal more about stu-dent writers’ sense of what is appropri-ately current information for a giventopic.

Are Citations More Accurate?Because citation accuracy is not part of thelibrary instruction program and receivedno new emphasis from English 102 instruc-tors, the quality of citations was not ex-pected to improve. This category was in-cluded because it was a standard datapoint in the rubrics the authors examinedwhen they prepared their own and be-cause the authors felt this category mightprovide interesting data to inform futureteaching. There was no significant differ-ence in the accuracy of citations before andafter the new program. (See table 7.)

TABLE 3Mean Number of Citations Used

(Standard deviation in parentheses)p-value

Before After for t- testText 0.66 (0.97) 0.47 (.063) 0.089Ref 0.12 (0.38) 0.10 (0.32) 0.603Book 1.87 (1.76) 1.73 (1.70) 0.548Journal 0.53 (1.22) 0.97 (1.51) 0.016Magazine 0.64 (1.27) 0.76 (1.27) 0.491Web 0.61 (1.35) 0.72 (1.35) 0.566Other 0.74 (1.28) 0.43 (0.90) 0.036

TABLE 4Manova

Multivariate Analysis of VarianceTest Statistic f-value Degree of Freedom p-value

Wilk�s 0.93378 2.178 7, 215 0.037Lawley-Hotelling 0.07091 2.178 7, 215 0.037Pillai�s 0.06622 2.178 7, 215 0.037

… students do not have a stronggrasp of the kind of information abibliographic citation shouldprovide.


This did prove interesting, though,because an overall mean accuracy of onlytwo on a three-point scale was observed,suggesting that students do not have astrong grasp of the kind of information abibliographic citation should provide. Inscanning the essays to consider whybooks are such a prevalent category, theauthors observed a substantial number ofinaccurate citations that characterized, forexample, articles from anthologies or ex-cerpts included in course packs as books.

Part II: Relevance, Credibility, andEngagementThe second part of the rubric asked read-ers to read each paper and examine howstudents used the works they cited in theirbibliographies. The readers were lookingfor evidence of the rhetorical approach toresearch taught in the library lesson andthe writing program, and called the char-acteristics they sought to observe “rel-evance,” “credibility,” and “engagement.”Relevance is high when sources directlyaddress the student’s question, are timely,are written at a level the writer can use,and display variety suitable to the writer’spurpose. Credibility is high when a stu-dent acknowledges the author’s authority,bias, and purpose, and shows understand-ing of main ideas rather than merely min-ing for “quotes.” Engagement is highwhen the student supports argumentswith evidence, challenges sources’ ideas,recognizes more than one perspective onan issue, and constructs meaning or pre-sents original ideas.

Were the Sources More Relevant?It was thought that the sources used bythe students might be more relevant be-cause the library instruction program, in

concord with the writing pro-gram, focuses attention on formu-lating a question and entering thescholarly conversation around it.In retrospect, the rubric did notask readers to give higher scoresto the kinds of sources empha-sized in the instruction programbut, instead, to accept works

clearly related to the students’ topic andpurpose without regard to format or con-text of publication. And students had nodifficulty locating sources that readersfound relevant to their topics. On a 12-point scale, the mean relevance score ofpostprogram papers was 9.6, compared to9.1 before the new program. (See table 8.)

Although the difference in mean scoresis of only borderline statistical significance,it does suggest a trend toward greater rel-evance, especially upon examining thedistribution of scores. The boxplot in fig-ure 3 separates the scores into quartiles;the gray area shows the middle 50 percent,with the median score indicated by thehorizontal line. Before the program, themiddle 50 percent clustered between 8.5and 9.5, and the top 25 percent (verticalline) extended to 11. After the program, themiddle 50 percent clustered from 8.5 up to11, and the top 25 percent extended almostto 12. Interestingly, the bottom 25 percent(lower vertical line) of postprogram scoresranged much lower than those before pro-gram implementation.

TABLE 5Time Frame

0�5 years 5�15 years 15+ years NoneBefore 75 21 22 19After 80 28 24 11Total 155 49 46 30

TABLE 6Time Frame Most Used

BeforeTime Frame 0�5 5�15 over 15

years years yearsNumber 75 21 22Proportion 0.64 0.18 0.19AfterTime Frame 0�5 5�15 over 15

years years yearsNumber 80 28 24Proportion 0.61 0.13 0.10


The authors would like tothink that this means more stu-dents (and perhaps some in-structors) are getting the idea ofan inquiry-based research pro-cess. But the lower range sug-gests that some students may bemore confused than their pre-decessors were. Clearly, this re-sult will require further study.

Were Students More Aware of Sources�Credibility?In contrast to the relatively high relevancescores, credibility scores for both groupswere approximately 4.8 on a 12-pointscale. It was clear from the readings thatstudents, both before and after the newprogram, viewed the contents of sourcesas indisputable fact and made little effortto assess authors’ credentials or to exam-ine their biases. (See table 8.)

Were the Students More Engaged with theIdeas They Wrote About?Again, because the library instructionprogram asks students to engage in a

conversation with the authors of theworks cited, it was thought that engage-ment scores might increase slightly.However, mean engagement scores were6.60 before and 6.38 after the program, astatistically insignificant difference. (Seetable 8.)

ConclusionThe new library instruction program ap-pears to have made a small difference inthe types of materials students chose andhow they found them. Students usedmore scholarly journal articles, whichthey most likely found more efficientlythrough EBSCO than with the tools that

TABLE 7Accuracy

Mean Score on a 3-Point Scale (Standard deviation inparentheses)

p-valueBefore After for t- test

Accuracy 2.073 (0.742) 2.087 (0.615) 0.882

FIGURE 3Distribution of Relevance Scores


were previously available.Books continued to be themost-cited type of material.The relevance scores suggestthat students are developing asomewhat greater awarenessof the types of sources thatmight address their researchquestions. These results sug-gest that, as a group, studentsare able to understand, inter-pret, and discuss ideas butrarely rise to the level of evalu-ation, analysis, or synthesis. Apparently,neither the writing program nor the li-brary instruction program is thus far mak-ing much headway in helping studentsunderstand their source texts or their ownresearch efforts as rhetorically situated.

Implications for the LibraryThe analysis indicates that the new En-glish 102 program was a step in the rightdirection, primarily in that it fostered con-versation between the library and theEnglish department and revealed thecommon theoretical ground they share.

But smaller-than-expected differences inthe quality of student research call forseveral refinements. First, there is a press-ing need to address the consistency ofinstruction, especially as it relates to theconceptual framework that drives the les-son plan. Second, the lesson plan shouldmake better connections with students’actual research topics. Third, greater col-laboration is needed among English 102instructors and library faculty.

At the start of each semester, the headof instruction services meets with theEnglish 102 faculty to demonstrate thelesson and talk about effective library as-signments. To ensure that students aremotivated and focused when they cometo the library, the English 102 faculty are

encouraged to schedule their library in-struction classes at a time when studentshave developed a question they wouldlike to answer. But with more than fortysections in the fall and eighty in thespring, it is not always possible to sched-ule the class at the optimal time. In read-ing the papers, the authors noticed thattopics used as examples in the library in-struction program were more highly rep-resented. Although this may be an indi-cation of having chosen very obvious ex-amples based on the readings, it also sug-gests that some students might select top-ics for which the library instruction hasgiven them a head start on research. Itwould be interesting to experiment withtopic examples in the future to see if thisdoes indeed have an impact on studenttopic selection. In the meantime, however,greater emphasis should be given toscheduling the session at a time when stu-dents are ready to explore research ques-tions of their own devising.

The optional follow-up labs appear tobe a promising innovation that supportsand extends the library instruction pro-gram. This element was introduced in thespring of 2000, and in the followingspring, almost a third of English 102 sec-tions made a second library visit. The En-glish faculty praise the additional sessionsin which students can work on their ownprojects with the help of a librarian. Thelabs provide an opportunity for both theEnglish 102 and the library instructors tocomment on the relevance and credibil-ity of the resources that students locateduring class time. Although the current

TABLE 8Relevance, Credibility, and Engagement

Mean Score on a 12-Point Scale (Standard deviationin parentheses)

p-valueBefore After for t- test

Relevance 9.118 (0.897) 9.640 (0.31) 0.083Credibility 4.79 (1.19) 4.77 (1.31) 0.954Engagement 6.60 (1.16) 6.38 (1.49) 0.532

And the quite modest scores forEngagement indicate that studentscontinue to view the researchedessay as an academic exercise morethan as a quest for knowledge.


sample included too few students whohad this experience to meaningfully as-sess its impact on their papers, futurestudies will be able to do so.

The kind of collaboration that takesplace in the lab also should be formal-ized during the instructional design pro-cess. One way to do this might bethrough the English 102 readings: Eachclass selects one of five books that arechosen to raise the research questions.Library instructors could conceivablybecome members of the groups that ana-lyze the books and come up with themesto explore. They then could create anarray of examples for use in the class in-stead of the one per book that is usedcurrently. An even simpler step wouldbe to require library instructors to attendthe English 102 training.

Such a large program demands wide-spread participation of library faculty andstaff. Each semester, the head of instruc-tion services offers training and refresh-ers to those who will be teaching. With asmany as nineteen faculty and staff mem-bers participating, it is essential that thelibrary instructors provide a consistentlesson to each class. However, this is notalways easy to ensure. Because of staff-ing constraints, it also has not been pos-sible to consistently observe instructorsat work. Limited observation and infor-mal reports, along with the results of theauthors’ analysis, indicate that many in-structors continue to emphasize tools overprocess. Future training needs to empha-size the importance of the rhetorical ap-proach to research. Perhaps the most im-portant step will be sharing the results ofthis study—the collaboration that beganwith two program heads interested inassessing a program could very well endup pushing library faculty and staff intogreater collaboration.

Implications for the English DepartmentThe analysis so far suggests that the En-glish department needs to focus more at-tention on the purposes of the researchedessay. The low level of accuracy and com-pleteness of citations is not just a cosmetic

matter but, rather, indicates that studentsare not clear on the concept of document-ing source material. It looks as thoughmany students continue to work withbroad topics rather than with focused re-search questions. Moreover, from the in-frequency with which students questionauthors’ assertions or put them into con-text, the authors infer that, despite instruc-tor efforts, most students view sources asplaces to get facts to support their asser-tions and fail to take into account thesources’ rhetorical situation or to developone of their own. And the quite modestscores for Engagement indicate that stu-dents continue to view the researched es-say as an academic exercise more than asa quest for knowledge. The authors planto analyze the data further to gain moreinsights into how students approach re-search tasks and will take their insightsback to the instructors in conversationsabout curriculum and teaching. In Janu-ary 2002, when the preliminary results ofthis study were shared with English 102instructors, the instructors were particu-larly interested in the rubric as a devicefor teaching students and for evaluatingthe quality of their research. Many instruc-tors are themselves new to the idea of re-search as a rhetorical process and receivelittle training in specific strategies forteaching research. Perhaps the rhetoricalresearch process, in conjunction with therubric, might be used in future training offirst-time English 102 instructors. In addi-tion, the authors will use the results toframe some focus groups with students tohelp clarify why students make the choicesthey do and how they can be helped tochoose more discerningly.

Although much remains to be done,this study represents a good start towardimplementing a library instruction pro-gram that will support students in an in-quiry-based approach to research. Work-ing with each instructor to provide a li-brary session tailored to the interests of heror his students, at a time when studentsare ready to begin library work, has greatlyincreased communication between the li-brary and the English department.


Despite these gains, however, the au-thors are not satisfied that students arefinding sources relevant to their topics aslong as they show so little awareness ofthe other questions they should be askingand are so little engaged in the real intel-lectual work of evaluating and analyzingthe ideas they use to fill out their papers.The real significance of this project so farhas been discovering the convergence ofthe two programs’ theoretical perspectives.Traditionally, library instruction has beenviewed as a “service” to the writing pro-gram in much the same way that the writ-ing program provides “service” to thelarger curriculum. But the authors’ worktogether has shifted that relationship be-cause they realize that the ACRL’s infor-

mation literacy standards are remarkablysimilar to the standards of critical think-ing they try to impart in freshman English.

This realization leads to a shift fromasking how the library can provide bet-ter service to freshman English to askinghow the two programs can extend theircollaboration to ensure that all of theuniversity’s first-year students get a re-ally effective introduction to this ap-proach to research. And having startedthis conversation across the hundred feetof Smith Plaza, the authors wonder howit can be expanded across the whole cam-pus and the whole range of the liberalarts, moving these fundamental strate-gies of inquiry from the edge of the first-year curriculum to its center.

Notes

1. “American Library Association, Presidential Committee on Information Literacy, FinalReport.” Web page. Available online from http://www.ala.org/acrl/nili/ilit1st.html.

2. “Association of College and Research Libraries, Information Literacy Competency Standardsfor Higher Education.” Web page. Available online from http://www.ala.org/acrl/ilcomstan.html.

3. Barbara Fister: “Teaching the Rhetorical Dimensions of Research,” Research Strategies 11,no. 4 (Fall 1993): 211-219.

4. Carol Collier Kuhlthau, Seeking Meaning: A Process Approach to Library and InformationServices (Norwood, NJ: Ablex Pub Co, 1993).

5. Lois M. Pausch and Mary Pagliero Popp. “Assessment of Information Literacy: Lessonsfrom the Higher Education Assessment Movement.” Presentation at the Association of Collegeand Research Libraries 9th National Conference, Detroit, Michigan, 8-11 April 1997. Web page.Available online from http://www.ala.org/acrl/paperhtm/d30.html.

6. Wanda Martin. “Outcomes Assessment as a Teaching Tool,” in The Writing Program Ad-ministrator as Researcher, ed. Shirley K. Rose and Irwin Weiser (Portsmouth, NH: Boynton/CookPublishers, 1999), 40-51.

7. “Association of College and Research Libraries, Information Literacy Competency Standardsfor Higher Education.” Web page. Available online from http://www.ala.org/acrl/ilcomstan.html.

8. Kathleen Collins, Bee Gallegos, Dennis Isbell and Lisa Kammerlocher. “Assessing Infor-mation Competency.” Workshop at the Association of College and Research Libraries 10thNational Conference, Denver, Colorado, 17 March 2001.

9. “Lowell Research Cycle Rubric.” Web page. Available online from http://www.bham.wednet.edu/mod8low.htm, used to evaluate research projects at the BellinghamPublic Schools, WA.

10. “Ideas for Incorporating Information Literacy into Writing 150.” Web page. Availableonline from http://www4.gvsu.edu/infolit/CourseEgs/Eng150KR.htm, includes scoring cri-teria for an English class assignment at Grand Valley State University, MI.

11. “Rubric for Literature Review - Fall 2000.” Web page. Available online from http://www.west.asu.edu/kwetzel/EMC675/AnBioRu3.html, used to assess an Educational Mediaand Computers class literature review.

Evaluating the Contribution of Library Instruction to the Quality ...

Documents

Transcript of Evaluating the Contribution of Library Instruction to the Quality ...