Education REFORM

University of Masssachusetts Amherst School of Education 1

Education REFORMTen Years after the Massachusetts Education Reform Act of 1993

Education CONNECTION 2003

2

3

6

11

15

19

23

28

31

34

An Overview of Massachusetts Education Reform

Impact of the Education Reform Act on Equalizing Education FinanceLinda Driscoll

Bridges and Barriers: Equity Issues in High Stakes TestingJoseph B. Berger

Setting Passing Scores on TestsRonald K. Hambleton

Are MCAS Scores Comparable Over Time?Lisa A. Keller

VRROOM: The Virtual Reference RoomRobert W. Maloy, David Hart, Paul Oh, and Victoria Getis

School AccountabilitySusan Bowles

Managing to Lead in the Decade of Education ReformFrancis L. Gougeon

Capacity to Implement Education ReformKathryn A. McDermott

Conclusions: The Impact of Education Reform after Ten YearsAndrew Churchill

University of Masssachusetts Amherst School of Education

An Overview of

MASSACHUSETTS EDUCATION REFORMIn June 1993, Governor William Weld signed into law the Massachusetts Education Reform Act(MERA). MERA greatly increased the state role both in funding public education and in guiding thelocal educational process. The state’s role changed to incorporate setting curriculum frameworksand holding schools accountable for student performance. Because MERA was designed to be asystemic reform of education, all of the various state activities and policies needed to fit togetherinto a coherent whole based on state educational standards. This comprehensive set of reformsincluded the following elements.

Increased State Funding for Public Education. MERA doubled state funding of K–12 educationfrom $1.3 billion in 1993 to $2.6 billion in 2000. The achievement of this ambitious target has beena major accomplishment.

A “Foundation Budget” for All Districts. MERA laid out the concept of a minimum budgetnecessary for each district to adequately educate all of its students. Poorer communities that werespending below foundation-budget levels would receive more than those at or above thatthreshold. By 2002, all districts were at or above foundation level. Revision of the foundationformula to more fully reflect actual spending needs is under consideration.

Learning Standards. The legislation instructed the Board of Education to develop curriculumframeworks and other standards such as vocational standards, and to support local districts’implementation of standards through alignment of curriculum and instruction. Frameworks havebeen developed in arts, English language arts, foreign languages, health, mathematics, history/social science, and science/technology/ engineering.

Student Assessment. MERA required an assessment of student learning based on the frameworksand specified a competency determination as a requirement for graduation. The MassachusettsComprehensive Assessment System (MCAS) has been developed and implemented for thosepurposes. MCAS tests have been administered in multiple grades in English language arts,mathematics, reading, science/technology/engineering, and history/social science. Students in theclass of 2003 and beyond must perform at least at the “Needs Improvement” level in Englishlanguage arts and mathematics to graduate.

An Accountability System for School and District Performance. MERA required the state tohold schools and districts accountable for student performance and to provide remedies forpersistent underperformance. This has proven to be quite a challenge for the state, with severalchanges of jurisdiction occurring. Over the past year, a “performance index” process has beendeveloped based on MCAS scores and has been judged to be in compliance with the federal NoChild Left Behind Act.

Changes in Local Education Governance and Management. MERA included significantchanges in the way schools and districts are run. School Committees’ power over personnel issueswas reduced, with superintendents and principals given more authority. All schools were to haveSchool Councils composed of parents, teachers, students, and administrators.

Enhancing Educator Quality. A portion of state aid to local districts was earmarked for teacherprofessional development. Teacher licensure has been revised, and teacher tests for new teachershave been instituted.

Ensuring Readiness to Learn Through Early Childhood Education Programs. Between 1996and 1999, spending on early childhood education increased by 247 percent.

Implementing Choice and Charter Schools. MERA expanded inter-district choice andauthorized the state to approve charter schools—public schools exempt from local control andunion contracts.

2 Education Connection Magazine 2003

s a first timeschool superintendent in June1993, I remember theexcitement I felt as I learned ofa new funding formulaconceived through the recentlypassed Massachusetts EducationReform Act (MERA) of 1993.Designed to expand the state’srole in financing schools andestablish an adequate and fairper pupil spending requirement,this formula seemed to be agreat improvement over pastfunding mechanisms. Mycolleagues and I had hopes thatthis formula would improve thebudget making process. Prior toMERA, Chapter 70 funds fromthe state to support educationwere not predictable, adequate,or equitable. Determining themunicipality’s share could beboth confusing and political. This new method, althoughquite complex, set a minimumspending level for each district,determined the municipality’sability to pay, and designatedstate funds to make up thedifference. The per-pupil rateswere based upon 19components including suchareas as teachers’ salaries,support, administration,professional development,

books, and maintenance.Student enrollment was thenweighted by such factors aselementary school, high school,special education, bilingual, andlow-income populations.Therefore, districts with similarenrollment numbers could havevery different foundationamounts if their schoolpopulations contained largenumbers of bilingual and low-income students. Specialeducation funds, however, werenot based upon actual studentnumbers but upon prescribedpercentages. Three and a halfpercent was the assumption forin-district special education andone percent for students whoare tuitioned out. Finally, awage adjustment factor eitherincreased or decreased theamount designated to salarylines based upon thecommunity’s living standard.Thus, a district’s foundationamount was determined. Thetown’s ability to pay its sharewas determined by an evenmore complicated procedure. Itcontained 35 points amongwhich were included themunicipality’s equalizedvaluation, growth rate, percapita income, and excess debt.

This formula, althoughcomplicated, did set aminimum spending level foreach district based uponenrollment needs anddetermined what percentage ofthis need would be expectedfrom the community, basedupon its ability to pay. Thedifference would come from thestate. The basic points madesense, and the factors offoundation budget, minimumcontribution, and Chapter 70aid are clear. This formula, begun inFY94, was authorized for aseven-year period, scheduled toend in FY00. It has not beenreauthorized and debate overthe formula is ongoing fromsuch organizations as theMassachusetts Superintendents’Association, MassachusettsMunicipal Association, Leagueof Women Voters,Massachusetts Department ofEducation, MassachusettsAssociation of SchoolCommittees, the Governor’sOffice, and the FoundationBudget Review Commission.Since FY93, state aid toschools, commonly referred toas Chapter 70 aid, has grownfrom $1.3 billion to about $3.2

IMPACT OF THE EDUCATION REFORM ACT

ON EQUALIZING EDUCATION FINANCE

Linda Driscoll

University of Massachusetts Amherst School of Education 3

A

Equalizing Education Finance


billion in FY03. The foundationper-pupil expenditures rosefrom $5,577 to $7,201 in thatsame time span. These figuresrepresent a change in state aidto education from 28.5 percentin FY93 to 46.9 percent inFY03. Massachusetts is nowcloser to the national average of49 percent. In fact, theDepartment of Education’sbudget for all programsincreased an average of $285million each year during thefirst eight years of educationreform.1

So, has this large increase instate aid accomplished the aimof ensuring that the schools areadequately funded in anequitable way? The answer tothat question depends uponhow the goal is defined. If thegoal is to assist all schoolsystems to meet the foundationlevel of spending, then thattarget has been reached.However, if the goal is to ensurethat all schools have the level ofresources necessary to providethe kind of excellence incurriculum and instruction thatleads to all students beingsuccessful, then we still havemuch to accomplish. Thenotion of providing merelyadequate resources whileexpecting excellence from allstudents is central to much ofthe debate concerning schoolfinance in Massachusetts. Has the goal of reducingdisparity of per-pupilexpenditures across property-rich and property-poorcommunities been achieved?Although there are still somelarge discrepancies, with somericher communities spendingwell above their minimumcontribution, it is evident that

this gap has lessened. In FY01,two-thirds of districts spent 10percent higher than theirfoundation, 31 districts wereabove 150 percent, and twodistricts were approaching 300percent.2 Has the money beenfairly allocated to municipalitiesand regions? It is obvious thatthe poorer urban districts havebenefited greatly financiallyfrom school reform. However,the regional districts, because oftheir unique situation, havestruggled with disparities inallocations from sending towns.Also, some high effortcommunities work harderspending a larger percentage oftheir taxes for support of theschools. There are districtswhose state portion of theirexpenses have increaseddramatically, with the Chapter70 funds reaching as high as 99percent of the foundation.Other districts’ state aid hasremained very close to theirFY93 percentages, with somereceiving as little as 12 percentfrom the state. There is little doubt that theschools in Massachusetts, onthe whole, have benefited fromthe financial increases due toeducation reform, but can wedo better and can this formulabe more effective and fairer?Ambitious goals have been setby the state of Massachusettsthrough the curriculumframeworks, high standards,and high stakes accountabilityas measured by the Massachu-setts Comprehensive AssessmentSystem (MCAS). In order tocontinue to strive towards thosegoals and to enable all childrento acquire the proficienciesthese standards demand, thefoundation budget needs to be

raised. It simply is not adequateto reach these goals. Whether the presentfoundation is scrapped andbegun again from scratch ormodified in its presentcondition, several pertinentissues must be addressed. At thetop of this list is specialeducation. Simply put, it isunder funded. It also has takena disproportionate share of newfunds allocated througheducation reform. For 56percent of districts, specialeducation (SPED) increasesexceeded 50 percent of newstate aid, and for 29 percent ofdistricts, SPED increasesexceeded 100 percent of newstate aid from FY93 to FY99.3

The percentage of moneyallocated to SPED in thefoundation budget isinadequate. It needs to beadjusted based upon reality, notprojections. Also, the CircuitBreaker law, where the statereimburses instructional andtuition costs for high coststudents both in and out ofdistrict, should be fully funded.Student teacher ratios need tobe adjusted. The numbersdeemed effective nearly tenyears ago, when the educationreform law was written, are nolonger valid. Per pupil spendingcategories need to be changed.Elementary and preschoolallocations, in particular, areunrealistic. The population ofstudents in preschool containsmany at-risk students who needmore resources to provide fortheir programming needs.Teachers’ salaries need to befactored into the foundation atactual amounts. Enrollmentnumbers should be bettercomputed, especially in those

districts with high growth. Thislag in determining foundationnumbers results in budgetsbeing based on lower thanactual student counts. Atechnology factor needs to beadded to the foundation budget.The financial demands offunding technology haveincreased greatly since MERA’sinception. Low-income studentsshould be counted using moreaccurate data than free/reducedlunch counts that are gatheredthrough an application process.Money must be included toaddress the cost ofimplementing high standardsand accountability. The cost ofremedial and after-schooltutorials to ensure that allstudents meet these standards isnot implicit in the foundation.The foundation budget was notdeveloped with anunderstanding of the costs ofredesigning standards-basedcurriculum and re-provisioningschools for these changes. It is not within the scope ofthis article to discuss the manyissues regarding equity on themunicipal side. Because ofvagaries in the formula,economically similarcommunities can have widelyvarying minimumcontributions. Let it suffice tomention that many towns arestruggling to provide for theirstudents within the mandates ofeducation reform and thatadjustments are warranted. After almost ten years ofoperating schools with thefoundation budget, much hasbeen learned about the pros andcons of such a fundingmechanism. The goal ofassisting all schools to meetfoundation has been met. This

reform movement—spurred bythe McDuffey lawsuit,4

formulated through “EveryChild a Winner,”5 supported bymotivated lawmakers and agrowing economy—hasimproved the financialconditions in many schools.There is a greater understandingof what needs to be done toimprove the structure andfairness of the formula. But canit happen, and moreimportantly, will it happen?When revenues to the statedecline, as is now the case,Chapter 70 funds aresubsequently decreased. Eventen percent declines, as were therecommendations this past year,can be devastating tocommunities that receive up to90 percent of their funds inChapter 70 aid. Even if the statekeeps its commitment to fundChapter 70 to the degree thatschools stay at foundation, it isnot enough to accomplishexpectations, particularly nowwith the federal requirements ofthe No Child Left Behind Act. Ifthe increasing partnershipbetween towns and the stateslips backward, it is difficult toimagine that the municipalitiesand regions will be able toassume the increased costs. Itgoes without saying, however,that if the federal governmentwould increase educational aidover the average seven percentit now allocates, the financialpicture definitely would bebrighter. If the formula is reauthor-ized and funded adequately tomeet the above mentionedadjustments, Massachusettsstands to be well on its way tofulfilling the promise of “EveryChild a Winner.”

ENDNOTES 1 Memorandum to SchoolSuperintendents from David Driscoll,Commissioner of Education, October22, 2001. 2 Report of The Foundation BudgetReview Commission, June 2001. 3 Impact of Special Education onEducation Reform, Mass. SpecialNeeds Task Force, November 2001. 4McDuffey v. Secretary of theExecutive Office of Education, 415Mass. 545 (1993). 5 Entitled A Proposal for LegislativeAction Plan for Systematic Reform ofMassachusetts’ Public Primary andSecondary Educational System by theMassachusetts Business Alliance forEducation (MBAE), 1991.

Linda Driscoll (M.Ed. 1982;Ed.D. 1996) has taught PublicSchool Finance at the Universityof Massachusetts Amherst.Recently appointedSuperintendent for ErvingSchool Union 28, she previouslyserved as Principal at the WestStreet School in Granby,Massachusetts, and has priorexperience as a Superintendentof Schools and Special EducationDirector.


Equalizing Education Finance


ver the past severaldecades, the call for educationalreform and accountability hasbeen one of the most visiblepublic policy issues in thenation.The need for excellence ineducation is undisputed, yet thenature of the current wave ofeducational reform movementshas been the source of publiccontroversy and ongoing debate.Testing and accountability havebeen at the core of recent at-tempts to improve and reformeducation at K–12 andpostsecondary levels and serve aslightning rods for much of thecontroversy. While testing andaccountability are not new toeducation, the prominent rolethey play in the current reformera is unprecedented. Moreover,not only are states and educa-tional institutions being increas-ingly mandated to use standard-ized tests for a variety of assess-ment and accountability pur-poses, but there is also a growingtrend toward implementingpolicies that attach “high stakes”to student performance on thesetests. High stakes testing refersto the practice of attachingconsequences of the highestorder to an individual’s perfor-mance on a particular test. Mostnotable are the high stakesattached to standardized tests atthe high school level; in somestates, students must now pass

these tests in order to receive ahigh school diploma. Manyplans attach stakes to the perfor-mance of schools and districts,however, thus far, states havefocused more on holding indi-vidual students accountable thanon finding ways to hold schoolsand districts responsible forensuring that students have beenadequately prepared to meet thechallenges of high stakes tests. Despite the controversy, theuse of high stakes testing as areform and accountability tool isbecoming increasingly common.As of the end of 2002, 18 stateshave implemented high stakestesting as a requirement for highschool graduation and six otherstates are in the process ofphasing in such a requirement.1

Other states are considering theimplementation of high stakestesting. It is also clear that therecently enacted No Child LeftBehind Elementary and Second-ary Education Act (ESEA)federal education policy thatheavily emphasizes requiredtesting in public schools will addmomentum to the movementtowards the increased role oftesting and may lead to a greaterlikelihood that stakes will beattached to those tests as well. The controversies surround-ing high stakes testing have beenpolarized for and against thispolicy. While many people

support the important role thattesting can and should play asone part of more comprehensiveefforts to improve and reformeducation, others are concernedabout the unintended andpotentially negative conse-quences that may exacerbateexisting educational inequities.More specifically, testing hasbeen promoted as an essentialcomponent of the educationalreform and accountabilitymovement as an impetus formotivating schools and studentsto perform at higher standardswhile also providing a means forassessing progress. From thispro-testing perspective, the useof tests will help build betterbridges to future success forstudents. Tests are seen as animportant feature of a largerreform effort that will ultimatelyresult in students who are betterprepared to succeed in collegeand career. It has been suggestedthat accountability systems thatuse testing as a means for ensur-ing that students are learning areessential to making sure suchbridges to future opportunity areavailable to all students, particu-larly those who have beentraditionally disadvantaged byexisting inequities in the educa-tional system. Such comprehen-sive reform and testing efforts atthe state and national levels areintended to provide impetus for

BRIDGES AND BARRIERS:

EQUITY ISSUES IN HIGH STAKES TESTING

Joseph B. Berger

O

chronically under-achievingdistricts, schools, and students tobe identified as such so thattargeted interventions can beimplemented to close theachievement gaps that have thusfar persisted in the absence ofsuch reforms. Proponents believethat the increased emphasis ontesting has benefits for students,schools, and society as a whole.Additionally, proponents assertthat the implementation of highstakes tests will help schoolsbetter assess their own perfor-mance, help districts and statesimprove their ability to evaluatethe educational performance ofschools, and provide a means forholding schools accountable forthe learning that occurs in theirclassrooms.2

Others have expressedconcern that an over-emphasison testing leads to the creationof more barriers that will denyfuture educational and voca-tional opportunities for manystudents, particularly those whoare already disadvantaged by aninequitable educational system.In particular, high-stakes gradua-tion tests may have a negativeeffect on access to postsecond-ary education for students who

have traditionally been under-served in K–12 educationalsettings and under-represented inhigher education. Blacks, His-panics, English-language learners(ELL), and low-SES students aremore likely to attend schoolsthat fail to provide high-qualitycurriculum and instruction thanWhite and high-SES students. Itis therefore not surprising thatlow-SES and minority studentsfail high-stakes graduation testsat much higher rates than dotheir high-SES and non-minoritypeers.3 Additionally, studiessuggest that students who aredoing well academically are themost likely to drop out of schoolas a result of failing a high-stakesgraduation test.4 Thus, failure onthe high-stakes graduation testmay dissuade many otherwisehigh-performing minoritystudents from attaining apostsecondary education. Thisalready unacceptable inequity isa growing problem, as thesegroups also represent a majorportion of the fastest growingsegment of the population—asegment of population that isprojected to constitute a majorityof our citizenry and workforcesometime later this century.Many constituents and stake-holders are also concerned thatheavy reliance on such tests willlead to increased “teaching tothe test,” in which curriculumand instruction focus solely onsubject matter covered on thestandardized tests; potentialmisalignments with other testsused throughout the educationalpipeline that may result instudents getting mixed messagesabout their academic progress;inappropriate use of tests todetermine college admissions andplacement; and barriers thatlimit students’ aspirations andaccess.

There is also a need

to align education reform effortsacross the K–16 educationalspectrum so that changes inprimary and secondary educa-tion lead to greater levels ofopportunity and achievement forstudents in postsecondaryeducation. Standards-basedreform cuts both ways for accessto college. On the positive side, itpromises to redress the inequitiesof the previous educationalsystem, ensuring that if allstudents in a state are proficientin the same minimum standardthen they will have equal oppor-tunity to master the same high-level curriculum. This canpotentially increase the pool ofcollege-ready students. Con-versely, when implemented withhigh stakes, reform will nega-tively impact many studentsfailing to meet minimum thresh-olds on standardized exams.Because of their lack of successon such standardized exams,often such negative impacts willbefall minority and low-incomestudents, preventing them fromgraduating from high school andmoving on to postsecondaryeducation. Whatever the long-term consequences of standards-based reform, in the absence of amajor policy change, the short-term consequence of high stakestests will almost certainly be asmaller pool of students eligiblefor postsecondary education.Current trends suggest that thosestudents from minority and lowincome backgrounds who arealready most under-representedin postsecondary education aremost likely to have their accessto college reduced even further.5

Massachusetts is oneof the states that has generated agreat deal of controversy regard-ing its use of high stakes testing.In 1993, Massachusetts insti-tuted the Massachusetts Educa-tion Reform Act (MERA), the



Equity Issues in High Stakes Testing

most comprehensive statewidestandards-and-accountabilityreform in New England. Inimplementing MERA, the statecreated the common core oflearning, state-level curriculumframeworks in major subjects.The curriculum alignment waswidely supported, and schoolsand districts have been workingto align their curricula with thecontent of the common core. In 1998, the MassachusettsComprehensive AssessmentSystem (MCAS) was initiatedand is currently used to teststudents in grades four, eight,and ten in four core subject areas(English language arts, math-ematics, science, and socialstudies). A requirement thatstudents pass the tenth gradeMCAS English and mathematicsexams in order to receive a highschool diploma has been phasedin, with the Class of 2003 beingthe first class held to this stan-dard in order to graduate. Thisrequirement, like other highstakes testing, has generatedmuch controversy and is just aspolarized in Massachusetts as isthe broader issue throughout therest of the nation. The Massa-chusetts Department of Educa-tion (DOE) considers the gradua-tion requirement important inthe effort to raise the academicachievement of its students;however, many parents andeducators think the requirementis an unnecessary obstacle topostsecondary admission andemployment opportunities. Withthe phase-in of the high-stakesimpact of the test, proponents ofMCAS argue that students willdo better on MCAS in the futurebecause they will have beenexposed to curriculum based onthe state frameworks and willtake the test more seriously.Many stakeholders are particu-

larly concerned about the impactof MCAS as a high stakes test onstudents who have been tradi-tionally disadvantaged in publicschools. A recent report6 by the Centerfor Education Policy at theUniversity of MassachusettsAmherst on the progress ofeducation reform in Massachu-setts summarizes some of theresults of MCAS. In spring 2001,68 percent of the Class of 2003achieved competency determina-tion status by passing bothportions of the tenth-gradeMCAS. Eighty-two percentpassed the English language artsassessment and 75 percentpassed the mathematics assess-ment. After the fall 2001 retest,approximately 75 percent hadachieved competency determina-tion status, and after the spring2002 test, approximately 81percent had passed both sections.Clearly, these numbers doindicate that the majority ofstudents can pass over thesebridges and on to high schoolgraduation and all of the doorsthat can be opened with a highschool diploma. However, MCAS perfor-mance varies across differentsub-populations. On one level,this can be seen as MCASfulfilling its mandate to buildfuture bridges by focusing onachievement gaps and allowingnecessary resources to be tar-geted toward the neediest stu-dents and schools. Unfortu-nately, the high-stakes nature ofthe assessment turns thesedisparities into barriers thatcause certain types of students tobe more likely to suffer negativeconsequences. For example,students from higher-incomedistricts were more likely to passMCAS than their peers in lower-income districts. Approximately

95 percent of the students in thehighest-income districts hadpassed the test in 2001, whileonly about 59 percent of theClass of 2003 had passed in thelowest-income districts. Overall,81 percent of tenth graders hadpassed both required MCAS testsafter the third attempt (spring2002). Passing rates for “regulareducation” students were signifi-cantly higher at 87 percent thanfor students with disabilities whohad a 55 percent passing rateand Limited English Proficient(LEP) students were much lower,at 35 percent. Achievement gaps also existacross racial and ethnic groups.White students in the Class of2003 had an 87 percent passingrate after the spring 2002 tests.Asian and Native Americanstudents had a passing rate of 83percent after three attempts.African-American and Hispanicstudents had passing rates of 56percent and 50 percent, respec-tively. These findings indicategaps that are far too widebetween these groups, suggestingthat students from these variousdisadvantaged groups are dispro-portionately likely to encounterbarriers as they attempt to passMCAS as a high stakes test. One factor that seems to bean important predictor of successon MCAS is, not surprisingly,the level of courses students havetaken prior to taking the test.Opponents and supporters ofMCAS agree that the test is not a“basic skills” test; it includesmaterial normally covered inalgebra and geometry courses. Astudy released in April 2000 bythe independent groupMassInsight Education foundstrong correlations betweenenrollment in higher-level math-ematics courses and success onMCAS. More than half of the

students who failed the MCASmathematics test in 1999 wereenrolled in Algebra I or an evenlower-level course in tenthgrade—the norm for college-bound students is to take Alge-bra I in eighth or ninth grade.7

According to the DOE, Latinotenth-graders are three timeslikelier to be enrolled in low-level mathematics courses thanWhites. Black tenth-graders aretwo and half times likelier to betaking low-level mathematics.Even more troubling, 12 percentof Latino tenth-graders are notenrolled in a mathematics courseat all.8 While overall test scoresimproved dramatically in 2001there are still large-scale achieve-ment gaps that remain, a pointacknowledged by the DOE.9

The scores for under-repre-sented minority students onMCAS, as on other standardizedtests around the country, remainlower than the scores of theirnon-minority peers. The resultsof the analysis of MCAS in onestudy, however, suggests thateven after controlling for SATscores, minority students failabout 10 percent more than non-minority students.10 This meansthat there is an additional 10percent of bias (above andbeyond any existing bias in SAT)in a test that students need topass in order to graduate fromhigh school. It is also troublingthat minority students are twiceas likely to fail MCAS as arenon-minority students. This typeof systemic result cries out for adetermination of whether this istest discrimination or otherdiscrimination and an answerregarding how we can rid suchexam results of the disparity. Ata minimum, a lack of answersleaves the state vulnerable tolegal challenges to its testingsystem. Holding students ac-

countable for a test that appearsbiased is not right; states shouldonly consider attaching highstakes to these tests once theinherent bias of the tests hasbeen thoroughly examined andeliminated. Currently, MCAS is the onlyhigh stakes test in New Englandthat requires students to passboth portions of the exam inorder to be eligible to receivetheir high school diplomas. Inthe future, there will be evenmore components than justEnglish language arts andmathematics that students mustpass in order to graduate fromhigh school. This particularapproach is only one way ofdetermining the consequences ofhigh stakes for students. Becausestudents learn in different waysand succeed using variouscombinations of skills,policymakers should stronglyconsider structuring tests toreflect this reality. Some studentscompensate for weaknesses inone academic area with strengthin another. Colleges and univer-sities have long recognized thisfact, as evidenced by theirwillingness to allow for higherscores in one portion of the SATto compensate for lower scoresin another. The NCAA, an armof the college’s athletic depart-ments, established a tradeoffbetween standardized test scoresand cumulative high schoolaverages. The same should bedone with high stakes standard-ized tests such as MCAS. Thiscould be accomplished in severalways. For example, a slidingscale could be established wherea score above a certain level inmathematics could balance ascore below a certain level inEnglish. Alternatively, the cutscore could be determined as acumulative score across the

various portions of the exam,rather than by the unique resultsof each portion. While there aretrade-offs associated with anyapproach to scoring tests anddetermining passing standards,the importance and consequencesof these tests mandate that allpotentially useful approaches begiven serious consideration. Onlyin this manner can one of ourmost fundamental values—thatstudents are given opportunitiesto grow and mature at their ownpace—be protected. Embedded within this recom-mendation is a call for the use ofmultiple assessments. The use ofmultiple indicators has beencriticized on a number of counts,including that they are fuzzy andthat there is no agreement onwhich sets of measures should beused.11 Yet, there is no consensusabout the effectiveness of anyone test. Indeed, the findingsfrom several studies reveal thatany one test only measures apartial facet of student learning;the use of additional informationprovides a more complete andaccurate picture of what studentshave learned and may yetachieve. Single tests are efficient,but are less effective than mul-tiple measures in assessingstudent learning and predictingfuture success. Higher standards and higherstakes mean greater challengesfor students. The appropriatebalance between level of chal-lenge and subsequent level ofsupport is recognized as a funda-mentally necessary condition tofacilitate growth, development,and learning. Given the impor-tance of the challenge studentsface when taking standardizedexams, particularly those withhigh stakes, it is of paramountimportance that appropriatelevels of support be provided to




help students prepare for thesetests. This means that states needto ensure that curriculum frame-works are fully implemented forevery child, that at-risk studentsare receiving additional re-sources, and that effectiveprofessional development isbeing provided for educatorsresponsible for student learning.All of these things should befully implemented prior toattaching any stakes to theperformance of individualstudents on such tests. Thisrecommendation also focuses onthe need to hold adults account-able first. Most states that haveimplemented high stakes testing,not just Massachusetts, havefound it more difficult to findways of holding teachers,schools, and districts account-able for preparing students tosucceed than they have found itto hold students accountable.Yet, testing may result in unfairand unequal treatment of stu-dents if learning environmentsare not all equal in supportingthe students who must pass thesetests or have further educationalopportunities denied to them.High stakes should be appliedonly at the point that schoolsand districts are providing theeducation to which every child isentitled. There is little doubt thattesting is here to stay. However,policymakers have a wide rangeof choices about how to teststudents and how to use the testsas a fair and effective means forimproving educational outcomes.Policymakers need to rememberthat testing is the means to anend, not the end in and of itself.This type of recognition canserve as the basis for makingsure that tests are constructedand used in ways that bestbenefit students. In order to do

this, testing should not only beevaluative; it should also berigorously and continuallyevaluated. The empirical resultsof such evaluations should beused to guide on-going adjust-ments to the tests and theassessment systems of whichthey are often a part. Fine tuningwill be necessary until testsbecome better bridges acrosswhich all can realize theireducational aspirations and notbigger barriers that impede ordeny opportunity. If we truly are going to“Leave No Child Behind” thenwe need to make sure that allchildren are properly preparedto cross the bridges we make forthem. If the students are unpre-pared, or if the bridges are notwell constructed or too difficultto get to or across, then we willhave constructed barriers insteadof bridges. The results of highstakes testing in Massachusettsand elsewhere suggest that theseexit exams are serving as bothbridges and barriers. Manystudents are crossing that bridge,but many others are encounter-ing it as a barrier. Unfortunately,many of those who encounter abarrier instead of bridge areminority and low-incomestudents who have traditionallyencountered more than their fairshare of barriers. Policymakersand educators need to make surethat educational policies, such ashigh stakes tests, are indeedbridges to future success and notbarriers to opportunity. Now isthe time to make sure that allstudents have equitable chancesto cross the bridge and createsuccessful futures on the otherside that will benefit themselves,their families, their communities,and their employers, as well asour larger society.

ENDNOTES

1Measuring Up: A Report on Stan-dards and Assessments for Massachu-setts. Washington, DC: Achieve, Inc.,2001. 2 op. cit. 3National Research Council. “Link-ing Test and Learning” News Report,49(3). 1999. p.3. 4 op.cit. 5Coelen, S. P. & Berger, J. B. Educa-tional Opportunity and the Impact ofGraduation Testing in New England.Amherst, MA: Center for EducationPolicy, School of Education, Universityof Massachusetts Amherst, 2001. 6Churchill, A., McDermott, K.,Bowles, S. Lee-Davis, C., Effrat, A.,Berger, J., Carey, J., Brooks, C. &Klock, R. 2002 Annual Report on theProgress of Education Reform inMassachusetts. Boston: MassachusettsEducation Reform Review Commis-sion, 2002. 7High Stakes and High Standards:Public Attitudes toward EducationReform. Boston: Mass Insight, 2000. 8Gaston Institute. “MCAS FailureRates for 1998 and 1999 Compared inNew Study.” Maurice Gaston InstituteReport. Boston: University of Massa-chusetts Boston, 2000. 9Spring 2001 MCAS Tests: StateResults by Race/Ethnicity and StudentStatus. Malden, MA: MassachusettsDepartment of Education, 2001. 10Coelen, S. P. & Berger, J. B.Educational Opportunity and theImpact of Graduation Testing in NewEngland. Amherst, MA: Center forEducation Policy, School of Education,University of Massachusetts Amherst,2001. 11Reville, S. P. “Multiple Measures?”Education Week. 21(11). 2001. p. 52.


Joseph B. Berger is an assistantprofessor in the Department ofEducational Policy, Research andAdministration. He currentlyserves as the interim chair of thedepartment and is an AssociateDirector of the Center forEducation Policy.

ardly a day goes bywithout an article, editorial, orletter to the editor in the news-paper about state testing. In mystate, the testing program iscalled the Massachusetts Com-prehensive Assessment System,better known as MCAS. MCASis about educational reform—reforms in curricula, teacherand principal training, classsize, after-school programs,school funding, testing, etc., butmany think MCAS is onlyabout testing. Just about everystate has its own version of theMCAS. With the new federallegislation, the No Child LeftBehind Act, states have littlechoice but to embark uponmajor educational reforms,especially in the subject areas ofreading and mathematics. Today’s newspaper headlinein Amherst was about 10,000high school seniors across thestate of Massachusetts whohave their last chance this weekto pass the English languagearts and mathematics tests priorto graduation next June. Theday before, the newspaper

reported on a mathematics itemon last year’s test that appearedto have two correct answers.And the day before that, a newsreport focused on a group ofstudents who managed to meetthe state’s graduation criteriafor English language arts andmathematics proficiency by analternative method to passingthe state tests—high attendance,coming close to passing thetests, and doing well in theirday-to-day classroom work.News coverage is relentless.Almost everyone wants toimprove education, and justabout everyone has opinions(and voices those opinions)about how this can and can’t bedone. Many people have ques-tioned the “passing scores” onstate tests. In Massachusetts,one of the popular explanationsfor the high failure rate amongtenth grade students has beenthat the passing scores are settoo high. Given all the talkabout passing scores, it issurprising that very few peopleseem to know much about


them. Who sets the passingscores? How are they set? Whatlevel of test performance isactually required to achieve apassing score of 220 on theMCAS? In my work around thestate I have yet to meet any-one—a school board member, ateacher, or a parent—who hasmore than the most superficialknowledge about how thepassing scores are actually set,or about the meaning of apassing score of 220. Theremainder of this paper will beabout setting passing scores.The specifics pertain to Massa-chusetts, but most states settheir passing scores in much thesame way as Massachusetts. I would make the point firstthat most states today do notsimply report scores as pass-fail.With the more demandingcurricula in place, many stateswant to recognize several levelsof academic accomplishment.Students are classified, basedupon their test performance,into four levels of proficiency:Advanced, Proficient, Basic, andBelow Basic. In Massachusetts,

SETTING PASSING SCORES ON TESTS…

Not Too High…Not Too Low…But Just About Right

Ronald K. Hambleton

H


Setting Passing Scores

four proficiency levels are used:Advanced, Proficient, NeedsImprovement, and Warning.Therefore, not one but threecut-scores (often called “perfor-mance standards”) are set onthe test score scale to classifystudents into the four perfor-mance categories. Performancestandards are scores on the testscore scale that are needed toseparate students into perfor-mance levels. On the 1998MCAS grade 10 mathematicstest, the performance standardswere set at approximately 40percent correct (for NeedsImprovement), 57 percentcorrect (for Proficient), and 76percent correct (for Advanced).The performance standards areabout 5 percent higher on thegrade 10 English language artstest. Students must achieve atleast the Needs Improvementlevel to “pass.” These are thelevels of performance that haveremained in place since 1998.In subsequent years, test scoresare adjusted to reflect the factthat later tests may be a biteasier or harder (this is called“equating” and is discussed inan article on page 15), and thenthe 1998 performance stan-dards are applied. Let’s take a look now athow states go about trying toset defensible and valid perfor-mance standards. First, perfor-mance standards on a state testare almost always recom-mended by a panel of 15 to 25persons to the State Board ofEducation, which has theresponsibility for setting thefinal performance standards. InMassachusetts, the recommen-dations from the panels wereaccepted without revision bythe Board of Education. Mem-

bers of these panels includeteachers and school administra-tors and, sometimes, especiallyfor tests at the higher gradelevels, the public. The composi-tion of the panels varies fromstate to state, but teachersnormally make up the largestgroup. When performancestandards were set in Massa-chusetts in 1998, 209 personsparticipated on 12 panels (anaverage of about 17 per panel).Fifty-one percent of the panelmembers were teachers. Theprevailing view is that teachershave the best information about(1) the challenges of the cur-ricula and (2) realistic expecta-tions for students. Other in-formed individuals and stake-holders are usually included onthese panels: in Massachusetts,22 percent of panel memberswere school administrators, 17percent were from the businesscommunity, 7 percent werefrom higher education, and 3percent were school committeemembers or members of local/state government. Every effortis made to form panels that

bring balance, experience, anddiversity to the process. Factorssuch as gender, ethnic back-ground, age, years of teachingexperience, subject area exper-tise, and geographic location inthe state are routinely consid-ered in forming panels. Theacceptability of a panel to stake-holders is just about the mostimportant consideration in theprocess of setting defensible andvalid performance standards.The 209 panel members inMassachusetts were divided into12 smaller panels and assignedto set performance standards onone of the tests at grades 4, 8,and 10. Second, panelists must haveclear descriptions of the perfor-mance levels. In Massachusettsand other states, these perfor-mance levels are very carefullyspelled out (and approved bythe Board of Education), andprovide a basis for setting theperformance standards on thetests and for interpreting stu-dent performance after testshave been administered. Forexample, at the grade 10 level in



mathematics the Needs Im-provement category is describedas follows: A student whoachieves this performance level1. Demonstrates partial under-standing of our numerationsystem.2. Performs some calculationsand estimations.3. Identifies examples of basicmath concepts.4. Reads and constructs graphs,tables, and charts.5. Applies learned procedures tosolve routine problems.6. Applies some reasoningmethods to solve simple prob-lems.7. Identifies and uses basicmathematical terms.Descriptions like the one aboveare prepared for each perfor-mance level, and used by thestandard-setting panels todetermine the test scores neededto be classified at the NeedsImprovement, Proficient, andAdvanced levels. Similar de-scriptions are available for otherperformance levels in grade 10mathematics and for the perfor-mance levels on the other tests. With the panels formed, anddescriptions of performancelevels available, step three canbegin: the judgmental process ofsetting performance standards.Following what is normally ahalf-day of training, panelsbegin their work. In Massachu-setts, panels used a method forsetting performance standardsthat involves sorting student testbooklets into four performancelevels. Panelists review theactual test booklets of a widesampling of students who tookthe test and sort the bookletsinto the four performancelevels. Panel members workindependently at first, and later

they discuss their classificationswith other panel members andrevise their classifications whenrevisions seem appropriate.There are no right or wrongclassifications. What is valuedare the judgments and thoughtprocesses of the panel members.Normally this task is completedwithout knowing the scoresattached to the student testbooklets. In this way, panelmembers can focus their atten-tion on the quality of work theyobserve and match it to theperformance levels. Poor workcan be assigned to the lowestcategory, Warning, and out-standing work can be assignedto the highest category, Ad-vanced. Other levels of studentwork must also be judged andclassified. A standard-setting study willoften run for two or three days.By the end of the first day,panelists in Massachusetts hadthe opportunity to review andclassify student test booklets,discuss their classifications withother panel members, andreclassify any student testbooklets when they felt changeswere in order. The standard-setting task took two days and,on the second day, panelistswere given more student testbooklets to review. This time,the student test booklets wererelatively close to the tentativeperformance standards set theprevious day. This second set ofstudent test booklets allows fora more precise determination ofthe placement of the perfor-mance standard between perfor-mance levels. Each perfor-mance standard is obtained inthe following way: a perfor-mance standard is taken to bethe average score of student

booklets that are judged to be“on the borderline.” These arethe student booklets that panel-ists are equally divided about—they could be at one perfor-mance level or one lower. Insetting the performance stan-dard for Needs Improvement,borderline students would bethose students where panelistswere divided about their place-ment—Warning or NeedsImprovement. The scores ofthese students mark the place-ment of the performance stan-dard. So, if the test bookletscores for these students hap-pened to be 18, 18, 19, 19, 20,and 20, the performance stan-dard would be set at 19, theaverage of the six borderlinestudent scores. The process isrepeated for each performancestandard. In a final step, panelists areasked to provide an evaluationof their training, the processthey went through to set perfor-mance standards, and theirconfidence in the resultingperformance standards. Otherevaluative criteria include theintra-consistency of panelists’ratings and the level of agree-ment among panelists in theirperformance standards. Figure 1 (on page 14) showsthe student test scores (0 to 100percent) on the vertical axiswith the three performancestandards marked. Once astudent’s score is known, theperformance category to whichthe student is assigned is obvi-ous. The process describedabove is repeated for each test.Performance standards are setby a panel and approved by theBoard of Education (e.g., 40percent, 60 percent, 75 percent),and then they are used for



classifying student performance.Ultimately, each student’s scoreis transformed to the “MCASscale” via the mapping shownin the figure. Thus the studentwho does just well enough tomeet the performance standardfor Needs Improvement receivesa score of 220. A student whodoes just well enough to meetthe performance standard forProficient receives a score of240. Scores between these twoperformance standards wouldreceive scores between 220 and240, and so on. In the figure, ascore of 50 would receive anMCAS score of 230. MCASscores in excess of 260 areAdvanced students. In Massachusetts, regardlessof grade level or subject, stu-dents are assigned to a perfor-mance level using the perfor-mance standards set by thepanel (and approved by theBoard of Education), and thentheir scores (after necessaryadjustments are made for thenon-equivalence of the testsfrom one year to the next) aremapped onto the MCAS scale.Thus the level of test perfor-mance represented by 220

depends on the panel, which inturn is working from the perfor-mance level descriptions and thetest itself. In grade 10 math-ematics in 2002, a studentneeded a score of 35 percent onthe test to obtain a passingscore of 220—students in theNeeds Improvement, Proficient,or Advanced levels are consid-ered to have passed the test. Ingrade 10 English language artsin 2002, a student needed ascore of about 55 percent. Thebest way to think about a scoreof 220 is to recognize that astudent with this score hasaccomplished just enough tomeet the minimum expectationsof the Needs Improvementperformance level. There are no “true perfor-mance standards” waiting to bediscovered from careful educa-tional research. Setting perfor-mance standards on educationaltests like the MCAS involvescareful professional judgmentbeing made by persons who areviewed as suitable for providingthe judgments. Validity of theperformance standards is notjudged by the percentage ofstudents who pass or fail (who

would know what these per-centages should be?) but insteadis judged by (1) the qualifica-tions and demographics of thepersons who are charged withthe responsibility of setting theperformance standards, (2) theclarity and suitability of theperformance level descriptors,(3) the reasonableness of theprocess the panel goes through,and associated statistics thatdescribe their judgments, and(4) the confidence the panelmembers have in the processitself and in the final perfor-mance standards that they set.For follow-up reading aboutsetting performance standards,Cizek’s book, Setting Perfor-mance Standards: Concepts,Methods, and Perspectives,would be an excellent startingpoint. For details on the stan-dard-setting process in Massa-chusetts, readers are referred tothe technical report published in1999 by the MassachusettsDepartment of Education.

REFERENCES Cizek, G. (Ed.). Setting PerformanceStandards: Concepts, Methods, andPerspectives. Mahwah, NJ: LawrenceErlbaum Associates, Publishers, 2001. 1998 Massachusetts ComprehensiveAssessment System Technical Report.Malden, MA: Massachusetts Depart-ment of Education, 1999.

Ronald K. Hambleton is aDistinguished University Profes-sor in the Department of Educa-tional Policy, Research andAdministration. He serves asDirector of the Research andEvaluation Methods program ofstudy and Co-Director of theCenter for Educational Assess-ment.

Figure 1

Mapping Test Scores to the MCAS Scale

MCAS Score

Test

Sco

re

100

50

A

P

N

0

200 220 240 260 280230

ne of the centralgoals of the MassachusettsEducation Reform Act (MERA)of 1993 is to ensure that allstudents achieve high academicstandards, and the Massachu-setts Comprehensive AssessmentSystem (MCAS) tests weredeveloped to monitor just howwell the schools are meeting thisobjective. Therefore, the abilityto compare students’ perfor-mance on MCAS over time is ofcentral importance in evaluatingthe impact of MERA. While theconcept of comparing scoresover time is simple, the processthat enables a valid comparisonis quite complex.

The cut scores used todetermine the proficiency levelof the student (Advanced,Proficient, Needs Improvement,Failing) were set in 1998. Giventhat different tests are adminis-tered each year, how can we besure that the changes in perfor-mance are due to changes instudent achievement and notchanges in test difficulty? Thatis, how can we be sure thatstudents are accurately catego-rized in 2002, or 2003, or anysubsequent year? Are students

actually improving in abilitiesor is the test getting easier? Theissue central to these questionsis how to disentangle the impacton performance of the differenttests and differing studentabilities.

The MCAS was devel-oped as an accountability toolconnected to the educationreform with the goal of moni-toring student-, school-, district-and state-level progress. TheMCAS tests are administered ingrades 3–8 and in grade 10, andmeasure the content areas ofmathematics, science andtechnology, English languagearts, social studies, history, andreading. Due to the comprehen-sive nature of MCAS, not alltests are administered at allgrade levels. The tests aredivided among the years toensure that students and teach-ers are not burdened withexcessive testing. At the tenthgrade level, students must passboth the English language artstest and the mathematics test toqualify for a high school di-ploma. (Students must alsofulfill the graduation require-ments of their particular school

in order to receive a diplomaand thus the graduation deci-sion rests on many factors, notthis one test.) In this way, theMCAS can be used as a moni-toring tool. In the earlier grades,tests are administered to moni-tor educational progress ofstudents so that remediation cantake place prior to the tenthgrade tests. The hope is that byimplementing this assessmentsystem, education can be im-proved for all students, and thata high school diploma will be atrue indicator of a level ofcompetency determined to bemeaningful by educators in theCommonwealth.

Of central importance isthe comparability of scoresfrom year to year. Again, sincethe scores used to determinewhich category a studentbelongs in were established in1998, it is essential that testscores from subsequent yearsare comparable to the scoresthat the same students wouldhave obtained if they took thetest in 1998. Test developersundergo extensive training inwriting test items and choosingthose items most appropriate

ARE MCAS SCORES

COMPARABLE OVER TIME?

Lisa A. Keller


O

for the test. Additionally, allitems are field-tested to ensurethat they are functioning prop-erly, and that the questions areappropriate and fair for allstudents. After the field-testing,items are selected based onstatistical and content qualities.These features of the test ques-tions allow test developers tobuild tests that are very similarfrom year to year, both incontent and difficulty. However,despite this rigorous process, itis impossible to constructidentical tests every year. As aresult, the difficulty of the testmay vary slightly from year toyear. Because of this, comparingtest scores over time becomes achallenge. To address this issue,statistical adjustments are madeto account for the differencesthat exist. The process ofadjusting scores to account forthe differences in difficulty oftest forms is known as equating.

In the case of MCAS,scores are reported on a 200–280 scale. That is, the minimumscore on a subject test is 200and the maximum score is 280.Regardless of administration(fall 2001 or fall 1999) a score

of 240 should have the samemeaning, and the goal of equat-ing is to make sure that it does.There are several methods thatcan be used to equate testscores. Each of the methods istheoretically sound, and there isno “best” method in the senseof a most accurate method.Research has shown1 that thedifferent methods produce verysimilar results. Therefore, thechoice of method depends moreupon the nature of the testingsituation and the types of scorecomparisons desired. Forexample, it may be of interest tocompare scores across severalyears, as in the case of MCAS.However, in other situations, itmay be desirable to comparescores only within a given year.Depending on the specificsituation, one method may bepreferred over another for easeand practicality of implementa-tion.

Test forms may vary indifficulty from year to year, butstudents may also differ inability from year to year. In fact,the hope is that students arebecoming more proficient at theskills contained in the content

standards. Thus, if the same testwere administered to twodifferent groups of students, theperformance may be differentbecause one group is more ablethan the other. Hence, when thequestions on the test change,differences in scores can resultfrom two sources: the difficultyof the test or the ability of thestudents. When scores areequated, it is necessary to makeadjustments that account for thedifferences in difficulty of thetest and not the differences inability of the student groups.This is a difficult task, whichrequires sophisticated statisticaltechniques. Since the process isso technical, the details of eachof the equating methods willnot be presented here. However,the main concepts behindequating will be discussed.

The goal of equating isto adjust test scores to accountfor differences in difficultiesamong test forms. To do so, areference test must be identified.Scores on any other test formwill be adjusted such that theyare comparable to those of thereference test. So, for example,in MCAS, when test scores from


Are MCAS Scores Comparable Over Time?

a 1999 administration arecompared to scores from the1998 administration, the 1998administration is the referenceform. Therefore, to comparethe scores from the 1999 tothose obtained in 1998, thescores from 1999 are adjustedand made comparable to thescores from 1998. In this case,we can consider the 1998 testform as Form A and the 1999test form as Form B (note thatthis concept can be generalizedsuch that any administrationother than 1998 can be consid-ered Form B). The group thattakes Form A will be referred toas Group A, and similarly, thegroup that receives Form B willbe referred to as Group B.

To simplify the conceptsome, consider the followingscenario. Assume Form A isidentical to Form B. Group Aperforms worse, on average,than Group B. The conclusionwould be that the differences inscores are due to differences inabilities of students: those whotook Form B knew the materialbetter than those that tookForm A. This conclusion isreasonable, and it is this funda-

mental concept on which theidea of equating is built. In theinstance that the two test forms(A and B) are not identical, thesituation becomes more diffi-cult, as now both the studentsand the questions change. Thissituation is more typical, and itis, in fact, the case with MCAS.Because it is, in general, impos-sible to decompose the differ-ences in scores into the twosources (students and ques-tions), it is necessary to holdone factor constant: eitherquestions or students. Clearly itis not possible to have studentstake both Forms A and B. It isalso unreasonable to keep thesame questions on Form A andForm B, as the test items usedto determine a student’s scoremust be made available to thepublic. Therefore, a compro-mise is made.

In the case of MCAS, aswell as many other large-scaleassessments, the compromise isto keep a subset of itemscommon from year to year.This subset of items is oftenreferred to as an anchor test. Bykeeping some items common,the performance of the students

on the anchor test can becompared. This scenario thenreduces to the simple scenarioabove, where Form A and FormB are identical. In this case, theanchor test is identical for bothgroups of students; thus therelative ability of the twogroups can be assessed usingthis anchor test. For example, incomparing the scores of GroupA to Group B on the anchortest, if Group B scores higherthan Group A, then the conclu-sion is that Group B is moreable than Group A (on this setof items). The differences inability are now known and canbe taken into account when thescores are compared. Theremaining differences in thescores are then attributed to thedifferences in difficulty of thetwo test forms, and adjustmentscan be made for these differ-ences.

In order for this processto be reasonable, the subset ofitems that is chosen for theanchor test must be similar incontent and format to the itemson the entire test. That is, theanchor test should be a minia-ture version of the whole test.



iven that different tests are administered each year,

how can we be sure that the changes in performance are

due to changes in student achievement and not due to

changes in test difficulty? The issue central to this ques-

tion is how to disentangle the impact on performance of

the different tests and differing student abilities.

G

So, if the whole test consists ofmultiple-choice items as well asshort answer items, then theanchor test should also containboth types of items. Similarly, ifthere are five content areasrepresented in the entire test,then these same five contentareas must also be representedin the anchor test. In controllingthe content and format of itemsin this way, the differences inability on the anchor test shouldbe representative of the differ-ences in ability in the entire test.

The final issue to resolveis the mandatory disclosure ofitems used to score students.Federal law requires the disclo-sure of all information in astudent’s record. Since theMCAS test scores are reportedat the student level, parentshave the right to see the itemsused in the computation of astudent’s score, which causes aproblem in developing ananchor test. If the items of theanchor test were released to thepublic, then students in 1999would have access to the itemsthat were used in 1998. Thestudents could then memorizethe answers to these questions,resulting in a much higherperformance on the anchor testthan would be expected. There-fore, the differences in theabilities of the two groupswould appear to be much larger,resulting in an incorrect as-sumption about the relativedifficulty of the two tests. Then,when the statistical adjustmentis made, the scores of thestudents in 1999 would beinflated, resulting in themisclassification of students. Toremedy this problem, the itemsin the anchor test are notdisclosed. Since they are not

disclosed, they cannot be usedtoward the student’s actualscore. This practice results in atest composed of two parts:items used for scoring purposesand items used for equatingpurposes.

Once the statisticaladjustments have been made,the resulting scores from GroupB on Form B are comparable tothe scores Group B would haveobtained on Form A if they hadtaken it. Therefore, a compari-son of scores across forms (i.e.,years) is possible. In the case ofMCAS, scores must be equatedback to the 1998 form so thatgrowth in student achievementsince 1998 can be assessed.

As part of the Depart-ment of Education’s ongoingconcern regarding the valid useof test scores, the Center forEducational Assessment (CEA)at the University of Massachu-setts Amherst has been con-tracted to provide qualitycontrol for the equating ofMCAS test scores. Each year,the CEA does the equating oftest scores. The results of theseanalyses are then compared tothose of the company con-tracted to develop and analyzethe test to ensure that theequating was done properly. Itis only after equivalence be-tween the two sets of analyses

can be established that theequating is considered final. Inthe event of a discrepancy,measures are taken to deter-mine the explanation. Once anexplanation is found, a judg-ment is made as to whichmethod is most accurate, andthe results of the more appro-priate method are used.

ENDNOTE 1 Hanson, B. A. & Béguin, A. A.“Obtaining a Common Scale for ItemResponse Theory Item ParametersUsing Separate Versus ConcurrentEstimation in the Common-ItemEquating Design,” Applied Psychologi-cal Measurement, 26 (1999). pp.3-24. Kim S. H., & Cohen A. S. “AComparison of Linking and Concur-rent Calibration under Item ResponseTheory,” Applied PsychologicalMeasurement, 22 (1998). pp.131-143. Li, Y. H., Griffith, W. D., & Tam, H.P. Equating Multiple Tests via an IRTLinking Design: Utilizing a Single Setof Anchor Items with Fixed CommonItem Parameters during the Calibra-tion Process. Paper presented at theannual meeting of the PsychometricSociety, Knoxville, TN, June, 1997.



n the case of MCAS, scores must be

equated back to the 1998 form so

that growth in student achievement

since 1998 can be accurately assessed.

I

Lisa A. Keller (Ed.D. 2002) is alecturer in the Department ofEducational Policy, Research andAdministration. Her mainresearch interests are in theareas of test equating, comput-erized adaptive testing, anditem response theory.

en years after the passageof the Massachusetts EducationReform Act (MERA) of 1993,classroom teachers, collegestudents preparing to becometeachers, and teacher educatorsin colleges and universities facea dramatically changed educa-tional landscape. State-wideK–12 curriculum frameworks inEnglish, mathematics, science/technology, history and socialscience, foreign language, arts,and health not only outlinewhat academic content to teachat different grade levels, butdemand that all students beeducated to high levels ofacademic achievement. Massa-chusetts Comprehensive Assess-ment System (MCAS) tests adda high stakes dimension to theseframeworks since students whodo not pass the tests cannotgraduate from high school.

Curriculum frameworksand high stakes tests mean thatexperienced educators and newteacher candidates must empha-size teaching of academiccontent at every grade level.Increasingly, they turn to theInternet for content resourcesand instructional strategies. Butthe Internet is not a one-stop

source of valuable information.General search engines putteachers in touch with vastamounts of material, but notnecessarily the resources neededto create effective lessons.Teacher-oriented sites providestep-by-step lesson plans, but itis unclear as to how they arejudged for quality. As the hitson a Google search pile up,teachers quickly encounteronline information overloadwithout the time to determinewhat is useful and what is not.

Enter VRROOM—theVirtual Reference Room—anonline resource for Massachu-setts teachers and new teachercandidates being developed bythe Center for Computer-BasedInstructional Technology(CCBIT) in the Department ofComputer Science in collabora-tion with teacher educationprograms in the School ofEducation at the University ofMassachusetts Amherst. (It isavailable online at http://ccbit.cs.umass.edu/vrroom.) Itsgoal is to improve the academiccontent knowledge of collegestudents who are preparing toteach and support the profes-sional development of veteran

teachers by providing rapidelectronic access to challengingand engaging Internet-basedacademic resources.

Our VRROOM collabo-ration extends beyond thecampus, bringing togetherorganizations devoted to usingthe Internet, video, and othertechnologies to improve teach-ing and learning in publicschools throughout Massachu-setts, including public televisionstation WGBY (Channel 57 inSpringfield), the MassachusettsSchool Librarians and MediaSpecialists Association(MSLMA), the MetroWestLibrary Group of GreaterBoston, and the Virtual Educa-tion Space (VES) initiative ofthe Massachusetts Departmentof Education.

Prior to last year, eachpartner organization had beenworking independently to linkacademic content and technol-ogy resources to the state’s K–12 curriculum frameworks.Now working in VRROOM,with a common homepage, thepartners intend to provideeducators around the state witheasy and rapid access to curricu-lum materials and instructional

VRROOM: THE VIRTUAL REFERENCE ROOM

Using the Internet to Support Teachingthe Massachusetts Curriculum Frameworks

Robert W. Maloy, David Hart, Paul Oh, and Victoria Getis


T

ideas, all linked to the curricu-lum frameworks (seeVRROOM Partners on p. 21).

STANDARDS CONNECTORThe University’s contribution toVRROOM is the StandardsConnector. It features activelinks to web-based resources forteaching the core content of thestate’s English, mathematics,science/technology, and historyand social science K–12 curricu-lum frameworks. (Eventually,we plan to add links for thestate’s foreign language, arts,and health frameworks). Nowin its second year, StandardsConnector is a web site inprogress, consisting of a grow-ing collection of primary docu-ments important to the study ofworld and U. S. history; interac-tive resources for improving thereading and writing skills ofchildren and adolescents; andinteractive mathematics andscience activities for differentgrade levels.

The Standards Connec-tor was initially intended as aresource for certification candi-dates in the School ofEducation’s elementary andsecondary teacher educationprograms. These college stu-dents were already familiar withthe Internet, and many used theweb to research curriculumideas. Our idea was to givestudents in teaching methodsclasses a way to connect excel-lent web-based resources to thecurriculum frameworks theywould be expected to teachwhen they did their studentteaching.

We began by enteringthe content of the state’s cur-riculum frameworks into adatabase. With the assistance ofgraduate student researchersand University faculty, we beganidentifying Internet-basedresources for meeting thesecontent expectations. Next, wegathered feedback about the sitefrom new teacher candidates

attending courses in the Schoolof Education and teachersattending regional meetings ofthe New England Council forthe Social Studies and theMassachusetts Computer UsingEducators group. From theircomments, we incorporated thefollowing key elements intoStandards Connector’s designand function, building a re-source not only for new teachercandidates, but for classroomteachers as well.

• Web Sites Linked tothe Curriculum Frameworks.Entering the system, users findthe key academic content takenverbatim from the state’s frame-works presented in green downthe left side of the screen withannotated web sites listed alongthe right side. Web links areactive so users can quickly movefrom Standards Connector tothe online resource that theywant to explore. Cooperatingteachers and new teachercandidates reported having thecurriculum frameworks andInternet resources for teachingacademic content together inone place was very helpful forlesson planning and classroominstruction.

• Web Sites EmphasizingAcademic Content. Newteacher candidates and veteranteachers told us they valuedhaving access to informationfrom which they can create theirown lessons rather than justseeing someone else’s alreadyassembled lesson plans. Sorather than identifying sitesconsisting mainly of lessonplans, we looked for webmaterials that offer primarysources (speeches, photographs,memoranda, art, etc.), visualdisplays (maps, charts, graphs,interactive models), and infor-


Teaching the Curriculm Frameworks

David Hart, Robert W. Maloy, and Paul Oh

mation summaries (biographiesof key figures, backgroundmaterial on important eventsand concepts).

• Strategies for Evaluat-ing Web Resources. To providenew teacher candidates, teach-ers, and K–12 students withcriteria and skills for evaluatingmaterial on the Internet, weincluded a section of essentialquestions for users to ask whenevaluating online materials.The goal is for every teacherand every student to be athoughtful, critical evaluator ofweb resources, aware of thedirect and indirect messagesthat web sites may contain. Atthe same time, to make our siteas instructionally reliable aspossible, all selections includedin Standards Connector havebeen checked by faculty mem-bers or graduate students andvetted by a second reviewer.

• Multiple Resources.To offer educators a variety ofresources as starting points forcurriculum planning, we devel-oped a system for identifyingthe kinds of web-based materialposted in the web site. Internetresources are designated aseither “teacher resources,”“student resources,” “lessonplans,” “unit plans,” “web-based activities,” or “historicdocuments.” Some sites havemore than one designation. Inaddition, we built a separateteacher resource section withmore than 50 general lessonplan sites containing thousandsof curriculum ideas, eachannotated to describe its em-phasis, commercial or non-profit status, and strengths andweaknesses. A list of searchengines is also part of the“teacher resources” section.

E-TEAMSOne of the interesting contra-dictions surrounding technologyin schools is that while mostteachers use computers (93percent of teachers in grades4–12 report use of computersprofessionally at school, athome, or both.1 ), only aboutone in four teachers report“significant” use of the web atschool. Among veteran teach-ers, the percentage is even lower(13 percent for those with 20 ormore years in the field).2 Inaddition, there is a continuingdigital divide between affluentand less affluent communities,with just over half of teachers inlow-income and minoritydistricts having access to theInternet at school.3 As a result,many students are not seeinghow technology can support thelearning of core academicsubjects.

Given these findings, acollection of electronic re-sources for teaching the curricu-lum frameworks is only part ofa solution to the problem ofintegrating technology morecompletely into teaching and

Standards Connectoras a Resource for

Classroom Teachers,New Teacher Candidates,

Parents, and Students

• presents excellent resources(print, Internet, video) directlylinked to curriculum frameworks.• contents screened by UMassSchool of Education graduate stu-dents and faculty.• offers lesson plans, unit plans,Web-based activities, teacher re-sources, and student resources.• serves as teaching tool whenuser implements lesson plans ordemonstrates concepts throughWeb-based activities.• extends learning as students ex-plore resources and Web-based ac-tivities in classroom and beyondclassroom.• familiarizes students and teach-ers with technology as they navi-gate site and links.• helps new teacher candidateslearn effective uses of technologyin classroom.• up-dated so teachers can learnabout new frameworks and findteaching tools linked to thoseframeworks.• provides homework assistanceand research resources for stu-dents.• familiarizes parents with curricu-lum content mandated by stateand taught in schools.• available free to school districtsand individuals.

VRROOM PartnersStandards Connector has been created by the Center for Computer-BasedInstructional Technology in the Department of Computer Science in col-laboration with teacher education programs in the School of Education. Itincludes web sites linked to specific academic content expectations in thestate’s curriculum frameworks. Available at http://ccbit.cs.umass.edu/standardsconnector/.

Selection Connection, developed by Massachusetts School Library andMedia Association, features notable children’s and adolescent literaturelinked to the required content areas in the curriculum frameworks.

Video Lending Library, of public television station WGBY (Springfield),offers an extensive collection of PBS television programs, broadly codedto the curriculum frameworks. All videos loaned at no charge to teachersin western Mass. Available at http://www.wgby.org/edu.

MetroWest Massachusetts Regional Library System, the newest mem-ber of the VRROOM Collaborative, provides resources relating to the frame-works for librarians in greater Boston.



sources. This way, teachers andcoaches together can take advan-tage of the instructional potentialof the Internet and other digitalmedia.

CONCLUSIONLast spring, at a campus-widemeeting of faculty and adminis-trators, Mark Weston of AppleCorporation likened recentdevelopments in educationaltechnology to the evolution of anultra-modern highway system.In the 1990s, affordable, power-ful computers burst on the scene,providing educators with trulyfast vehicles. As the decadeprogressed, the Internet, elec-tronic mail, and other systemsallowed for the networking ofthese computers, giving teachersand students superhighways onwhich to drive their machines.Today, a new era of change isupon us, one in which educatorsmust decide how to best realizethe potentials of computers andnetworks. In this revolution, thefocus will shift to the content oftechnology and the competenciesof users—that is, to how weprovide fuel for the cars andskills for the drivers.

The analogy of technol-ogy and highways extends tocurriculum frameworks andeducation reform. In schoolstoday, teachers must focus onacademic content and the com-petencies of students.VRROOM, the Virtual Refer-ence Room with its StandardsConnector resource web site andits new idea of classroom E-TEAMS, seeks to advance thecontent knowledge and techno-logical competencies of newteacher candidates, classroomteachers, and K–12 students.Technology alone cannot ensurethat successful teaching and

learning will happen in schools.Given guidance, support, andresources, educators and studentsalike will make greater use of theelectronic tools at their disposalto experience lessons and activi-ties full of innovation and learn-ing.

ENDNOTES 1Becker, H. J., Ravitz, J. L., & Wong, Y.Executive Summary: Teacher and Teacher-Directed Student Use of Computers andSoftware, November, 1999. [Online].Available: http://www.crito.uci.edu/TLC/findings/ComputerUse/html/body_startpage.htm. 2 “Percentage of Public School TeachersReporting Significant Use of Computers,Email, and the Internet at School, by Yearsof Teaching Experience.” Connection, Fall,2001. p. 17. 3 “Digital Teaching.” Connection, Fall,2001. p. 17. 4 Becker, H. J., & Ravitz, J. L.,“InternetUse on Teachers’ Pedagogical Practices andPerceptions.” Journal of Research onComputing in Education. Summer 1999,Vol. 31, No. 4. pp. 356-384.

learning in schools. An exten-sive database coded to thecurriculum frameworks is onlyuseful if teachers, new teachercandidates, and public schoolstudents use the resources tocreate lively, engaging classroomactivities. Indeed, studies haveshown that teachers who fullyintegrate computers, theInternet, and other technologiesin their classrooms are morelikely to engage in inquiry-based,problem-solving forms of teach-ing with their students.4

In an effort to supportclassroom teachers in increasingtheir digital literacy, theVRROOM partners have em-barked this year on the creationof E-TEAMS to serve high-needschool districts throughout thearea. An E-TEAM consists ofthe following members:• a public school teacher who isa participant in the NationalTeacher Training Institute(NTTI) program offered bypublic television station WGBY,• a content coach, generally agraduate student from theSchool of Education, well-versedin the uses of technology toenhance teaching and learning,and• a resource person from eitherthe Center for Computer-BasedInstructional Technology staff orthe School of Education faculty.

The coach acts as atechnology mentor to the teacherwho has the responsibility ofcreating technology-infusedlessons over the course of theyear that will be added to anational database of NTTIlessons. In this mentoring role,the coach provides expertise andresearch support, teaches modellessons in school classrooms,answers questions, and serves asa guide to web and video re-

Robert W. Maloy (MAT 1971) is alecturer in the Secondary TeacherEducation Program where he coordi-nates the history and political scienceteacher certification program, co-directs the 180 Days in SpringfieldProject, co-directs the TEAMS Tutor-ing Project, and co-directs the VirtualReference Room (VRROOM) andStandards Connector initiatives.

David Hart is Executive Director ofthe Center for Computer-BasedInstructional Technology (CCBIT) inthe Department of Computer Science.CCBIT created and maintains theVRROOM and Standards Connectorweb sites.

Paul Oh is a technology resourceteacher at the Mark's Meadow Schoolin Amherst, Massachusetts, and K-12Projects Director at CCBIT.

Victoria Getis, formerly K-12Projects Director at the Center forComputer-Based Instructional Tech-nology, is an instructional designer atthe Institute for Technology En-hanced Learning and Research at TheOhio State University.



ccountability hasbecome a fact of life in publiceducation systems across theUnited States. We have studentaccountability measures thatlead to adult accountabilitymeasures for schools anddistricts, but what exactly is“adult accountability?”Policymakers identify adultaccountability systems, likeschool accountability, as anessential characteristic in im-proving the public educationsystem. Bolstered by federallegislation and grounded instate education reform move-ments aimed at providingadequate funding levels for allstudents and schools, particu-larly those in low incomecommunities, accountability isthe stick that makes the carrotof increased funds from thestate to all schools palatable tolegislators and the public alike. Heading into unchartedterritory, states take the lead inholding adults in the localschools and districts account-able for student outcomes, butwhat if a school fails to meetthe standards set by the state?The answer to this question isdifferent in each state and has

been implemented only in themost dire cases. School ac-countability is the first step onthe journey to hold the adults inthe public education systemaccountable for student out-comes. The stakes are high. Ifthe past decade of standards-based reform is to be heraldedas a success, states must movebeyond the student accountabil-ity systems of tests toward thehard work of improving schoolsfor all our children. In 1993, the MassachusettsEducation Reform Act (MERA)was passed by the state legisla-ture. Aiming to ensure adequatefunding to all students withinthe public education system, thestate leaders mandated a com-prehensive accountabilitysystem be put in place to ac-count for the increase in stateaid to schools. A decade afterthe passage of the legislation,the state has just begun to holdschools accountable for studentperformance. To understandschool accountability in Massa-chusetts, it is helpful to use theanalogy of the state as thedoctor and the school as thepatient. Under an accountabilitysystem, the state (as the doctor)

is required to assess the healthof the school (as the patient): toobserve symptoms, to make adiagnosis, and to prescribetreatment. Clearly, comprehen-sive reform of public educationas mandated by MERA hasadded to the state’s traditionalrole in public education. Schoolaccountability is creating ascenario in which leaders in thestate education agencies arebecoming the doctors of publiceducation with under-perform-ing schools becoming theirpatients. But, what is the curethat the doctor can provide forthe patient? The complexities ofthe public education systemmake a universal remedy impos-sible. The research points tounique needs and customapproaches for nursing anunder-performing school backto health. This need for customapproaches may require greateramounts of expertise andgreater financial resources thanthe states currently have. The adult accountabilitysystem in Massachusetts startsat the school level, utilizingstudent scores on the Massachu-setts Comprehensive AssessmentSystem (MCAS) to determine

SCHOOL ACCOUNTABILITY

Susan Bowles


A

schools in need. Mirroringinitiatives in other states,MERA mandates a foundationlevel of funding for all schoolsin the Commonwealth (thecarrot) in exchange for a com-prehensive accountabilitysystem based on student out-come measures (the stick). TheCommonwealth is at a criticaljuncture in the implementationof accountability. If MERA is tobe deemed successful, it willneed to move beyond thesestudent tests to transform thedelivery of public education inschools that currently are notmeeting state standards. The state legislature has keptits promise to provide statefunds to schools as set out inMERA. However, the largersystem of adult accountabilityenvisioned within the legislationis ready to begin without thestate resources needed to imple-ment the system. In fiscal year2002 the state spent nearly 77percent of the education budgeton funds going directly toschools. Facing a declining stateeconomy, the percentage of thestate budget went up for schoolsbetween fiscal year 2001 and2002, however, the funds forthe adult accountability systemdecreased from .16 percent ofthe state education budget to.07 percent.1 Consequently, asof spring 2002, the state hadconducted diagnostic team visitsto only 32 out of 802 schools, or40 percent of schools consid-ered to be critically failing andnot meeting improvement goalsfor student performance in thefirst rating cycle. Of thoseschools visited, only a handfulwith the most flagrant problemshave been labeled “under-performing” and placed under

the auspices of state and districtattention, support, and techni-cal assistance. Essentially, thestate must rank schools in needfrom the bottom-up to functionwithin the parameters of limitedresources, personnel, time, andmoney that currently exist.Rather than ramping up thestate adult accountabilitysystem, there has been a de-crease in resources available.This undermines the concept ofsystemic reform driven by thecomprehensive accountabilitysystem envisioned in MERA. The first rating cycle forschools within the Common-wealth was based on 1999–2000 MCAS results. An analysisof the first rating cycle reveals80 schools considered “criticallylow” and “failing to meet”improvement goals, and anadditional 195 considered “verylow” and “failing to meet”improvement goals. Because ofthe high number of schools and

the limited capacity of the state,a balance had to be struck todetermine those schools that aremost in need. The first school diagnosticintervention team visits oc-curred in spring 2000. Duringthis part of the process a teamof state and local educatorsfamiliar with the improvementplans of the potentially under-performing school, visit theschool for a half a day, observ-ing classrooms and interviewingstaff and leaders within theschool. As of fall 2002, a totalof 32 schools had been selectedfor review, and the commis-sioner of education declared atotal of eight schools under-performing. Interestingly,another 15 schools were ini-tially declared under-perform-ing, but when the superinten-dents and principals of theseschools were notified, theyappealed the decision resultingin a new “under review” cat-


School Accountability

egory. The remaining nineschools were found to be ableto improve without outsideintervention from the state. This dynamic of the superin-tendent and principal challeng-ing the under-performingdetermination made by the stateis worth investigation. Whilethere may be exceptions, mostprincipals, superintendents, andparents want to provide thebest possible education forstudents. However, whenschools were declared to beunhealthy (under-performing),their communities, superinten-dents, and principals foughtback rather than accept the“support” from the state. Aprincipal interviewed about ateam visit to her school said,

…[O]ne thing I cantell you about the review: [it is]stressful… [I]t is how youmarket what you personallybelieve and … your mission andvision of taking your institutionto another level. You have tobe able to articulate that, and,above all, you have to be in theposition that if you think the[team visit] was not doneproperly, be ready to defendand be ready to rebut what youthink is not right because we aretalking about how [this will]effect the lives of students, theprofession of teachers, the beliefsystem of the public in aninstitution…

Plainly threatened by thestate visit, the principal’scomment reveals a crucialdifference in the state and thelocal vision of accountability. Inthe eyes of the state, a schoolmay have symptoms of illnessand be diagnosed as having acurable disease. The state (asthe doctor) will nurse theschool back to health by pro-viding extra care and negotiat-ing a prescription for treatmentwith the school (in the role of

the patient). The school, how-ever, views any diagnosis madeby the state as life threateningand even terminal. The naturalresponse of the school is to seeka second opinion before accept-ing diagnosis and to take drasticmeasures to fight for its life. This difference in under-standing may be attributed tothe recurring issue of localversus state control of thepublic education system. Adultaccountability measures withinMERA move the responsibilityfor educating children, onceentirely in the hands of the localcommunity, to the state. Schoolaccountability alters the tradi-

tional power boundaries andcontributes to the perceptionthat rather than helping theschool, the state will be usurp-ing control of the school awayfrom the community. The mediacontributes to this fear, assubstantiated by a principal inone of the potentially under-performing schools. He voicedhis dismay, referring to the way“the state and the media hadused this visit to shame [the]school within the community.”In a district during the secondyear of state team visits, asuperintendent remarked, “Themedia can make this processdevastating for the commu-

School InterventionDecision-making Framework

Each state accountability system has its own distinctive qualities, but todevelop any accountability system each must work through a decision-mak-ing framework. The following model developed after interviews and re-search on the activities of states currently embarking on the journey ofadult accountability.

Performance Criteria are the standards and measures according to whicha school (or district) is declared to be under-performing. These criteriamay include: student test scores, attendance rates, dropout rates and/orstudent scores disaggregated by race, ethnicity and SES.

Strategic Criteria are other criteria used to determine in which school ordistrict situations the state will intervene and to what degree. Factorssuch as state capacity and political considerations heavily influence thesedecisions.

Diagnostic Intervention is a first stage of intervention in which the stateanalyzes the situation of a school or district to determine the appropriatetactics and targets for corrective action. (Note: Many states have a gen-eral approach, such as a focus on “instruction” or “leadership.” Interven-tions will often be customized within the boundaries of the general ap-proach.)

Corrective Interventions emerge from the diagnostic process and in-clude a wide range of potential treatment/remedies, including self-study,financial assistance, expert assistance, professional development, rewardsfor progress, on-site audits/monitoring, probation, suspension of accredi-tation, transfer or replacement of staff, transfer of/choice for students,takeover, reconstitution, or closure.

Exit Criteria are the criteria according to which a decision to conclude orchange the intervention is made and the school is determined to be healthyor on the road to recovery.

Source: Bowles, S., Churchill, A., Effrat, E., & McDermott, K. School and District Intervention: A Decision Making

Framework for Policymakers. Center for Education Policy: Amherst, MA, 2002.



nity…” She added that becauseof the potential devastation,“…the superintendent must stepup to work with staff and bufferthe school.” A doctor andpatient relationship must be apartnership with some level ofdiscretion if it is to work—adifficult proposition in thepublic arena. Perhaps it is withthis understanding that since thefirst year of school accountabil-ity visits, the state has madespecial efforts to soften theblow within the communitieswhose schools are potentiallyunder-performing. Efforts todisseminate information, enlistdistrict superintendent support,3

and provide more informationto principals to manage fearsand anxiety within the commu-nity and among the staff withinthe school, reveal the state’sability to improve upon schoolaccountability practices. The state and district are themedical team that watches theschool closely and works withthe school to develop a regimenfor self-improvement. If thestate is the doctor, after thesecond year of visits the districtis conscripted to be the nursefor a school that is found to beunder-performing. Each of theunder-performing schoolsreceives technical assistance inusing data to drive improve-ment planning, additionalfunds, and greater attentionfrom the state and, ideally, thedistrict. District superinten-dents in partnership with thestate Department of Educationmonitor and provide technicalassistance to the school princi-pals and staff members. Still inits earliest phases of implemen-tation of adult accountability,the state has taken an evolution-ary approach to developing

policies and strategies to sup-port schools, learning from eachexperience and approachingschool improvement as some-thing in which the school, as thepatient, must find the strengthto improve from within, albeitwith the direction and supportof the state, serving as thedoctor. If the patient does not showsigns of recovery, the doctorstake drastic measures for a new,more intense level of treatment.According to MERA, after twoyears of showing no improve-ment by the subjective judgmentof the Board of Education, achronically under-performingschool may be reconstituted. Bydefinition, school reconstitutionis the disbanding of the staffand faculty within a school andstarting from scratch in themost extreme measures. InMassachusetts, reconstitutionmeans that the state Board ofEducation may fire a principal,and the local district superinten-dent may appoint a principalwith “extraordinary powers” tohire and fire employees. In the2002 Quality Counts publishedby Education Week, Massachu-setts is reported as one of 15states with reconstitutionpowers. To date, the Massachu-setts Board of Education hasnot reconstituted any schooldue to student performance onMCAS. As stated earlier, the stateassuming the role of doctor isnew to the field of schoolaccountability. MERA dramati-cally altered the state’s role inpublic education. Since 1993,the Massachusetts Departmentof Education has taken on manynew roles including that ofdeveloper of state standards andassessments as well as monitor,

enforcer, and supporter of thosedistricts and schools unable toperform at the levels establishedby the Massachusetts Board ofEducation. Required to makedeterminations about whether aschool is meeting the standardsset forth in MERA, the state hasgrappled with the conflictingroles of supporting strugglingschools and regulating them. Following in the footsteps ofother states, one might look tothose who had gone before tomake some determinationsabout what helps and improvesschools. Unfortunately, the cureis still not clear. School im-provement, as determined bystudent academic achievement,is complex. It takes years toaccomplish, and each scenarioof school improvement isattributed to unique localcircumstances. For example, acharismatic principal or teacherfrom inside the school withexemplary leadership skill canmove a school into transitionand improvement, so research-ers cite leadership as an impor-tant component of schoolimprovement. This and otherrecommendations about schoolimprovement seem obviousfrom a distance, but the pre-scription for any school treat-ment must take into account theexisting resources within theschool. Customized prescrip-tions based on expert analysisare needed; consequently, broadideas and suggestions forimprovement are not alwayshelpful and are more likelysituational. So, the Common-wealth continues to search forways to improve schools wherestudents are not achieving atstate-mandated levels. We already know the Massa-chusetts public education



system is failing students by thestandards developed by thestate. Only 69 percent of thestate’s tenth graders (Class of2003) passed both the Englishlanguage arts and mathematicsMCAS exams in spring 2001.This is significant becausemembers of the Class of 2003will be the first required to passthe English and mathematicsMCAS in order to receive adiploma. After the spring 2002retest results were releasedabout 80 percent of the Class of2003 had passed, meaning thatnearly 12,000 students had notpassed both tests and were noteligible for a diploma. Yet, as ofthe spring 2002, only two highschools, out of 45 rated “criti-cally low” and “failing tomeet” improvement goals, havebeen reviewed by a state teamfor under-performance. This isclear evidence that the adultaccountability system in Massa-chusetts is too far behind thestudent accountability systemand lacking the resources to beeffective. Since 1993, Massachusetts’education reform has revealedthe gaps in student achievementthat exist among schools in theCommonwealth. Now that thedoctors have identified the testto determine illness and dys-function in schools, it is time todiagnose and treat the deepercauses of this problem. Com-prehensive reform, legislated inMERA, has created a situationwhere the students have becomevictims of an accountabilitysystem that is using them assymptoms to diagnose largerproblems within public educa-tion. We are, essentially, treat-ing the symptoms of a cold,when pneumonia may belurking deeper within the

system. Faced with a limitedsupply of resources for doctorsand high demand for them fromthe undiagnosed patients withsymptoms of illness, the statefaces a dilemma. The legislature and thegovernor have kept the promiseof providing additional funds toschools, but to keep the promiseof holding schools accountable,a significant investment in state-level accountability is vital.Otherwise, we will continue totreat those schools that are inthe most need, while those thatmay need less, but are stillfailing by state measures, gountreated. On their own, theseschools may improve, or theymay not, but without a statesystem available to supportthem, Massachusetts leaders aregambling with what maybecome an epidemic of chronicunder-performance. Challengesabound. There is no clear oruniversal remedy for under-performing schools. It may takeyears to improve those schoolswith the most egregious prob-lems, and researchers and stateleaders may have difficultyputting a price on the endeavorof healing schools. Adequatefunding of our schools is essen-tial, but to truly transformeducation through the existing

legislation, we must also bewilling to invest fully in thedoctors who will work tosupport and to improve theeducation of children in allMassachusetts schools.

ENDNOTES 1 These figures are based on datafrom the Massachusetts Board ofEducation “Budget Analysis” of K-12education found within the 2001Board of Education Annual Reportand the state budget for the Office ofEducational Quality and Accountabil-ity, part of the Governor’s office andoutside of the Department of Educa-tion. 2 Figures calculated from theMassachusetts Department ofEducation’s School PerformanceRatings 2000 (September 2002). 3 Referring to a superintendent’sgroup comprised of superintendentsfrom larger urban districts that metregularly with the Director of Account-ability and Assessment at the Massa-chusetts Dept. of Education in thewinter and spring of 2001.



Susan Bowles (M.Ed. 2002) iscurrently pursuing her doctorateat the University of Massachu-setts Amherst School of Educa-tion. She is a research assistantat the University of Massachu-setts Amherst Center for Educa-tion Policy and works closelywith staff and faculty at theCenter on a variety of Massachu-setts K-12 education policyprojects.

The Federal No Child Left Behind ActExpands the State Sanctionsto Under-performing Schools

Recent federal legislation reauthorizing the Elementary and Second-ary Education Act passed in late 2001, better known as the No ChildLeft Behind Act, increases the state’s powers to intervene within aschool so that after four years of chronic underperformance, or notmaking “adequate yearly progress,” a school may be reconstituted,taken over by the state, placed under private management or replacedby a charter school. This in effect increases the threat of the stateusurping local control of schools in all states wishing to continue re-ceiving federal education funds.

arly in his presidency, Franklin D.Roosevelt commented on his effort to reform thefederal government. He said,

The Treasury is so large and far flungand ingrained in its practices that Ifind it almost impossible to get theactions and results I want…But theTreasury is not to be compared to theState Department. You should gothrough the experience of trying to getany changes in thinking, policy, andaction of the career diplomats, andthen you’d know what a real problemis. But the Treasury and StateDepartment put together are nothingas compared to the Navy….To changeanything in the Navy is like punchinga feather bed. You punch it with yourright and you punch it with your leftuntil you are finally exhausted, andthen you find the damn bed just as itwas before you started punching. 1

I would venture to say that more than a fewMassachusetts school superintendents andprincipals felt much like President Rooseveltwhen confronted with the education reformmandate of 1993. To say that the publiceducation system is analogous to the Navy wouldbe an understatement. Reform of the system hasbeen an agenda item for decades, always, itseems, with the same result: an adjustment hereor an adjustment there, the appearance of somechange, with the “steady state” prevailing in theend. The feather bed emerges slightly rumpled,but otherwise unscathed.

STRUGGLE TO REFORMThe reason our reform efforts struggle to producesignificant or lasting results is because they arelegislated and commanded by the state with greatspecificity and detail. Oversight and regulation,with accompanying paperwork, become the orderof the day and the time committed to such thingsis taken from other endeavors relevant to theeducation of children. Our current effort started out broad-based,with a promise to deliver systemic change, andthen quickly seemed to focus on the thingsgovernment can actually control and afford, suchas curriculum frameworks and high stakestesting. By 2003, any child who has not passedthe MCAS test in mathematics and English willnot graduate from a public high school. Not tobe outdone, the federal government authorized itsown version of command reform in the No ChildLeft Behind (NCLB) Act. Like many otherfederal initiatives, it is long on regulation(accountability) and short on financial support.

CURRENT REFORM SETS A PRECEDENTThe state-level education reform of 1993, itshould be pointed out, is different from previousmanifestations because it sets a precedent bymoving important aspects of public educationfrom local to state control. Because this wasaccomplished by shifting from local curriculumstandards to state standards (curriculumframeworks and MCAS testing), many are stillunaware of what has happened. Similarly, thefederal government has become more seriouslyinvolved in public education by demanding

MANAGING TO LEAD IN THE DECADE

OF EDUCATION REFORM

Francis L. Gougeon


E

adherence to certain regulationsin order for local schools tocontinue to receive federalfunds. This new degree ofinvolvement by the federalgovernment is, as many areaware, out of proportion to theoverall financial contribution itmakes to local school systems.This shift to greater state andfederal control of publiceducation has been made in thename of standard setting andaccountability, which issomething many would agree isa worthwhile goal, whilereserving the right to disagreewith the method employed toachieve it. The tension all this hascreated surfaces periodically inthe form of a resolution orpronouncement by local schoolcommittees stating that theywill award diplomas to theirstudents whether they havepassed the MCAS test or not.This is usually followed by anadvisory from the Departmentof Education reiterating thatassessment testing is required bystate law and any local decisionto the contrary will be inviolation of that law.

SHARPENING THE FOCUSWhen reform reaches the pointwhere we shift from looking atthe system as a whole, usuallybecause of declining resources,to focusing exclusively onspecifics like curriculumframeworks and testing, weseem to lose sight of the impactthis action is having on allaspects of the schoolorganization. If we take a minute to lookat curriculum revision(frameworks) and mandatedhigh stakes testing (MCAS), wesee ripples throughout the entiresystem, bumping up againsttime constraints (planning andrevising), personnel (adequatenumber of staff and staffdevelopment), students(readiness and remediation),budget (adequate resources),and administration (staffdevelopment opportunities forretraining).

IMPACT OF REFORMON ADMINISTRATORS

If we focus on administration inparticular, we can see howcurriculum revision and testingmight be problematic. At the

outset it appears to bode well,because the latest reformenhances the authority ofsuperintendents and principalswhile decreasing the authorityof school committees. The idea,which many agree with, is tohave less micro-management byelected officials while placingincreased responsibility in thehands of on-site appointedofficials. The reformsuperintendent then becomes aCEO and the school committeesomething akin to a board ofdirectors. The buildingprincipal, similarly transformed,becomes a unit or divisionmanager with a correspondingincrease in authority. Bothsuperintendent and principal areempowered to make importantpersonnel decisions, as well asdecisions regarding budget,curriculum, staff development,and goal setting. It wouldappear that with this newauthority administrators shouldbe in an ideal position toimplement education reform,i.e., revise the curriculum,institute standards-basededucation, and move studentsalong a continuum ofimprovement from failing toadvanced standing. Unfortunately, the doingturns out to be morecomplicated than simplyexpanding administrativeauthority. The post-1993administrator, in addition tohaving greater authority, hasalso assumed much moreresponsibility. In the new era,he or she is judged, to a greatdegree, on how well students doon the state’s assessment tests.Success or failure at this pointhinges on student scores, asdoes the designation of a schoolas exemplary or on probation.


All of this has changed thedynamic within schools andschool systems, making the roleof administrator much moredemanding and complicated.Some even say that schools aredangerously close to obsessingover testing, which can makethe administrator’s relationshipwith the staff more difficult.Superintendents may pressureprincipals, who in turn pressureteachers and staff.

THE BIG PICTURE:CAPACITY OF

ADMINISTRATORSIn retrospect, if a seriousanalysis of the educationalsystem was undertaken beforelaunching reform (some wouldargue there was), policymakersmight have become more awareof the actual readiness of schooladministrators to undertake thechallenge of reform. In otherwords, since we have aworthwhile goal (improvingstudent achievement), what isour administrative capacity tolead this initiative? Do we havethe training and resources inplace that are necessary to dothe job? If such a thoughtful analysishad been done, we would havediscovered that prior to 1993administrators, in general, andprincipals, in particular, werefocused primarily on managerialtasks such as budget/finance,interviewing/hiring, supervision/evaluation, organization of theschool day and year, as well asthe establishment of a variety ofschedules. Many schooladministrators, by their ownadmission, were comfortablewith this well-defined role, andlongtime service was the rule inthis era and not the exception.

On closer analysis it wouldhave been determined thatplacing greater emphasis oncurriculum revision andmandatory testing would, ineffect, substantially change therole of the administrator. Itwould have been clear that withthe advent of standards-basededucation, much more would beexpected from the administratoras the position shifted (out ofnecessity) from manger tomanager/leader—which is nosmall distinction. Even systemsthat realized this on their ownhad problems finding the timeand financial resources to dothe necessary administrativeretraining. In order to functioneffectively in a reformenvironment, administratorswould have to acquire an arrayof leadership skills, such asarticulating a vision anddeveloping shared decisionmaking, systems analysis, staffdevelopment, performancestandards for teachers,mentoring teachers, curriculumalignment, and addressingstudent failure. These are skillsthat many administrators tell usthey do not have. With theadditional issues of time andincreased paperwork, it is easyto see why many principalshave expressed concern abouttrying to learn a new role whilesimultaneously being heldaccountable for theimplementation of educationreform.

SUCCESSFUL REFORMIn efforts at education reform,we are drawn to the machineapproach because it promiseswhat we want—the illusion of aquick fix. Bolt on this part (the

curriculum frameworks) or thatpart (the MCAS), and themachine (the school system) willrun smoothly. However, if weare going to be successful witheducation reform, we have tostop treating the school systemlike a machine. Educationreform must be based on seeingthe system as a whole. We mustunderstand that before weimplement standards andtesting, we need the time andresources to provide thenecessary prerequisites forsuccess. Finally, President FranklinRoosevelt’s view of organiza-tions like the Navy was probab-ly no more comprehensive thanour view of public education.Like President Roosevelt, if wedo not make the effort to basereform on a complete under-standing of the organization, wewill continue to “find the damnbed just as it was before [we]started punching.”

ENDNOTE 1Edwards, III, G. C., & Sharkansky,I. The Policy Predicament: Makingand Implementing Public Policy. SanFrancisco: W. H. Freeman, 1978.

Francis L. Gougeon is alecturer in the Department ofEducational Policy, Research andAdministration. His extensiveteaching and administrativeexperience includes serving asSuperintendent of Schools in theHatfield and the Granby,Massachusetts, public schoolsystems. He also has served asPresident of the ConnecticutValley Superintendents’Roundtable and is currentlypursuing doctoral studies with aconcentration in Policy andLeadership at the University ofMassachusetts Amherst.


Managing to Lead

lthough passage oflegislation grabs all theheadlines, it is really only thefirst step of making policy.Public agencies then begin thefar less visible process of policyimplementation—translatingabstract legislative languageinto concrete regulations andprograms. Implementing policyis basically a matter of ensuringthat the legislation has itsintended consequences whilealso heading off negativeunintended consequences.When the policy is ambitious, itmay be necessary to makefundamental changes in how theaffected government agenciesfunction. The MassachusettsEducation Reform Act (MERA)of 1993 fits in this category.Historically, local control hasbeen “the ‘Battle Hymn of theRepublic’ for New Englandeducators.” 1 MERA challengedthe local districts’ autonomy byempowering the state toproduce curriculum frame-works, test students’achievement, and hold districtsaccountable for studentperformance. Ultimately, thestate might even take overmanagement of a failing schoolor district. In order to carry out

these major tasks, the stateDepartment of Education(DOE) had to expand itsactivities in related areas, suchas collection and use of data,professional development foreducators, and technicalassistance for schools anddistricts. Although MERA wasfairly specific about what thestate would be doing, itprovided little guidance as tohow, and with what resources. Since 1993, many observersinside and outside the stategovernment have questioned thestate’s capacity to implementthe MERA. Three issues mustbe considered in analyzing statecapacity to implement a law:the staff and financial resourcesavailable within the relevantstate agencies, the staff andfinancial resources outside theagency, on which the agencymight reasonably expect todraw, and the organizationallinks among all these resources.

INTERNAL RESOURCESTo a great extent, the seeds ofDOE’s capacity problem weresown by the same economiccircumstances that produced thepressure for MERA. In 1980,DOE had 990 full-timeequivalent (FTE) staff. Staffing

decreased throughout the1980s, particularly during therecession of the late 1980s andearly 1990s. When MERApassed in 1993, DOE’s stafftotaled only 325 FTE. As statefunds evaporated, theproportion of DOE staff whowere in federally fundedpositions increased from 37percent in 1980 to over half inthe late 1990s. DOE has lessflexibility in how it usesfederally funded staff becausetheir positions are directly tiedto specific federal programssuch as Title I. As economic conditionsimproved in the mid-1990s,DOE did not recover many ofits lost positions. GovernorWilliam Weld and hisappointees in the otherexecutive branch agencies werecommitted to curbing growth inthe state workforce. Membersof the Democratic-controlledlegislature might havesupported more staff for DOE,but did not want to make thefirst move and risk beingportrayed as big spenders by theRepublicans. DOE filled someof its staffing gaps with contractworkers (also known asconsultants or “03s”), but itcontinued to face increased

CAPACITY TO IMPLEMENT

EDUCATION REFORM

Kathryn A. McDermott


A

responsibilities with a decreasedworkforce until only a few yearsago. Between 1998 and 2002,108 new positions were addedto DOE. However, the currenteconomic downturn and statebudget cuts threaten thesegains.2

DOE acts as a conduit ofstate funds for local districts,but does not receive muchfunding for programadministration and evaluation.In fiscal year 2002, for example,DOE received $3.9 billion instate funds, $3.3 billion ofwhich it paid back out as aid todistricts.3

EXTERNAL RESOURCESFor many early advocates ofeducation reform, DOE’s lackof staff did not seem especiallyproblematic. They werecommitted to a theory of reformas a “tight-loose” process.According to this theory,schools were likeliest toimprove when the state was“tight” in specifying goals forstudent achievement but“loose” in the sense of notclosely regulating how the goalswere attained. DOE’s role

would essentially be to establisha framework of incentives forreform, leaving superintendents,principals, teachers, and otherproviders of educationalservices to plan and carry outthe actual work. This minimalrole could be carried outwithout a large staff. In a book called DemocraticGovernance, James G. Marchand Johan P. Olson point outthat the capacity of any givengoverning institution depends inlarge part upon the capacities ofothers.4 Viewed in these terms,the “tight-loose” vision ofeducation reform would haveallowed the relatively weakDOE to harness the energy andcapacities of local educators,who were also closest to theactual teaching and learningprocess. Early in the process, itlooked like the “tight-loose”model might work. The DOEwent out of its way to involvelocal educators and laypeople indeveloping a common core oflearning that would unify andprovide context for the subject-area curriculum frameworks.By the DOE’s estimate, about50,000 people participated inthe process in some way.Educators saw the commoncore and the curriculumframeworks as part of a largerproject of reforming curriculumand pedagogy alongconstructivist lines, which theysupported.5

However, the tight-looseassumptions proved inaccuratein two respects. First, manylocal school districts facedcapacity challenges at least asgreat as the DOE’s. Aligninglocal curricula with stateframeworks required skills thathad not been part of most

educators’ training orexperiences. Some of the largerschool districts hadadministrators who could leadthis process, but the smallerones did not. Given all thesechallenges, many educatorsresisted the process andcriticized the state for failing toprovide them with adequatesupport and direction. The second challenge to the“tight-loose” model was thatrather than being “tight,” statepolicymakers’ goals changedcourse several years intoeducation reform. John Silber,who became chair of the stateBoard of Education in 1996,was an outspoken critic of thepublic schools, of the“education establishment,” andof the education reformassumption that all children canlearn. In a 1998 Boston Heraldinterview, Silber said, “When Icame on [the board], I foundthat the board had endorsed thestatement that all children canlearn at high levels. Thathappens to be false, and it isirresponsible for anybody tomake a claim of that sort.”6

Under Silber, the boardrevised the curriculumframeworks, eliminating theirconstructivist tendencies. Theeducators who had beeninvolved in producing thecommon core and theframeworks, and in aligningtheir schools’ curricula with thestate requirements, opposed thenew direction and criticized thestate for changing its goals tosuit leaders’ whims. Problemswith the first administration ofthe qualifying test for teachers,and state policymakers’ chargethat the prospective teacherswho failed the test were“idiots,” reinforced the


Capacity to Implement Reform

impression that the state wasdeliberately making schools andteachers look bad. Local schoolcommittees, administrators, andteachers lost their desire tocooperate with the educationreform process. In the absenceof voluntary cooperation, theDOE lacks the capacity to makeeducation reform happen.

ORGANIZATIONAL LINKSCommunication and inter-organizational relations havealso been problematic in threekey sets of relationships:between state and localeducational authorities, withinDOE itself, and between DOEand other offices within thestate government. Many tasks related toeducation reform, such asaltering curricula in response tostudents’ MCAS scores anddetermining which schools thestate should review because theyare potentially “under-performing,” require timelysharing of data between stateand local education authorities.Local administrators resent theheavy reporting requirements ofstate and federal educationpolicies. Districts must providedata for 15 different statereports, generate four of theirown state-mandated reports,and produce five requiredannual plans.7 DOE’s staffingshortages have hindered itsability to communicate databack to districts. In a 2001survey, a majority of localeducators indicated that they donot believe DOE makes gooduse of the data it collects orfeeds useful information back todistricts and schools.8

Within the state,communication and allocationof responsibilities pose

problems. When interviewed in2001, many DOE senior staffidentified lack of communica-tion among staff clusters as aproblem.9 Competition betweenDOE and other state officesover components of MERAhave slowed the law’s imple-mentation. This problem hasbeen most acute in the area ofschool and district account-ability. Late in 1999, the Boardof Education approvedregulations on how under-performing schools and districtswould be identified. TheAccountability and TargetedAssistance cluster within DOEbegan working on site visits toschools where the state mightintervene, and on planning forwhat interventions would becarried out. However, in thesummer of 2000, the statelegislature approved a proposalby then-Governor Cellucci toshift the school and account-ability function out of DOE andinto the Office of the Governor.It took about a year for thegovernor (by this point, ActingGovernor Jane Swift) to appointmembers to the EducationManagement Audit Council andseveral more months for theOffice of Educational Qualityand Accountability to begin itswork.

WHY THE STATE CAPACITYISSUE MATTERS

Ten years after MERA, the law’sdifferent components have beenunevenly implemented. Thisunevenness reflects differencesin the level of administrative ortechnical challenge that variousparts of the law posed for stategovernment. The most crucialparts of the law—reforming thetechnical core of teaching andlearning and holding adults

accountable for studentperformance—have made thegreatest demands on capacityand thus have been the leastcompletely implemented.Whatever the outcome of thepending lawsuit claiming thatstudents have not had asufficient opportunity to learnthe material included on thetenth-grade MCAS, the state’sshortcomings in capacity andthe reasons for thoseshortcomings are likely to loomlarge in the trial.

ENDNOTES 1Jerome T. Murphy, “Title I of ESEA:The Politics of Implementing FederalEducation Reform,” in Allan R. Odden,ed., Education Policy Implementation.Albany: State University of New YorkPress, 1991, p. 34. 2 DOE staffing information is drawnfrom Kathryn A. McDermott, et al., AnAnalysis of State Capacity to Implementthe Massachusetts Education Reform Actof 1993 (Massachusetts EducationReform Review Commission, 2001) andAndrew M. Churchill, et al., 2002Annual Report on the Progress ofEducation Reform in Massachusetts(Massachusetts Education ReformReview Commission, 2002). 3 Churchill, et al., 2002. 4 James G. March and Johan P. Olsen,Democratic Governance. New York: TheFree Press, 1995, p. 107. 5 Dan French, “The State’s Role inShaping a Progressive Vision of PublicEducation.” Phi Delta Kappan,November 1998, pp. 185-194. 6 J. Sullivan, “True Test: Silber Gives Ed-Reform Answers.” Boston Herald, May3, 1998, p. 18. 7Churchill, et al., 2002. 8 McDermott, et al., p. 66. 9 McDermott, et al., p. 15.

Kathryn A. McDermott is anassistant professor in theDepartment of EducationalPolicy, Research andAdministration and is anAssociate Director of the Centerfor Education Policy at theUniversity of MassachusettsAmherst.


Capacity to Implement Reform


ith support fromthe Massachusetts EducationReform Review Commission,the Center for Education Policyat the University of Massachu-setts Amherst conducted acomprehensive study of theimplementation status of theMassachusetts EducationReform Act (MERA), resultingin the recently released 2002Annual Report on the Progressof Education Reform inMassachusetts. The reportpresents primarily quantitativedata on the Commonwealth’sstatus in five areas: the contextfor education reform, financeaspects of education reform,standards-based reforms,student achievement and gaps,and capacity for educationreform at state and local levels.Based upon this research, weoffer the following conclusions.

AREAS OF SUCCESS1. The legislature has broughtvirtually all districts to acommon level of foundationfunding. When the legislaturecommitted itself in1993 tosupporting an unparalleledincrease in state support for

K–12 education, few thoughtthey would actually do it. Theydid it, and a greater level ofdistrict equity is the result.

2. The Department ofEducation (DOE) has developedhigh-quality, well-alignedstandards and assessments.There is growing recognition,supported by externalevaluations, that Massachusettshas some of the best standardsand assessments in the country.The use of quality standards toalign everything from teacherpreparation to classroominstruction to school anddistrict accountability is a trulysystemic reform effort.

3. The state is paying moreattention to the needieststudents. While controversial intheir implementation, thesystem of standards andassessments has served toilluminate the persistentinadequacy of education formany of our young people whomight previously have fallenthrough the cracks. Thefoundation formula andsubsequent appropriations for

Massachusetts ComprehensiveAssessment System (MCAS)remediation have significantlyboosted resources for thosestudents who historically havebeen the least well-served.

4. Districts are paying moreattention to curriculum andinstruction. With relatively clearstandards, and as the standardsstabilize and assessment resultscome in, schools have been ableto re-examine their offeringsand align them both internally,from grade to grade, andexternally with the curriculumframeworks. The majority ofdistricts appear to haverealigned their curricula, teacherpreparation programs haveexamined their standards andalignment, and professionaldevelopment has beensupported as a matter of statepolicy.

5. Students, overall, areinproving their performance.The passing threshold is stilllow (Needs Improvement), andthe achievement gaps are causefor real concern (see below), but87 percent of regular-education

CONCLUSIONS:

THE IMPACT OF EDUCATION REFORM

AFTER TEN YEARS

Andrew Churchill

W


students in the Class of 2003have been able to pass theMCAS to date. Englishlanguage arts performance isimproving across the board.Improved NAEP results andrelatively high SAT and TIMMSresults further indicate that, onaverage, Massachusetts studentsare making significant progress.

1. Achievement gaps. Whileregular-education students arefor the most part meeting theminimum graduation standard,special education students andparticipating LEP students, onaverage, are not. Students frompoor districts, in general, arenot. And African-American andHispanic students, on average,are not. Further research isneeded to delve into therelationships between thesevariables to identify key areas ofopportunity. At the least,attention to instruction,resources, and a realistictimeframe for success will beimportant.

2. Mathematics. The state hasmade progress in Englishlanguage arts performance, butin mathematics, particularly atthe 4th and 8th-grade levels,progress has been lesssatisfactory. One-third of 8th-graders are still scoring at theWarning level in mathematics,and the percentage in theAdvanced or Proficientcategories has increased onlyslightly, from 31 percent to 34percent over the past five years.Educators have successfullyplaced increased emphasis onliteracy and writing across thecurriculum in response to thewriting components of MCAS;

new approaches to mathematicsnow seem warranted as well.

3. Proper implementation foraccountability. Effectiveimplementation is as importantas high standards andsystematic assessments. Anumber of stakeholders havenoted a significant gap betweenthe implementation of studentaccountability and theimplementation of adultaccountability. Others havecountered that high stakes forstudents were necessary ifstudents and educators were totake the goal of meetingstandards seriously. Havingreached this point, it will beimportant to ensure thatstudents and educators havesufficient “opportunity tolearn” resources to enable themto meet the standards (includingcurriculum aligned to standards,teacher professionaldevelopment, current textbooks,and time for these to takeeffect). If the resources andtimeline are insufficient, this

could have dire consequencesfor both individual students andthe collective reform effort.

4. Post-12th grade options. IfMassachusetts is to maintainthe expectation that all studentscan reach common standards,post-12th grade pathways willbe an important vehicle forensuring that the system canaccommodate student needs, interms of both time andresources. As of the publicationof our report, no programs,policies, or resources had beenfinalized for meeting thislooming challenge. Post-12th

grade pathways to competencydetermination would representsignificant progress toward asystem in which standards areheld constant and time spent onlearning varies with studentneeds—as opposed to thetraditional system of time beingheld constant and learningvarying among students.

5. From Needs Improvement toProficient. Massachusetts hasset an ambitious target ofgetting all students to meet theminimum standards requiredfor competency determinationstatus. However, it should notbe forgotten that NeedsImprovement is an intermediatestage, and that the standard ofProficient is the level that moreclosely approximates the skilllevels required for college-levelwork and/or skilled careers.

6. Educator supply, quality, anddistribution. As standards havebecome more ingrained,attention has begun to shift tothe critical importance of highlyqualified teachers in helpingstudents meet the standards.Looming retirements of

AREAS FOR FURTHER WORK


experienced teachers,insufficient retention of newteachers, teacher shortages inurban schools, teacher shortagesin certain content areas(especially mathematics,sciences, special education, andlanguages), higher expectations,and a dwindling supply ofprincipals present formidablechallenges. New methods ofpreparation, induction/mentoring, professionaldevelopment, scheduling, andpay will all requireconsideration and evaluation.

7. Updating the fundingformula. Our research appearsto indicate that the currentformula inadequately estimatescosts in some areas. Now thatwe have a number of years’experience with the demands ofeducation reform, it is time torevisit the formula to ensurethat it reflects the real demandsof the new environment. Inaddition to accurately fundingthe costs of regular education,attention should be paid toensuring that the special costs ofbringing all students tocompetency are supportedequitably.

8. Improved systems for data-based decision-making. MERAhas generated a great quantityof potentially useful data, andthe implementation of the NoChild Left Behind Act willgenerate much more. Districtsare already overburdened withinformation requests, and theyfeel that they have littleopportunity or capacity to makeuse of the information that iscollected. The DOE has recentlymade TestWiz softwareavailable to all districts to helpthem analyze their MCAS

results. This is a very positivestep, but issues of local andstate data capacity requirecontinued effort. Moreattention will need to be paid tomaking the overall datagathering and distributionsystem more efficient,connected, and effective—bothat DOE and at the local level.

9. Re-connection of policy-makers and educators in thefield. A great deal has beenaccomplished to date, but manyfront-line educators have cometo feel disengaged from theprocess. As the state thinksabout fine-tuning MERAimplementation, it will beimportant for the governor, thelegislature, the Board ofEducation, the Department ofEducation, and the variouseducation associations to agreethat education reform is aworthy goal, that we have muchto celebrate, and that we willmove forward together.

10. Continued monitoring andevaluation of the reform effort.CEP’s Annual Report is a usefulstep in the collection andanalysis of information aboutthe successes and challenges ofMERA, but there are a numberof areas for future research.These include analyses of thefollowing areas:• various achievement gaps andhow they interact, usingstudent-level data;• foundation budget formulaand the real costs of ensuringthat all students have theopportunity to learn in astandards-based environment;• exemplary programs for help-ing special education, LimitedEnglish Proficient (LEP), andhigh-poverty students to reach

achievement goals;• roles of postsecondary andcommunity institutions inexpanding opportunities forstudents to meet standards;• supply and demand “pipeline”for teachers and administrators;• effectiveness of various teacherpreparation, induction, andprofessional developmentapproaches; and• data needs and capacity of thestate and local districts and howa data system might bedeveloped to accommodatethem.

Andrew Churchill is AssistantDirector of the Center forEducation Policy. He has amaster’s degree in public policyfrom the University of CaliforniaBerkeley and has worked in localschool districts, state and federalagencies, and national researchorganizations.

Conclusions: Impact of Education Reform

The Centerfor Education Policy

Through the Center forEducation Policy (CEP), Universityof Massachusetts Amherst Schoolof Education faculty andgraduate students conductstudies, convene conferences,and evaluate programs on policytopics relating to K–12 educationreform and K–16 educationalalignment and transitions.Research topics have includedtopics such as teacherrecruitment and retentionprograms, state testing andgraduation requirements, stateintervention options forsubstandard schools and districts,state capacity to implementcomprehensive reforms, and theimpact of various policies oncollege access and attainment.

For more information on this studyor the Center for Education Policy,contact Andrew Churchill [email protected].

Education REFORM

Documents

Transcript of Education REFORM