The monitoring, Evaluation, reporting, verification, and certification of energy-efficiency projects

THE MONITORING, EVALUATION, REPORTING, VERIFICATION,AND CERTIFICATION OF ENERGY-EFFICIENCY PROJECTS

EDWARD L. VINE and JAYANT A. SATHAYEEnergy Analysis Department, Environmental Energy Technologies Division, Lawrence Berkeley

National Laboratory, Berkeley, CA 94720, USA

(Received 23 Aptril 1999; accepted in final form 14 February 2000)

Abstract. In this paper, we present an overview of guidelines developed for the monitoring, eval-uation, reporting, verification, and certification (MERVC) of energy-efficiency projects for climatechange mitigation. The monitoring and evaluation of energy-efficiency projects is needed to determ-ine more accurately their impact on greenhouse gas (GHG) emissions and other attributes, and toensure that the global climate is protected and that country obligations are met. Reporting, verific-ation and certification will be needed for addressing the requirements of the Kyoto Protocol. Whilethe cost of monitoring and evaluation of energy-efficiency projects is expected to be about 5–10% ofa project’s budget, the actual cost of monitoring and evaluation will vary depending on many factors,including the level of precision required for measuring energy and GHG reductions, type of project,and amount of funding available.

Keywords: certification, Clean Development Mechanism, energy efficiency, evaluation, global cli-mate change, greenhouse gas emissions, joint implementation, monitoring, reporting, verification

1. Introduction

Because of concerns with the growing threat of global climate change from in-creasing concentrations of greenhouse gases (GHG) in the atmosphere, more than176 countries (as of October 7, 1998) have become Parties to the U.N. FrameworkConvention on Climate Change (FCCC) (UNEP/WMO 1992). The FCCC enteredinto force on March 21, 1994, and the Parties to the FCCC drafted the KyotoProtocol for continuing the implementation of the FCCC in December 1997 (UN-FCCC 1997). The Protocol requires developed countries to reduce their aggregateemissions by at least 5.2% below 1990 levels by the 2008–2012 time period.

The Kyoto Protocol includes two project-based mechanisms for activities acrosscountries. Article 6 of the Protocol allows for joint implementation (JI) projectsbetween developed (Annex I) countries: i.e., project-level trading of emissions re-ductions (transferable ‘emission reduction units’) can occur among countries withGHG emission reduction commitments under the Protocol. Article 12 of the Pro-tocol provides for a Clean Development Mechanism (CDM) that allows legal en-tities in the developed world to enter into cooperative projects to reduce emissionsin the developing world for the benefit of both parties. Developed countries willbe able to use certified emissions reductions from project activities in developing

Mitigation and Adaptation Strategies for Global Change5: 189–216, 2000.© 2000Kluwer Academic Publishers. Printed in the Netherlands.

190 EDWARD L. VINE AND JAYANT A. SATHAYE

countries to contribute to their compliance with GHG targets. The key provisionsof the Kyoto Protocol remain to be developed in more detail as negotiations clarifythe existing text of the Protocol.

Projects that are to be undertaken within the CDM or under JI will involveseveral tasks: project development and registration; project implementation; andmonitoring, evaluation, reporting, verification and certification. There will mostlikely be different types of arrangements for implementing these projects: e.g., (1)a project developer might implement the project with his/her own money; (2) adeveloper may borrow money from a financial institution to implement the project;(3) a developer may work with an energy service company who will be responsiblefor all project activities; etc. While the flow of funds might change as a result ofthese different arrangements, the guidelines described in this paper should be ofrelevance for all parties, independent of the arrangement.

1.1. PURPOSE OFMERVC GUIDELINES

Monitoring, evaluating, reporting, verifying, and certifying (MERVC) guidelinesare needed for joint implementation and CDM projects in order to determine moreaccurately their impact on GHG and other attributes (see Box 1) (Vine and Sathaye1997). Implementation of MERVC guidelines is also intended to: (1) increasethe reliability of data for estimating GHG impacts; (2) provide real-time data soprograms and plans can be revised mid-course; (3) introduce consistency and trans-parency across project types, sectors, and reporters; (4) enhance the credibilityof the projects with stakeholders; (5) reduce costs by providing an international,industry consensus approach and methodologies; and (6) reduce financing costs,allowing project bundling and pooled project financing.

These guidelines are important management tools for all parties involved incarbon mitigation. We expect official guidelines to be developed by the Confer-ence of Parties of the FCCC. However, we expect that there will be a certainamount of discretion in the implementation of the guidelines and rules. In fact,there will likely be different approaches (models) in how the monitoring, eval-uation, reporting, verification, and certification of energy-efficiency projects willbe conducted: e.g., a project developer might decide to conduct monitoring andevaluation, or might decide to contract out one or both of these functions. Verifica-tion and certification will most likely be implemented by independent third parties.Similarly, some projects might include a portfolio of projects. Despite the diversityof responsibilities and project types, the Lawrence Berkeley National Laboratory’s(LBNL) MERVC guidelines should be seen as relevant for all models and projectapproaches. While the focus of this paper is on energy-efficiency projects, theissues will be similar for energy supply and non-energy (e.g., forestry) projects; forthese other projects, other issues may be more important than for energy-efficiencyprojects (e.g., sustainability of carbon (C) sequestration in forestry projects).

ENERGY-EFFICIENCY PROJECT 191

Box 1. Definitions of project-level activities

Estimation: refers to making a judgement on the likely or approximate energy use, GHG emissions, andsocioeconomic and environmental benefits and costs in the with- and without-project (baseline) scenarios.Estimation can occur throughout the lifetime of the project, but plays a central role during the project designstage when the project proposal is being developed.

Monitoring: refers to the measurement of energy use, GHG emissions, and socioeconomic and environmentalbenefits and costs that occur as a result of a project. Monitoring doesnot involve the calculation of GHGreductions nor does it involve comparisons with previous baseline measurements. For example, monitoringcould involve the number of compact fluorescent lamps installed in a building. The objectives of monitoringare to inform interested parties about the performance of a project, to adjust project development, to identifymeasures that can improve project quality, to make the project more cost-effective, to improve planning andmeasuring processes, and to be part of a learning process for all participants (De Jong et al. 1997). Monitoringis often conducted internally, by the project developers.

Evaluation: refers to both impact and process evaluations of a particular project, typically entailing a morein-depth and rigorous analysis of a project compared to monitoring emissions. Project evaluation usually in-volves comparisons requiring information from outside the project in time, area, or population (De Jong et al.1997). The calculation of GHG reductions is conducted at this stage. Project evaluation would include GHGimpacts and non-GHG impacts (i.e., environmental, economic, and social impacts), and the re-estimation ofthe baseline, positive project spillover, etc., which were estimated during the project design stage. Evaluationorganizes and analyzes the information collected by the monitoring procedures, compares this informationwith information collected in other ways, and presents the resulting analysis of the overall performance ofa project. Project evaluations will be used to determine the official level of GHG emissions reductions thatshould be assigned to the project. The focus of evaluation is on projects that have been implemented fora period of time, not on proposals (i.e., project development and assessment). While it is true that similaractivities may be conducted during the project design stage (e.g., estimating a baseline or positive projectspillover), this type of analysis is estimation and not the type of evaluation that is described in this paper andwhich is based on the collection of data.

Reportingrefers tomeasuredGHG and non-GHG impacts of a project (in some cases, organizations mayreport on theirestimatedimpacts, prior to project implementation, but this is not the focus of this paper).Reporting occurs throughout the MERVC process (e.g., periodic reporting of monitored results and a finalreport once the project has ended).

Verificationrefers to establishing whether the measured GHG reductions actually occurred, similar to an ac-counting audit performed by an objective, accredited party not directly involved with the project. Verificationcan occur without certification.

Certification refers to certifying whether the measured GHG reductions actually occurred. Certification isexpected to be the outcome of a verification process. The value-added function of certification is in thetransfer of liability/responsibility to the certifier.

In the longer term, MERVC guidelines will be a necessary element of any in-ternational C trading system, as proposed in the Kyoto Protocol. A country couldgenerate C credits by implementing projects that result in a net reduction in emis-sions. The validation of such projects will require MERVC guidelines that areacceptable to all parties. These guidelines will yield verified findings, conductedon an ex-post facto basis (i.e., actual as opposed to predicted project performance).

https://www.researchgate.net/publication/226456084_A_Framework_for_Monitoring_and_Evaluating_Carbon_Mitigation_by_Farm_Forestry_Projects_Example_of_a_Demonstration_Project_in_Chiapas_Mexico?el=1_x_8&enrichId=rgreq-c4a664a8-2ba1-4516-8711-3715359a1c09&enrichSource=Y292ZXJQYWdlOzIyNjQwODYxODtBUzo5OTE0MDk3NzU2MTYwMUAxNDAwNjQ4NDU5MjE2


Figure 1. Project activities during the design, development and implementation of GHG offsetactivities.

The Kyoto Protocol contains emissions targets for six major greenhouse gases:carbon dioxide (CO2), methane (CH4), nitrous oxide (N2O), hydrofluorocarbons(HFCs), perfluorocarbons (PFCs), and sulfur hexafluoride (SF6). LBNL’s MERVCguidelines only examine MERVC issues dealing with CO2.

The objective of this paper is to inform the reader about the key MERVC issuesthat need to be addressed if one is involved in the design, development, implement-ation, evaluation, or verification of an energy-efficiency project for climate changemitigation. In this paper, we first describe the MERVC process and the conceptualframework underlying the measurement and evaluation activities. After review-ing the monitoring and evaluation methods, we examine key evaluation issuessuch as the establishment of realistic and credible baselines, free riders, positiveproject spillover, and market transformation. We then briefly review the need foraccounting for environmental and socioeconomic impacts, followed by a reviewof reporting, verification and certification. We conclude by analyzing the costs ofmonitoring and evaluation.


2. The MERVC Process

Climate change mitigation projects to be undertaken within the CDM or underJI will likely involve several tasks (Figure 1). In Figure 1, we differentiate ‘re-gistration’ from ‘certification.’ Certification refers to certifying whether the meas-ured GHG reductions actually occurred. This definition reflects the language inthe Kyoto Protocol regarding the Clean Development Mechanism and ‘certifiedemission reductions.’ In contrast, when a host country approves a project for im-plementation, the project is ‘registered’ (see UNFCCC 1998). For a project to beapproved, each country will rely on project approval criteria that they developed:e.g., (1) the project funding sources must be additional to traditional project de-velopment funding source; (2) the project must be consistent with the host coun-try’s national priorities (including sustainable development); (3) confirmation oflocal stakeholder involvement; (4) confirmation that adequate local capacity existsor will be developed; (5) potential for long-term climate change mitigation; (6)baseline and project scenarios; and (7) the inclusion of a monitoring protocol (Wattet al. 1995).

A country may also use different administrative or legal requirements for re-gistering projects. For example, the project proposal (containing construction andoperation plans, proposed monitoring and evaluation of changes in C stock andenergy use and emissions, and estimated changes in C stock, energy use and emis-sions) might have to be reviewed and assessed by independent reviewers. Afterthis initial review, the project participants would have an opportunity to makeadjustments to the project design and make appropriate adjustments to the ex-pected changes in energy use, C stock, and emissions. The reviewers would thenapprove the project, and the project would be registered. Individuals or organiza-tions voicing concerns about the project would have an opportunity to appeal theapproval of the project, if desired.

3. Conceptual Framework

The analysis of energy use occurs when a project is being designed and duringthe implementation of an energy-efficiency project. In the design stage, the firststep is estimating the baseline (i.e., what would have happened to energy use ifthe project had not been implemented) and the project impacts. Once these havebeen estimated, then the net energy savings are simply the difference between theestimated project impacts and the baseline (P-B, in Figure 2). After a project hasstarted to be implemented, the baseline can be re-estimated and the project impactswill be calculated based on monitoring and evaluation methods. The net savingswill be the difference between the measured project impacts and the re-estimatedbaseline (P∧-B∧, in Figure 2). The example in Figure 2 illustrates a case wheremeasured energy use is lower than estimated as a result of an energy-efficiency


Figure 2.Example of changes in energy use over time for an energy-efficiency project.

project. On the other hand, energy use in the re-estimated baseline is higher thanwhat had been estimated at the project design stage. In this case, the calculated netenergy savings (P∧-B∧) is larger than what was first estimated (P-B).

4. Monitoring and Evaluation of GHG Emissions

As an example of the type of monitoring and evaluation that is needed, we presentin Figure 3 an overview of one approach used in evaluating energy savings andGHG emissions. In this approach, gross energy savings are first measured, usingone of the options provided in the U.S. Department of Energy’s (DOE) Interna-tional Performance Measurement and Verification Protocol (IPMVP); the IPMVPprotocol can be downloaded via the World Wide Web at http://www.ipmvp.org.Next, the project’s baseline is re-estimated, accounting for free riders. Free ridersare project participants who would have installed the same energy-efficiency meas-ures if there had been no project. The net change in energy use is equal to thegross change in energy use minus the re-estimated baseline. Net emissions are thenestimated, using either default emission factors or emissions based on generationdata.


Figure 3. A schematic diagram of an energy-efficiency project, GHG emission baseline andmonitoring system


4.1. ESTABLISHING THE MONITORING DOMAIN

The domain that needs to be monitored (i.e., the monitoring domain) is typicallyviewed as larger than the geographic and temporal boundaries of the project. Inorder to compare GHG reductions across projects, a monitoring domain needs tobe defined. Consideration of the domain needs to address the following issues: (1)the temporal and geographic extent of a project’s direct impacts; and (2) coverageof positive project spillover and market transformation. Project spillover refers tochanges in the behavior of project participants who are not directly related to theproject as well as to changes in the behavior of other individuals not participatingin the project (i.e., nonparticipants). Market transformation is described later.

The first monitoring domain issue concerns the appropriate geographic bound-ary for evaluating and reporting impacts. For example, an energy project mighthave local (project-specific) impacts that are directly related to the project in ques-tion, or the project might have more widespread (e.g., regional) impacts (leadingto project spillover and market transformation). Also, energy projects may impactenergy supply and demand at the point of production, transmission, or end use.The MERVC of such impacts will become more complex and difficult as oneattempts to monitor how emission reductions are linked between energy end usersand energy producers (e.g., tracking the emissions impact of 1,000 kWh saved bya household in a utility’s generation system).

The second issue concerns coverage of positive project spillover, as discussedbelow. It is important to note that not all secondary impacts can be predicted. Infact, many secondary impacts occur unexpectedly and cannot be foreseen. Andwhen secondary impacts are recognized, a commitment needs to be made to ensurethat resources are available to evaluate these impacts.

One could broaden the monitoring domain to include off-site baseline changes(which are normally perceived as occurring outside the monitoring domain). Widen-ing the system boundary, however, will most likely entail greater MERVC costsand could bring in tertiary and even less direct effects that could overwhelm anyattempt at project-specific calculations (Trexler and Kosloff 1998).

In the beginning stages of a project, the secondary impacts of a project maybe modest as the project gets underway, so that the MERVC of such impacts maynot be a priority. These effects may also likely to be insignificant or small forsmall projects. Under these circumstances, it may be justified to disregard theseimpacts and simply focus on energy savings from the project. This would helpreduce MERVC costs. As the projects become larger or are more targeted to markettransformation, these impacts should be evaluated. Of course, if these secondaryimpacts are expected to be large at the beginning of a project, then they need to beevaluated at that time.

https://www.researchgate.net/publication/263489409_The_1997_Kyoto_Protocol_What_Does_It_Mean_for_Project-Based_Climate_Change_Mitigation?el=1_x_8&enrichId=rgreq-c4a664a8-2ba1-4516-8711-3715359a1c09&enrichSource=Y292ZXJQYWdlOzIyNjQwODYxODtBUzo5OTE0MDk3NzU2MTYwMUAxNDAwNjQ4NDU5MjE2


4.2. MONITORING AND EVALUATION METHODS

For energy-efficiency projects, the first step in measuring emission reductions isthe measurement of gross energy savings: comparing the observed energy use ofproject participants with pre-project energy consumption. Several data collectionand analysis methods are available which vary in cost, precision, and uncertainty.Thedata collectionmethods include engineering calculations, surveys, modeling,end-use metering, on-site audits and inspections, and collection of utility bill data.Most monitoring and evaluation activities focus on the collection of measured data;if measured data are not collected, then one may rely on engineering calculationsand stipulated (or default) savings (as described in EPA’s Conservation VerificationProtocols and in DOE’s International Performance Measurement and VerificationProtocol).Data analysismethods include engineering methods, basic statisticalmodels, multivariate statistical models (including multiple regression models andconditional demand models), and integrative methods. If the focus of the monitor-ing and evaluation is an individual building, then some methods will not be utilized(e.g., basic statistical models, multivariate statistical models, and some integrativemethods), since they are more appropriate for a group of buildings.

4.2.1. Engineering methodsEngineering methods are used to develop estimates of energy savings based ontechnical information from manufacturers on equipment in conjunction with as-sumed operating characteristics of the equipment. The two basic approaches todeveloping engineering estimates are engineering algorithms and engineering sim-ulation methods (Violette et al. 1991). Engineering analyses need to be ‘calibrated’with onsite data (e.g., operating hours and occupancy).

Engineering algorithmsare typically straightforward equations showing how en-ergy (or peak) is expected to change due to the installation of an energy efficiencymeasure. The accuracy of the engineering estimate depends upon the accuracy ofthe inputs, and the quality of data that enters an engineering algorithm can varydramatically.

Engineering building simulationsare computer programs that model the perform-ance of energy-using systems in residential and commercial buildings. These mod-els use information on building occupancy patterns, building shell, building ori-entation, and energy-using equipment. The input data requirements for the morecomplex simulation models are extensive and require detailed onsite data collectionas well as building blueprints.

Although engineering approaches are improving and increasing in sophistic-ation, they cannot by themselves produce estimates of net project impacts. Theengineering estimates generally produce estimates of gross impacts and do notcapture behavioral factors such as free riders and project spillover. It is possible


to incorporate free rider and spillover factors from surveys and other evaluationsources in order to calculate net impacts.

4.2.2. Basic statistical models for evaluationStatistical models that compare energy consumption before and after the install-ation of energy efficiency measures have been used as an evaluation method formany years (Violette et al. 1991). The most basic statistical models simply lookat monthly billing data before and after measure installation using weather nor-malized consumption data (this is particularly important where weather-dependentmeasures are involved – e.g., heating and cooling equipment, refrigerators, etc.). Ifthe energy savings are expected to be a reasonably large fraction of the customer’sbill (e.g., 10% or more), then this change should be observable in the project’sbills. Smaller changes (e.g., 4%) might also be observed in billing data, but moresophisticated billing analysis procedures are often required. This method can beused for comparing changes in energy use for project participants and a comparisongroup. Statistical models are most useful where many projects (or one project withmany participants) are being implemented.

4.2.3. Multivariate statistical models for evaluationIn project evaluation, more detailed statistical models may need to be developed tobetter isolate the impacts of an energy-efficiency project from other factors that alsoinfluence energy use. Typically, these more detailed approaches use multivariateregression analysis as a basic tool (Violette et al. 1991). Regression methods aresimply another way of comparing kWh or kW usage across dwelling units or facil-ities and comparison groups, holding other factors constant. Regression methodscan help correct for problems in data collection and sampling. If the samplingprocedure over- or under-represents specific types of projects among either pro-ject participants or the comparison group, the regression equations can capturethese differences through explanatory variables. Two commonly applied regressionmethods are conditional demand analysis and statistically adjusted engineeringmodels (Violette et al. 1991).

4.2.4. End-use meteringEnergy savings can be measured for specific equipment for specific end uses throughend-use metering (Violette et al. 1991). This type of metering is conducted beforeand after a retrofit to characterize the performance of the equipment under a varietyof load conditions. The data are often standardized for variations in operations,weather, etc. End-use metering reduces measurement error (assuming the meteringequipment is reliable) and reduces the number of control variables required inmodels. In addition, the meter can calculate the energy change on an individualpiece of equipment in isolation from the other end-use loads.


4.2.5. Short-term monitoringShort-term monitoring refers to data collection conducted to measure specific phys-ical or energy consumption characteristics either instantaneously or over a shorttime period. This type of monitoring is conducted to support evaluation activitiessuch as engineering studies, building simulation and statistical analyses (Violetteet al. 1991). Examples of the type of monitoring that can take place are spotwatt measurements of efficiency measures, run-time measurements of lights ormotors, temperature measurements, or demand monitoring. Short-term monitor-ing is gaining increasing attention as evaluators realize that for certain energy-efficiency measures with relatively stable and predictable operating characteristics(e.g., commercial lighting and some motor applications), short-term measurementswill produce gains in accuracy nearly equivalent to that of longer-term metering ata fraction of the cost.

Short-term monitoring is a useful tool for estimating energy savings when theefficiency of the equipment is enhanced, but the operating hours remains fixed. Spotmetering of the connected load before and after the activity quantifies this changein efficiency with a high degree of accuracy. For activities where the hours ofoperation are variable, the actual operating (run-time) hours of the activity shouldbe measured before and after the installation using a run-time meter.

4.2.6. Integrative methodsIntegrative methods combine one or more of the above methods to create an evenstronger analytical tool. These approaches are rapidly becoming the state of thepractice in the evaluation field (Raab and Violette 1994). The most common integ-rative approach is to combine engineering and statistical models where the outputsof engineering models are used as inputs to statistical models. These methodsare often called Statistically Adjusted Engineering (SAE) methods or EngineeringCalibration Approaches (ECA). Although they can provide more accurate results,integrative methods typically increase the complexity and expense. To reduce thesecosts while maintaining a high level of accuracy, a related set of procedures hasbeen developed to leverage high cost data with less expensive data. These lever-aging approaches typically utilize a statistical estimation approach termed ratioestimation that allows data sets on different sample sizes to be leveraged to produceestimates of impacts (see Violette and Hanser 1991).

4.2.7. Best methodsThere is no one approach that is ‘best’ in all circumstances (either for all projecttypes, evaluation issues, or all stages of a particular project). The costs of alternat-ive approaches will vary and the selection of evaluation methods should take intoaccount project characteristics and the kind of load and schedule for the load beforethe retrofit. The load can be constant, variable, or variable but predictable, and theschedule can either be known (timed on/off schedule) or unknown/variable. Themonitoring approach can be selected according to the type of load and schedule.


Table I

Advantages and disadvantages of data collection and analysis methods for GHG offset projects

Methods Application Advantages Disadvantages

EngineeringMethods

Individualbuildings andgroups ofbuildings

Relatively quick andinexpensive for simpleengineering methods. Mostuseful as a complement toother methods. Methods areimproving. Useful forbaseline development.

Relatively expensive formore sophisticatedengineering models. Needto be calibrated with onsitedata. By themselves, notgood for evaluation ofspillover.

BasicStatisticalModels

Primarily forgroups ofbuildings

Relatively inexpensive andeasy to explain.

Assumptions need to beconfirmed with survey dataand other measured data.Limited applicability.Cannot evaluate peakimpacts. Large sample sizesneeded.

MultivariateStatisticalModels


Can isolate project impactsbetter than basic statisticalmodels.

Same disadvantages as forbasic statistical models.Relatively more complex,expensive, and harder toexplain than basic statisticalmodels.

End-useMetering


Most accurate method formeasuring energy use. Mostuseful for data collection,not analysis.

Can be very costly. Smallsamples only. Requiresspecialized equipment andexpertise. Possible samplebiases. Difficult togeneralize to other projects.Does not, by itself, calculateenergy savings. Difficult toobtain pre-installationconsumption.

Short-termMonitoring


Useful for measures withrelatively stable andpredictable operatingcharacteristics. Relativelyaccurate method. Mostuseful for data collection,not analysis.

Limited applicability. Usingthis method alone, energysavings cannot becalculated.

IntegrativeMethods


Relatively accurate. Relatively more complex,expensive, and harder toexplain than some of theother models.


In addition to project characteristics, the appropriate approach depends on thetype of information sought, the value of information, the cost of the approach, andthe stage and circumstances of project implementation. The applications of thesemethods are not mutually exclusive; each approach has different advantages anddisadvantages (Table I), and there are few instances where an evaluation method isnot amenable to most energy-efficiency measures. Using more than one method canbe informative. Employing multiple approaches, perhaps even conducting differentanalyses in parallel, and integrating the results, will lead to a robust evaluation.Such an approach builds upon the strengths and overcomes the weaknesses ofindividual approaches. Also, each approach may be best used at different stagesof the project life cycle and for different measures or projects. An evaluation planshould specify the use of various analytical methods throughout the life of theproject and account for the financial constraints, staffing needs, and availability ofdata sources.

IPMVP approachIn an earlier report, we reviewed several protocols and guidelines that were de-veloped for the MERVC of GHG emissions in the energy sectors by governments,nongovernmental organizations, and international agencies (Vine and Sathaye1997). Although not targeted to carbon emissions, we believe that the IPMVP isthe preferred approach for monitoring and evaluating energy-efficiency projectsfor individual buildings and for groups of buildings, since the IPMVP covers manyof the issues discussed in these guidelines as well as offering several measure-ment and verification methods for user flexibility (USDOE 1997). North America’senergy service companies have adopted the IPMVP as the industry standard ap-proach to measurement and verification. States ranging from Texas to New Yorknow require the use of the IPMVP for state-level energy efficiency retrofits. TheU.S. Federal Government, through the Department of Energy’s Federal EnergyManagement Program (FEMP), uses the IPMVP approach for energy retrofits inFederal buildings. Finally, countries ranging from Brazil to the Ukraine have adop-ted the IPMVP, and the Protocol is being translated into Bulgarian, Chinese, Czech,Hungarian, Polish, Portuguese, Russian, Spanish, Ukrainian and other languages.

A key element of the IPMVP is the definition of two measurement and veri-fication (M&V) components: (1) verifying proper installation and the measure’spotential to generate savings; and (2) measuring (or estimating) actual savings. Thefirst component involves the following: (a) the baseline conditions were accuratelydefined and (b) the proper equipment/systems were installed, were performing tospecification, and had the potential to generate the predicted savings. The generalapproach to verifying baseline and post-installation conditions involves inspec-tions, spot measurement tests, or commissioning activities. Commissioning is theprocess of documenting and verifying the performance of energy systems, so thatthe systems operate in conformity with the design intent.


Table II

Overview of IPMVP’s measurement and verification options

Measurement & Verification options1 How savings arecalculated [reference toLBNL’s MERVCmethods]

Initialcost2,3

Annualoperatingcost4

Option A:

• Focuses on physical inspection ofequipment to determine whetherinstallation and operation are tospecification. Performance factors areeither stipulated (based on standards ornameplate data) or measured.

• Key performance factors (e.g., lightingwattage or ‘motor’ efficiency) aremeasured on a snapshot or short-termbasis.

• Operational factors (e.g., Lightingoperating hours or motor runtime) arestipulated based on analysis of historicaldata or spot/short-term measurements.

Engineering calculationsor computer simulationsbased on metered dataand stipulated operationaldata.[Engineering methods(4.2.2)][Short-term monitoring(4.2.6)]

0.5 to 3% 0.1 to 0.5%

Option B:

• Intended for individual energyconservation measures (ECMs) (retrofitisolation) with a variable load profile.

• Both performance and operational factorsare measured on a short-term continuousbasis taken throughout the term of thecontract at the equipment or system level.

Engineering calculationsafter performing astatistical analysis ofmetered data.[Engineering methods(4.2.2)][End-use metering(4.2.5)]

2 to 8% 0.5 to 3%

Option C:

• Intended for whole-building M&V whereenergy systems are interactive (e.g.efficient lighting system reduces coolingloads) rendering measurement ofindividual ECMs inaccurate.

• Performance factors are determined at thewhole-building or facility level withcontinuous measurements.

• Operational factors are derived fromhourly measurements and/or historicalutility meter (electricity or gas) orsub-metered data.

Engineering calculationsbased on a statisticalanalysis ofwhole-building data usingtechniques from simplecomparison tomultivariate (hourly ormonthly) regressionanalysis.[Basic statistical models(4.2.3)][Multivariate statisticalmodels (4.2.4)]

0.5 to 3%(utility billanalysis)2 to 8%(hourlydata)

0.5 to 3%


Table II

Continued

Measurement & Verification options1 How savings arecalculated [reference toLBNL’s MERVCmethods]

Initialcost2,3

Annualoperatingcost4

Option D:

• Typically employed for verification ofsaving in new construction and incomprehensive retrofits involvingmultiple measures at a single facilitywhere pre-retrofit data may not exist.

• In new construction, performance andoperational factors are modeled based ondesign specification of new, existingand/or code complying componentsand/or systems.

• Measurements should be used to confirmsimulation inputs and calibrate themodels.

Calibrated energysimulation/ modeling offacility componentsand/or the whole facility;calibrated with utilitybills and/or end-usemetering data collectedafter project completion.[Engineering methods(4.2.2)][Integrative methods(4.2.7)]

2 to 8% 0.5 to 3%

1 It is assumed that the cost of minimum M&V, in projects not following IPMVP, involves an initialcost of 0.5%, and an annual operating cost of 0.1% to 0.2%, of the project cost. The costs in thistable are uncertain and should be used for general guidance; developers need to estimate costs basedon real projects.2 The initial M&V cost includes installation and commissioning of meters.3 In new construction, this is the % of the difference in cost between baseline equipment and up-graded/more efficient equipment.4 Annual operating cost includes reporting, data logger and meter maintenance cost over the periodof the contract.

The IPMVP was built around a common structure of four M&V options (Op-tions A, B, C, and D) (Table II). These four options were based on the two compon-ents to M&V defined above. The purpose of providing several M&V options is toallow the user flexibility in the cost and method of assessing savings. A particularoption is chosen based on the expectations for risk and risk sharing between thebuyer and seller and onsite and energy-efficiency project specific features. Theoptions differ in their approach to the level and duration of the verification measure-ments. None of the options are necessarily more expensive or more accurate thanthe others. Each has advantages and disadvantages based on site specific factorsand the needs and expectations of the customer. Project evaluators should use oneof these options for reporting on measured energy savings.


4.3. BASELINE USE: RE-ESTIMATING THE BASELINE

For JI (Article 6) and CDM (Article 12) projects implemented under the KyotoProtocol, the emissions reductions from each project activity must be ‘additional toany that would otherwise occur,’ also referred to as ‘additionality criteria’ (Articles6.1b and 12.5c). Determining additionality requires a baseline for the calculation ofenergy saved, i.e., a description of what would have happened to energy use had theproject not been implemented (see Violette et al. 1998). Additionality and baselinesare inextricably linked and are a major source of debate (Trexler and Kosloff 1998).Determining additionality is inherently problematic because it requires resolving acounter-factual question: What would have happened in the absence of the specificproject?

Because investors and hosts of energy-efficiency projects have the same interestin an energy-efficiency project (i.e., they want to get maximum energy savings fromthe project), they are likely to overstate and over-report the amount of energy savedby the project (e.g., by overstating business-as-usual energy use). Cheating may bewidespread if there is no strong monitoring and verification of the projects. Even ifprojects are well monitored, it is still possible that the real amount of energy savedis less than estimated values. Hence, there is a critical need for the establishmentof realistic and credible baselines.

Future changes in energy use may differ from past levels, even in the absence ofthe project, due to growth, technological changes, input and product prices, policyor regulatory shifts, social and population pressure, market barriers, and other exo-genous factors. Consequently, the calculation of the baseline needs to account forlikely changes in relevant regulations and laws, and changes in key variables (e.g.,population growth or decline, and economic growth or decline).

Ideally, when first establishing the baseline, energy use should be measured forat least a full year before the date of the initiation of the project. The baseline shouldbe re-estimated based on monitoring and evaluation data collected during projectimplementation. The re-estimated baseline should describe the existing technologyor practices at the facility or site. It is important to point out that some investorsmay be averse to the re-estimation of a baseline, since their investment was basedon the original baseline. However, we believe that this source of uncertainty can beaccounted or (or negotiated) in contracting.

Some types of projects may not require a full year of monitoring: e.g., in energy-efficiency projects, if the loads and operating conditions are constant over time,one-time spot measurement may be sufficient to estimate equipment performanceand efficiency. And some have argued that ‘performance benchmarks’ could beused instead of project-specific baselines (Section 4.3.2). Finally, in order to becredible, project-specific baselines need to account for free riders.

https://www.researchgate.net/publication/263489409_The_1997_Kyoto_Protocol_What_Does_It_Mean_for_Project-Based_Climate_Change_Mitigation?el=1_x_8&enrichId=rgreq-c4a664a8-2ba1-4516-8711-3715359a1c09&enrichSource=Y292ZXJQYWdlOzIyNjQwODYxODtBUzo5OTE0MDk3NzU2MTYwMUAxNDAwNjQ4NDU5MjE2


4.3.1. Free ridersIn energy-efficiency projects, it is possible that the reductions in energy use areundertaken by participants who would have installed the same measures if therehad been no project. These participants are called ‘free riders.’ The savings as-sociated with free riders are not truly ‘additional’ to what would occur otherwise(Vine 1994). Although free riders may be regarded as an unintended consequenceof an energy-efficiency project, free ridership should still be estimated, if possible,during the estimation of the baseline. While free riders can also cause positiveproject spillover, this impact is typically considered to be insignificant comparedto the impacts from other participants.

Free ridership can be evaluated either explicitly or implicitly (Goldberg andSchlegel 1997; Saxonis 1991). The most common method of developing explicitestimates of free ridership is to ask participants what they would have done in theabsence of the project (also referred to as ‘but for the project’ discussions). Basedon answers to carefully designed survey questions, participants are classified asfree riders (yes or no) or assigned a free ridership score. Project free ridership isthen estimated as the proportion of participants who are classed as free riders. Twoproblems arise in using this approach: (1) very inaccurate levels of free ridershipmay be estimated, due to questionnaire wording; and (2) there is no estimate of thelevel of inaccuracy, for adjusting confidence levels.

Another method of developing explicit estimates of free ridership is to use dis-crete choice models to estimate the effect of the program on customers’ tendency toimplement measures. The discrete choice is the customer’s yes/no decision whetherto implement a measure. The discrete choice model is estimated to determine theeffect of various characteristics, including project participation, on the tendency toimplement the measures.

For energy-efficiency projects, a method for calculating implicit estimates offree ridership is to develop an estimate of savings using billing analysis that maycapture this effect, but does not isolate it from other impacts. Rather than takingsimple differences between participants and a comparison group, however, regres-sion models are used to control for factors that contribute to differences betweenthe two groups (assuming that customers who choose to participate in projectsare different from those who do not participate). The savings determined from theregression represent the savings associated with participation, over and above thechange that would be expected for these customers due to other factors, includingfree ridership.

The U.S. Environmental Protection Agency’s Conservation Verification Proto-cols reward more rigorous methods of verifying free riders by allowing a highershare of the savings to qualify for tradable SO2 allowances. Three options areavailable for verifying free riders: (1) default ‘net-to-gross’ factors for convertingcalculated ‘gross energy savings’ to ‘net energy savings’; (2) project-estimatednet-to-gross factors, based on measurement and evaluation activities (e.g., marketresearch, surveys, and inspections of nonparticipants); or (3) if a developer does

https://www.researchgate.net/publication/222770257_The_Human_Dimension_of_Program_Evaluation?el=1_x_8&enrichId=rgreq-c4a664a8-2ba1-4516-8711-3715359a1c09&enrichSource=Y292ZXJQYWdlOzIyNjQwODYxODtBUzo5OTE0MDk3NzU2MTYwMUAxNDAwNjQ4NDU5MjE2


not do any monitoring nor provide documentation and the default net-to-grossfactors are not used, then the net energy savings of a measure will be 50% ofthe first-year savings (USEPA 1995 and 1996). The ‘net-to-gross’ factor is definedas net savings divided by gross savings. The gross savings are the savings directlyattributed to the project and include the savings from all measures and from allparticipants; net savings are gross savings that are ‘adjusted’ for free riders andpositive project spillover. Multiplying the gross savings by the net-to-gross factoryields net savings.

4.3.2. Performance benchmarksConcerned about an arduous project-by-project review that might impose prohib-itive costs, some researchers have proposed an alternate approach, based on acombination of performance benchmarks and procedural guidelines that are tiedto appropriate measures of output (e.g., Lashof 1998; Michaelowa 1998; Swisher1998; Trexler and Kosloff 1998). In all cases, measurement and verification of theactual performance of the project is required. The performance benchmarks for newprojects could be chosen to represent the high performance end of the spectrum ofcurrent commercial practice (e.g., representing roughly the top 25th percentile ofbest performance). In this case, the benchmark serves as a goal to be achieved. Incontrast, others might want to use benchmarks as a reference or default baseline:an extension of existing technology, and not representing the best technology orprocess.

A panel of experts could determine a baseline for a number of project types,which could serve as a benchmark for the UNFCCC. This project categoriza-tion could be expanded to a categorization by regions or countries, resulting ina region-by-project matrix. Project developers could check the relevant elementin the matrix to determine the baseline of their project. Most of the costs in thisapproach relate to the establishment of the matrix and its periodical update. Be-fore moving forward with this approach, analysis is needed to consider the costsin developing the matrix and its update, the potential for projects to qualify, andthe potential for free riders. The U.S. Environmental Protection Agency (EPA) isassessing the feasibility and desirability of implementing a benchmark approachfor evaluating additionality (Hagler Bailly 1998).

4.3.3. Comparison groupsFor many energy-efficiency projects, comparison groups can be used for evaluatingthe impacts of energy-efficiency projects. Acting as a baseline, comparison groupscan capture time trends that are unrelated to project participation. For example, ifthe comparison group shows an average reduction in energy use of 5% between thepre- and post-periods, and the participants’ bills show a reduction of 15%, then itmay be reasonable to assume that the estimated project impacts will be 15% minusthe 5% general trend for an estimated 10% reduction in use being attributed to theproject.


4.4. PROJECT CASE: MONITORING AND EVALUATION

4.4.1. Positive project spilloverFor many programs, the number of eligible nonparticipants is far greater thanthe number of participants. For example, when measuring energy savings, it ispossible that the actual reductions in energy use are greater than measured be-cause of changes in participant behavior not directly related to the project, aswell as to changes in the behavior of other individuals not participating in theproject (i.e., nonparticipants). These secondary impacts stemming from an energy-efficiency project are commonly referred to as ‘positive project spillover.’ Positiveproject spillover may be regarded as an unintended consequence of an energy-efficiency project; however, as noted below, increasing positive project spillovermay also be perceived as a strategic mechanism for reducing GHG emissions and,therefore, project crediting should include positive project spillover and markettransformation effects.

Spillover effects can occur through a variety of channels including: (1) projectparticipants that undertake additional, but unaided, actions based on positive exper-ience with the project; (2) manufacturers changing the efficiency of their products,or retailers and wholesalers changing the composition of their inventories to reflectthe demand for more efficient goods created through the project; (3) governmentsadopting new building codes or appliance standards because of improvements toappliances resulting from one or more energy efficiency projects; or, (4) tech-nology transfer efforts by project participants which help reduce market barriersthroughout a region or country.

The methods for estimating positive project spillover are similar to those usedfor free ridership (Goldberg and Schlegel 1997; Weisbrod et al. 1994). Explicitestimates can be obtained by asking participants and nonparticipants survey ques-tions, and discrete choice models can be used (e.g., the effect on implementationof program awareness, rather than program participation, is estimated). Participantand nonparticipant spillover effects can be included in savings estimates in billinganalyses, similar to how gross savings are calculated.

4.5. MARKET TRANSFORMATION

Project spillover is related to the more general concept of ‘market transforma-tion,’ defined as: "the reduction in market barriers due to a market intervention,as evidenced by a set of market effects, that lasts after the intervention has beenwithdrawn, ‘reduced or changed’ (Eto et al. 1996). In contrast to project spillover,increasing market transformation is expected to be a strategic mechanism (i.e., anintended consequence) for reducing GHG emissions for the following reasons:• To increase the effectiveness of energy-efficiency projects: e.g., by examining

market structures more closely, looking for ways to intervene in markets morebroadly, and investigating alternative points of intervention.

https://www.researchgate.net/publication/241972927_A_Scoping_Study_on_Energy-Efficiency_Market_Transformation_by_California_Utility_DSM_Programs?el=1_x_8&enrichId=rgreq-c4a664a8-2ba1-4516-8711-3715359a1c09&enrichSource=Y292ZXJQYWdlOzIyNjQwODYxODtBUzo5OTE0MDk3NzU2MTYwMUAxNDAwNjQ4NDU5MjE2


• To reduce reliance on incentive mechanisms: e.g., by strategic interventionsin the market place with other market actors.

• To take advantage of regional and national efforts and markets.• To increase focus on key market barriers other than cost.• To create permanent changes in the market.

Market transformation has emerged as a central policy objective for future publiclyfunded energy-efficiency projects in the United States, but the evaluation of suchprojects is still in its infancy. Furthermore, regulatory authorities have little exper-ience in accepting savings from market transformation. Nevertheless, because ofits importance, we encourage project developers to consider savings from markettransformation, particularly since other countries are starting to implement markettransformation programs (Vine and Sathaye 1999).

Most evaluations of market transformation projects focus on market effects(e.g., Eto et al. 1996; Schlegel et al. 1997). Market effects are the effects of energy-efficiency projects on market structure or the behavior of market actors that leadto increases in the adoption of energy-efficiency products, services, or practices.In order to claim that a market has been transformed, project evaluators need todemonstrate the following (Schlegel et al. 1997):• There has been a change in the market that resulted in increases in the adoption

and penetration of energy-efficiency technologies or practices.• That this change was due at least partially to a project (or program or initiat-

ive), based both on data and a logical explanation of the program’s strategicintervention and influence.

• That this change is lasting, or at least that it will last after the project is scaledback or discontinued.

The first two conditions are needed to demonstrate market effects, while all threeare needed to demonstrate market transformation. The third condition is relatedto persistence: if the changes are not lasting (i.e., they do not persist), then markettransformation has not occurred. Because fundamental changes in the structure andfunctioning of markets may occur only slowly, evaluators should focus their effortson the first two conditions, rather than waiting to prove that the effects will last. Toimplement an evaluation system focused on market effects, one needs to carefullydescribe the scope of the market, the indicators of success, the intended indices ofmarket effects and reductions in market barriers, and the methods used to evaluatemarket effects and reductions in market barriers (Schlegel et al. 1997).

Evaluation activities will include one or more of the following: (1) measuringthe market baseline; (2) tracking attitudes and values; (3) tracking sales; (4) model-ing of market processes; and (5) assessing the persistence of market changes (Prahland Schlegel 1993). As one can see, these evaluation activities will rely on a largeand diverse group of data collection and analysis methods, such as: (1) surveysof customers, manufacturers, contractors, vendors, retailers, government organiza-tions, energy providers, etc.; (2) analytical and econometric studies of measure costdata, stocking patterns, sales data, and billing data; and (3) process evaluations.

https://www.researchgate.net/publication/241972927_A_Scoping_Study_on_Energy-Efficiency_Market_Transformation_by_California_Utility_DSM_Programs?el=1_x_8&enrichId=rgreq-c4a664a8-2ba1-4516-8711-3715359a1c09&enrichSource=Y292ZXJQYWdlOzIyNjQwODYxODtBUzo5OTE0MDk3NzU2MTYwMUAxNDAwNjQ4NDU5MjE2


4.6. CALCULATION OF NET GHG EMISSIONS

Once the net energy savings have been calculated (i.e., measured energy use minusre-estimated baseline energy use), net GHG emissions reductions can be calculatedin one of two ways: (1) if emissions reductions are based on fuel-use or electricity-use data, then default emissions factors can be used, based on utility or nonutilityestimates; or (2) emissions factors can be based on generation data specific to thesituation of the project (e.g., linking a particular project on an hourly or daily basisto the marginal unit it is affecting). In both methods, emissions factors translateconsumption of energy into GHG emission levels (e.g., Mg of a particular GHGper kWh saved). In contrast to default emission factors (method #1), the advantageof using the calculated factors (method #2) is that they can be specifically tailoredto match the energy-efficiency characteristics of the activities being implementedby time of day or season of the year. For example, if an energy-efficiency projectaffects energy demand at night, then baseload plants and emissions will probablybe affected. Since different fuels are typically used for baseload and peak capacityplants, then emission reductions will also differ.

The calculations, however, become more complex (but more realistic) if onedecides to use the emission rate of the marginal generating plant (multiplied by theenergy saved) for each hour of the year, rather than the average emission rate forthe entire system (i.e., total emissions divided by total sales) (Swisher 1997). Forthe more detailed analysis, one must analyze the utility’s existing expansion plan todetermine the generating resources that would be replaced by saved electricity, andthe emissions from these electricity-supply resources. Thus, one would establish abaseline (current power expansion plan, power dispatch, peak load/base load, etc.),select a monitoring domain, conduct monitoring option, measure direct emissionreductions (e.g., reductions occurring at the neighboring power plant to lower de-mand), measure indirect emissions (e.g., modification in the power system due tolower output at the neighboring plant), and calculate net carbon reductions.

One would have to determine if the planned energy-efficiency measures wouldreduce peak demand sufficiently and with enough reliability to defer or obviateplanned capacity expansion. If so, the deferred or replaced source would be themarginal expansion resource to be used as a baseline. This type of analysis mayresult in more accurate estimates of GHG reductions, but this method will be morecostly and require expertise in utility system modeling. In addition, this type ofanalysis is becoming more difficult in those regions where the utility industryis being restructured: e.g., the supply of energy may come from multiple energysuppliers, either within or outside the utility service area.

The decision on which methodology to use will depend on project size (e.g.,kWh, kW, carbon credits requested, project expenditures) or relative project size(e.g., MW/utility service MW). It is up to the evaluator to decide on the best methodfor the project. Certain thresholds may need to be developed. If a project is of a


certain relative magnitude (e.g., a project is 50 MW and the utility’s service area is400 MW), the evaluator should probably select the second method above.

5. Environmental and Socioeconomic Impacts

The Kyoto Protocol exhorts developed countries, in fulfilling their obligations, tominimize negative social, environmental and economic impacts, particularly ondeveloping countries (Articles 2.3 and 3.14). Furthermore, one of the primary goalsof the CDM is sustainable development. At this time, it is unclear on what indic-ators of sustainable development need to be addressed in the evaluation of energy-efficiency projects. Once there is an understanding of this, then MERVC guidelinesfor those indicators will need to be designed. At a minimum, energy-efficiencyprojects should meet current country guidelines for non-CDM projects.

The persistence of GHG reductions and the sustainability of energy-efficiencyprojects depend on individuals and local organizations that help support a projectduring its lifetime. Both direct and indirect project benefits will influence the mo-tivation and commitment of project participants. Hence, focusing only on GHGimpacts would present a misleading picture of what is needed in making a projectsuccessful or making its GHG benefits sustainable. In addition, a diverse groupof stakeholders (e.g., government officials, project managers, non-profit organiza-tions, community groups, project participants, and international policymakers) areinterested in, or involved in, energy-efficiency projects and are concerned abouttheir multiple impacts.

5.1. ENVIRONMENTAL IMPACTS

Energy-efficiency projects have widespread and diverse environmental impactsthat go beyond GHG impacts. The environmental benefits associated with energy-efficiency projects can be just as important as the global warming benefits. Directand indirect project impacts need to be examined, as well as ‘avoided negativeenvironmental impacts’ (e.g., the deferral of the construction of a new power plant).Both gross and net impacts need to be evaluated.

At a minimum, evaluators need to evaluate the environmental impacts asso-ciated with the project. Evaluators need to collect some minimal information onpotential impacts via surveys or interviews with key stakeholders. The evaluatorshould also check to see: (1) whether any existing laws require these impactsto be examined, (2) if any proposed mitigation efforts were implemented, and(3) whether expected positive benefits ever materialized. Evaluators may want toconduct some short-term monitoring to provide conservative estimates of envir-onmental impacts. The extent and quality of available data, key data gaps, anduncertainties associated with estimates should be identified and estimated.


5.2. SOCIOECONOMIC IMPACTS

In examining socioeconomic impacts, evaluators need to ask the following ques-tions: who the key stakeholders are, what project impacts are likely and upon whatgroups, what key social issues are likely to affect project performance, what therelevant social boundaries and project delivery mechanisms are, and what socialconflicts exist and how they can be resolved. To address these questions, evalu-ators could conduct informal sessions with representatives of affected groups andrelevant non-governmental organizations.

After a project has been implemented, MERVC activities should assess whetherthe project led to any social and economic impacts and whether any mitigation wasdone. Direct and indirect project impacts need to be examined, as well as ‘avoidednegative socioeconomic impacts’ (e.g., the preservation of an archaeological siteas a result of the deferral of the construction of a new power plant). Evaluat-ors should collect some minimal information on potential impacts via surveys orinterviews with key stakeholders. The evaluator should also check to see if anyproposed mitigation efforts were implemented and whether expected positive be-nefits ever materialized. The extent and quality of available data, key data gaps, anduncertainties associated with estimates may need to be identified and estimated.

6. Reporting

Reporting refers tomeasuredGHG and non-GHG impacts of a project (in somecases, organizations may report on theirestimatedimpacts, prior to project imple-mentation, but this is not the focus of these guidelines). Reporting occurs through-out the MERVC process (e.g., periodic reporting of monitored results and a finalreport once the project has ended). LBNL has developed a Monitoring and Evalu-ation Reporting Form (MERF) that evaluators may follow when reporting energyuse and carbon emissions (Vine and Sathaye 1999).

6.1. MULTIPLE REPORTING

Several types of reporting might occur in climate change mitigation projects: (1)impacts of a particular project could be reported at the project level and at theprogram level (where a program consists of two or more projects); (2) impacts of aparticular project could be reported at the project level and at the entity level (e.g.,a utility company reports on the impacts of all of its projects); and (3) impacts of aparticular project could be reported by two or more organizations as part of a jointventure (partnership) or two or more countries. To mitigate the problem of multiplereporting, project-level reporters should indicate whether other entities might bereporting on the same activity and, if so, who. If there exists a clearinghouse withan inventory of stakeholders and projects, multiple reporting might not constitutea problem. For example, in their comments on an international emissions trading


regime, Canada (on behalf of Australia, Iceland, Japan, New Zealand, Norway,Russian Federation, Ukraine and the U.S.) proposed a national recording systemto record ownership and transfers of assigned amount units (i.e., C offsets) at thenational level (UNFCCC 1998). A synthesis report could confirm, at an aggreg-ate level, that bookkeeping was correct, reducing the possibility of discrepanciesamong Parties’ reports on emissions trading activity.

7. Verification and Certification

If C credits become an internationally traded commodity, then verifying the amountof C reduced or fixed by projects will become a critical component of any tradingsystem. Investors and host countries may have an incentive to overstate the GHGemission reductions from a given project, because it will increase their earningswhen excessive credits are granted; as an example, these parties may overstatebaseline emissions or understate the project’s emissions. To resolve this problem,there is a need for external (third party) verification.

As part of the verification exercise, an overall assessment of the quality andcompleteness of each of the GHG impact estimates needs to be made by requestinginformation in a Verification Reporting Form (VRF), similar to the MERF. Forenergy-efficiency measures, verifying baseline and post-project conditions mayinvolve inspections, spot measurement tests, or assessments, as well as requestingdocumentation on key aspects of the project (similar to what is done as part of theIPMVP). In addition, the following general questions need to be asked: (1) havethe monitoring and evaluation methods been well documented and reproducible?(2) have the results been checked against other methods? (3) have results (e.g.,monitored data and emissions) been compared for reasonableness with outside orindependently published estimates? (4) have the sources of emission factors beenwell documented? and (5) have the sources of emission factors been compared withother sources? At this time, certification is expected to simply be the outcome of averification process: i.e., no other monitoring and evaluation activities are expectedto be conducted.

8. Costs

Monitoring and evaluation costs will depend on what information is needed, whatinformation and resources are already available, the size of the project area, themonitoring methods to be used, and frequency of monitoring. Furthermore, somemethods require high initial costs: e.g., in metering, start-up costs in terms of equip-ment and personnel training may make the installation of meters very expensive,while making continuous metering exceedingly cost effective. Based on the exper-ience of U.S. utilities and energy service companies, monitoring and evaluationactivities can easily account for 5–10% of an energy-efficiency project’s budget.


Due to the availability of funding, we realize that some project developers andevaluators will not be able to conduct the most data intensive methods proposed inthis paper; however, we expect each project to undergo some evaluation and veri-fication in order to receive carbon credits (especially, certified emission reductionunits). Moreover, we believe that monitored projects will save more carbon andoffset the cost of the monitoring because: (1) installations following a monitoringand evaluation protocol should come in near or even above the projected level ofenergy savings; and (2) installations with some measurement of energy savingsshould tend to have higher levels of energy savings initially and experience energysavings that remain high during the lifetime of the measure. In the end, the costof monitoring and evaluation will be partially determined by its value in reducingthe uncertainty of carbon credits: e.g., will one be able to receive carbon creditswith a value greater than 10% of project costs that are spent on monitoring andevaluation?

Because of concerns about high costs, MERVC activities cannot be too burden-some: in general, the higher the costs, the less likely organizations and countrieswill try to develop and implement energy-efficiency projects. However, in somecases, due to the enormous cost differential between the carbon reduction optionsof UNFCCC Parties, fairly high costs can be accommodated before these costsbecome prohibitive. Nevertheless, MERVC costs should be as low as possible. Insum, actual (as well as perceived) MERVC costs may discourage some transactionsfrom occurring. Tradeoffs are inevitable, and a balance needs to be made betweenproject implementation and the level of detail (and costs) of MERVC reportingguidelines.

Project estimates of impacts could be adjusted, based on the amount of uncer-tainty associated with the estimates, without conducting project-specific analyses.Projects with less accurate or less precisely quantified benefit estimates wouldhave their estimates adjusted and therefore have their benefits rendered policy-equivalent to credits from projects that can be more accurately quantified. TheU.S. Environmental Protection Agency’s Conservation Verification Protocol re-ward more rigorous methods of verifying energy savings by allowing a higher shareof the savings to qualify for tradable SO2 allowances. Three options are availablefor verifying subsequent-year energy savings: monitoring, inspection and a defaultoption (USEPA 1995 1996). In themonitoring option, a utility can obtain creditfor a greater fraction of the savings and for a longer period: biennial verificationin subsequent years 1 and 3 (including inspection) is required, and savings for theremainder of physical lifetimes are the average of the last two measurements. Themonitoring option requires a 75% confidence in subsequent-year savings (like inthe first year). In contrast, thedefault optiongreatly restricts the allowable savings:50% of first-year savings, and limited to one-half of the measure’s lifetime. For theinspection option(confirming that the measures are both present and operating): autility can obtain credit for 75% of first-year savings for units present and operatingfor half of physical lifetime (with biennial inspections), or 90% of first-year savings


for physical lifetimes of measures that do not require active operation or mainten-ance (e.g., building shell insulation, pipe insulation and window improvements).Thus, utilities could use a simpler evaluation method at a lower cost and receivefewer credits, or they could use a more sophisticated method and receive morecredits.

9. Summary

Monitoring and evaluation of energy-efficiency projects is needed to determinemore accurately their impact on GHG emissions and other attributes, and to ensurethat the global climate is protected and that country obligations are met. Articles 6and 12 of the Kyoto Protocol require MERVC activities. The challenges to suc-cessful monitoring and evaluation will not be insignificant: e.g., the evaluationof positive project spillover, market transformation, and free riders. LBNL hasdeveloped MERVC guidelines that address these issues by describing methods,procedures, and forms which can be used in preparing monitoring and evaluationplans and in reporting the results of monitoring, evaluation, and verification (Vineand Sathaye 1999). The next phase of LBNL’s work will be to develop a proceduralhandbook providing information on how one can complete monitoring, evaluationand verification forms. We then plan to test the usefulness of these handbooks inthe real world.

Acknowledgements

We would like to thank Maurice N. LeFranc, Jr. of the U.S. Environmental Pro-tection Agency, Climate Policy and Program Division, Office of Economics andEnvironment, Office of Policy, Planning and Evaluation for his assistance. Finally,we appreciate the helpful suggestions provided by the four anonymous review-ers of this journal article. This work was supported by the U.S. EnvironmentalProtection Agency through the U.S. Department of Energy under Contract No.DE-AC03-76SF00098.

References

De Jong, B., Tipper, R. and Taylor, J.: 1997, ‘A framework for monitoring and evaluating carbonmitigation by farm forestry projects: example of a demonstration project in Chiapas, Mexico’,Mitigation and Adaptation Strategies for Global Change2(2-3), 231–246.

Eto, J., Prahl, R. and Schlegel, J.: 1996,A Scoping Study on Energy-Efficiency Market Transform-ation by California Utility DSM Programs.LBNL-39058. Berkeley, CA, Lawrence BerkeleyNational Laboratory.


Goldberg, M. and Schlegel, J.: 1997, ‘Technical and statistical evaluation issues’, in J. Schlegel, M.Goldberg, J. Raab, R. Prahl, M. Keneipp and D. Violette (eds.),Evaluating Energy-EfficiencyPrograms in a Restructured Industry Environment: A Handbook for PUC Staff. Washington,D.C., National Association of Regulatory Utility Commissioners, chapter 7.

Hagler Bailly: 1998,Evaluation of Using Benchmarks to Satisfy the Additionality Criterion for JointImplementation Projects,prepared for the U.S. Environmental Protection Agency. Boulder, CO,Hagler Bailly.

Lashof, D.: 1998,Additionality Under the Clean Development Mechanism,unpub. New York,Natural Resources Defense Council.

Michaelowa, A.: 1998, ‘Joint implementation – the baseline issue: economic and political aspects’,Global Environmental Change8(1), 81–92.

Prahl, R. and Schlegel, J.: 1993, ‘Evaluating market transformation’, inThe Proceedings of the 1993International Energy Program Evaluation Conference, Chicago, IL, National Energy ProgramEvaluation Conference, pp. 469–477.

Raab, J. and Violette, D.: 1994,Regulating DSM Program Evaluation: Policy and AdministrativeIssues for Public Utility Commissioners. Washington, DC., National Association of RegulatoryUtility Commissioners.

Saxonis, W.: 1991, ‘Free riders and other factors that affect net program impacts’, in E. Hirst andJ. Reed (eds.),Handbook of Evaluation of Utility DSM Programs,Oak Ridge, TN, Oak RidgeNational Laboratory.

Schlegel, J., Prahl, R. and Raab, J.: 1997, ‘Nest steps for evaluation of market transformation ini-tiatives: an update to the NARUC guidebook’, in J. Schlegel, M. Goldberg, J. Raab, R. Prahl,M. Keneipp and D. Violette (eds.),Evaluating Energy-Efficiency Programs in a RestructuredIndustry Environment: A Handbook for PUC Staff. Washington, D.C., National Association ofRegulatory Utility Commissioners, chapter 4.

Swisher, J.: 1997, ‘Joint implementation under the U.N. framework convention on climate change:technical and institutional challenges’,Mitigation and Adaptation Strategies for Global Change2(1), 57–80.

Swisher, J.: 1998,Project Baselines and Additionality in the Clean Development Mechanism,presented at The Aspen Global Forum, Aspen, CO.

Trexler, M. and Kosloff, L.: 1998, ‘The 1997 Kyoto Protocol: what does it mean for project-basedclimate change mitigation?’Mitigation and Adaptation Strategies for Global Change3, 1–58.

UNEP/WMO Information Unit on Climate Change: 1992,United Nations Framework Conventionon Climate Change. Geneva, Switzerland, UNEP/WMO Information Unit on Climate Change.

UNFCCC: 1997,Kyoto Protocol to the United Nations Framework Convention of Climate Change,FCCC/CP/1997/L.7/Add.1, December 10, 1997. At UNFCCC Web site (under CC:INFOProducts): http://www.unfccc.de/ccinfo.

UNFCCC: 1998,Review of the Implementation of Commitments and of Other Provisions of theConvention. Activities Implemented Jointly: Review of Progress Under the Pilot Phase (De-cision 5/CP.1),FCCC/CP/1998/INF.3. At UNFCCC Web site (under CC:INFO Products):http://www.unfccc.de/ccinfo.

U.S. Department of Energy (USDOE): 1997,International Performance Measurement and Verifica-tion Protocol. Washington, D.C., U.S. Department of Energy.

U.S. Environmental Protection Agency (USEPA): 1995,Conservation and Verification Protocols,Version 2.0.EPA 430/B-95-012. Washington, D.C., U.S. Environmental Protection Agency.

U.S. Environmental Protection Agency (USEPA): 1996,The User’s Guide to the Conservation andVerification Protocols, Version 2.0.EPA 430/B-96-002. Washington, D.C., U.S. EnvironmentalProtection Agency.

Vine, E. 1994, ‘The human dimension of program evaluation’,Energy – The International Journal19(2), 165–178.


Vine, E. and Sathaye, J.: 1997,The Monitoring, Evaluation, Reporting, and Verification of ClimateChange Mitigation Projects: Discussion of Issues and Methodologies and Review of ExistingProtocols and Guidelines. LBNL-40316. Berkeley, CA, Lawrence Berkeley National Laboratory.Available at the World Wide Web: http://eetd.lbl.gov/ea/ccm/ccPubs.html.

Vine, E. and Sathaye, J.: 1999,Guidelines for the Monitoring, Evaluation, Reporting, Verification,and Certification of Energy-Efficiency Projects for Climate Change Mitigation. LBNL-41543.Berkeley, CA, Lawrence Berkeley National Laboratory. Available at the World Wide Web:http://eetd.lbl.gov/ea/ccm/ccPubs.html.

Violette, D. and Hanser, P.: 1991, ‘Utilizing information from multiple sources in DSM impactevaluation’, inThe Proceedings of the 5th National Demand-Side Conference,July 1991. EPRICU-7394. Palo Alto, CA, Electric Power Research Institute.

Violette, D., Ozog, M., Keneipp, M. and Stern, F.: 1991,Impact Evaluation of Demand-Side Man-agement Programs, Volume I and II: A Guide to Current Practice. EPRI CU-7179. Palo Alto,CA, Electric Power Research Institute.

Violette, D., Ragland, S., and Stern, F.: 1998,Evaluating Greenhouse Gas Mitigation through DSMProjects: Lessons Learned from DSM Evaluation in the United States,Boulder, CO, HaglerBailly Consulting, Inc.

Weisbrod, G., Train, K., Hub, A. and Benenson, P.: 1994,DSM Program Spillover Effects.Cambridge, MA, Cambridge Systematics, Inc.

The monitoring, Evaluation, reporting, verification, and certification of energy-efficiency projects

Documents

Transcript of The monitoring, Evaluation, reporting, verification, and certification of energy-efficiency projects