Introduction

Evaluation involves a systematic assessment of energy efficiency program performance. Evaluations fall into two main categories:

1. Impact evaluations measure quantitative results (usually energy savings) and are used to assess program performance against goals.
2. Process evaluations provide information on how the program is working in the market and are used to inform adjustments in program design and implementation strategy.

There are several reasons why evaluating energy efficiency program performance is important. Evaluation helps to ensure accountability and transparency in the use of public funds. It provides independent validation of the amount of energy savings that result from program activities, which is particularly important when energy efficiency resource acquisition is pursued as an alternative to supply-side investment. As key assumptions used in resource planning efforts are updated based on evaluation findings, the reliability of planning estimates increases. In addition, energy efficiency program design is ideally an iterative process (see "Energy Efficiency Planning Cycle" flow chart in Setting Goals and Budgets chapter). Evaluation helps to determine whether market interventions are working as expected, informing adjustments to program strategy and allocation of program resources.

Program evaluation provides numbers and compelling narratives to tell to utility governing bodies, the city council and other customers. When good tracking, reporting and evaluation models are used, the utility has all of the data needed to communicate effectively about the program portfolio.

In some cases, evaluation activities are governed by specific regulatory requirements. In the future, the importance of rigorous evaluation protocols will likely grow given the potential for new federal regulation governing carbon emissions. Such regulations will likely stipulate the evaluation methodologies that are acceptable for measuring the carbon reduction impacts that result from energy efficiency program activities.

Evaluation best practices

Whether involving a quantitative assessment of program impacts or qualitative assessment of how the program is working in the market, evaluation efforts should align with overall goals for the energy efficiency portfolio. If a utility is responsible for meeting energy savings targets, impact evaluation is necessary to verify savings results. If a utility has qualitative goals (e.g., increase customer satisfaction), evaluation can be structured to assess whether programs have achieved those objectives.

To ensure that evaluation funds are well spent, it is important to create feedback mechanisms so that program strategies can be adjusted in response to evaluation findings. The National Energy Efficiency Best Practices Study summarizes approaches used by leading programs to ensure that evaluations lead to improvement in program designs and delivery strategies:

• Upper management support for evaluation
• Organizational culture that emphasizes continuous improvement
• Internal communications strategy that ensures staff have a clear understanding of the value of evaluation and how results will be used
• Direct dialog between evaluators and program staff to discuss evaluation findings and next steps for implementing evaluation recommendations
• Internal processes that facilitate timely adoption of evaluation results (i.e., as part of annual program review/planning process)

For accountability and transparency reasons, evaluations are conducted by independent third parties. Utilities typically outsource evaluation functions to consulting firms experienced in conducting energy efficiency program evaluations. Such firms are familiar with established industry protocols such as the California Energy Efficiency Evaluation Protocols which are used as a reference by evaluation professionals across the country. Use of standard protocols enhances the credibility of evaluation results.

Planning for evaluation activities ideally should occur in conjunction with energy efficiency program design, or shortly thereafter. Understanding how a program will be evaluated helps to ensure that the necessary data are collected from the outset, saving time and resources during the evaluation process. Evaluators can also provide useful input during program planning stages, given their professional experience in evaluating energy efficiency programs in other markets. However, it is also important to retain arms-length distance between evaluators and program staff to ensure evaluator independence and the credibility of evaluation findings.

According to the National Action Plan for Energy Efficiency, program administrators typically allocate between two and five percent of the energy efficiency portfolio budget to evaluation. Evaluation resources should target the areas of largest savings and greatest uncertainty. It may also make sense to devote greater resources to evaluation in the early years of program implementation, so that timely adjustments can be made if market interventions are not working as expected.

Evaluation approaches

A variety of data collection and analysis approaches are used in evaluating energy efficiency programs, and costs range widely depending on the complexity and level of effort required. The graphic below presents the range of typical evaluation approaches, from low-cost to high-cost.

Adapted from: K. Johnson, Johnson Consulting Group (June 2009). Presentation during the proceedings of the ECEEE 2009 Summer Study: Best Practices for Developing Cost Effective Evaluation, Measurement, and Verification Plans: Lessons Learned from 12 Northern California Municipal Utilities .

Determining the appropriate evaluation approach for any given program depends on a number of factors:

• The level of precision that is necessary to meet portfolio management objectives and/or regulatory requirements
• The magnitude of program savings as a percentage of the overall portfolio
• The complexity and scale of individual energy-saving projects within the program
• The amount of funding available for evaluation efforts

Even when evaluation funding resources are limited, low-cost approaches can provide useful information to program managers. For impact evaluations, a low-cost strategy could consist of the following approaches:

• Installation verification through review of invoices or other purchase documentation submitted with the incentive application
• Telephone surveys of a sample of participants/non-participants
• Comparison of deemed savings values with values used in other jurisdictions
• Review of attribution factors (net-to-gross ratios) from studies conducted in other jurisdictions

The Measurement & Verification section provides more detailed information on assessing the energy savings impacts from individual customer projects.

For process evaluations, a low-cost strategy could consist of the following approaches:

• Review tracking database to identify areas where reporting functions can be streamlined (e.g., creation of standardized templates/reports)
• Review marketing materials with a small group of customers to determine the effectiveness of the message
• Conduct interviews with program and customer service staff to assess what is working well in addition to areas for improvement
• Conduct focus groups or interviews with key outside partners (large customers, trade allies) to assess what is working well and to identify areas that need improvement
• Review program design to ensure it is well-founded and logical, and to identify any risk factors that require further investigation

In all cases, utilities should contract with an experienced and independent evaluator to implement the low-cost evaluation strategies listed above.