STATE OF CALIFORNIA - California Measurement Advisory Council

292 Pages·2006·1.05 MB·English

STATE OF CALIFORNIA

PUBLIC UTILITIES COMMISSION

505 Van Ness Avenue
San Francisco, California 94102

California Energy Efficiency Evaluation

Protocols: Technical, Methodological,

and Reporting Requirements for

Evaluation Professionals

APRIL 2006

Prepared for the California Public Utilities Commission

by The TecMarket Works Team
Under Contract with and Directed by the CPUC’s
Energy Division, and with guidance from Joint Staff

California Energy Efficiency Evaluation Protocols:
Technical, Methodological and Reporting
Requirements for Evaluation Professionals
{a.k.a. Evaluators’ Protocols}

Prepared under direction of the Energy Division, with the guidance by Joint Staff, for the
California Public Utilities Commission

APRIL 2006

Submitted by

Nick Hall, Johna Roth, Carmen Best
TecMarket Works
TecMarket Business Center
165 West Netherwood Road, Second Floor, Suite A
Oregon, WI 53575
[email protected]
608 835 8855

And sub-contractors

Sharyn Barata
Opinion Dynamics, Irvine, California

Pete Jacobs
BuildingMetrics, Inc., Boulder, Colorado

Ken Keating, Ph.D.
Energy Program Consultant, Portland, Oregon

Steve Kromer
RCx Services, Oakland, California

Lori Megdal, Ph.D.
Megdal & Associates, Acton, Massachusetts

Jane Peters, Ph.D.
Research Into Action, Portland, Oregon

Richard Ridge, Ph.D.
Ridge & Associates, Alameda, California

Francis Trottier
Francis Trottier Consulting, Penn Valley, California

Ed Vine, Ph.D.
Energy Program Consultant, Berkeley, California
Evaluators’ Protocols Acknowledgements
Evaluators’ Protocols Acknowledgements
Acknowledgements
The authors wish to acknowledge and express our appreciation to the many individuals who
contributed to the development of the California Evaluation Protocols. Without the support and
assistance of these individuals this effort would not have been possible.

The Joint Staff (California Public Utilities Commission and the California Energy Commission)
provided considerable Protocol development guidance and conducted multiple rounds of reviews
of all sections of the Protocols. These individuals and their affiliations are the following:
• Ariana Merlino, Project Manager, Energy Division, California Public Utilities
Commission
• Mike Messenger, California Energy Commission

Appreciation is also extended to the Administrative Law Judge, Meg Gottstein, who ordered the
development of the Protocols and who provided instructive guidance and policy direction along
the way.

In addition to the oversight and guidance provided by the above individuals, others within the
Energy Division of the California Public Utilities Commission and the California Energy
Commission provided valuable contributions and support. For these efforts we thank the
following individuals:
• Nancy Jenkins, California Energy Commission
• Tim Drew, Energy Division, California Public Utilities Commission
• Zenaida Tapawan-Conway, Energy Division, California Public Utilities Commission
• Peter Lai, Energy Division, California Public Utilities Commission
• Nora Gatchalian, Energy Division, California Public Utilities Commission
• Jeorge Tagnipes, Energy Division, California Public Utilities Commission
• Sylvia Bender, California Energy Commission

We also wish to thank the California investor owned utilities and their program management and
evaluation staff who have attended workshops and provided both written and verbal comments
during the Protocol development process. And we wish to thank the public representatives who
attended the workshops and provided verbal and written comments. All of these combined
efforts helped move the development of the Protocols to a successful completion in a very short
period of time.

Lastly, we wish to thank the TecMarket Works Protocol Project Team who under direction from
the ALJ and the Joint Staff, and with useful comments from the IOUs and the public, took the
Protocols from concept to completion under the oversight of Joint Staff.
This team was made up of the following individuals:
• Nick Hall, Johna Roth, Carmen Best, TecMarket Works
CPUC i TecMarket Works Team
Evaluators’ Protocols Acknowledgements
• Sharyn Barata, Opinion Dynamics Corporation
• Pete Jacobs, Building Metrics Inc.
• Ken Keating, Ken Keating and Associates
• Steve Kromer, RCx Services
• Lori Megdal, Megdal & Associates
• Jane Peters, and Marjorie McRae, Research Into Action
• Rick Ridge, Richard Ridge and Associates
• Ed Vine, Edward Vine and Associates
• Francis Trottier, Francis Trottier Consulting

CPUC ii TecMarket Works Team
Evaluators’ Protocols Table of Contents

Table of Contents

ACKNOWLEDGEMENTS..........................................................................................................I
TABLE OF TABLES..................................................................................................................IX
TABLE OF FIGURES................................................................................................................XI
CALIFORNIA ENERGY EFFICIENCY EVALUATION PROTOCOLS: TECHNICAL,
METHODOLOGICAL AND REPORTING REQUIREMENTS FOR EVALUATION
PROFESSIONALS {A.K.A. EVALUATORS’ PROTOCOLS}...............................................1
INTRODUCTION.........................................................................................................................1
HOW THE PROTOCOLS WERE DEVELOPED...............................................................................5
HOW THE PROTOCOLS WORK TOGETHER.................................................................................6
HOW THE PROTOCOLS MEET CPUC GOALS.............................................................................7
USE OF THE EVALUATION RESULTS TO DOCUMENT ENERGY SAVINGS AND DEMAND IMPACTS9
THE EVALUATION IDENTIFICATION AND PLANNING PROCESS..................................................9
EVALUATION RIGOR AND BUDGETS.......................................................................................13
Impact Evaluations........................................................................................................................................13
Process Evaluations......................................................................................................................................14
Market Effects Evaluations...........................................................................................................................14
Codes and Standards and Emerging Technology Program Evaluations......................................................14
Evaluation Budgets.......................................................................................................................................15
RECOMMENDATIONS FOR USING THE PROTOCOLS..................................................................15
THE DETAILED EVALUATION WORK PLAN.............................................................................15
CONFIDENTIALITY ISSUES.......................................................................................................17
CONTACTING THE CUSTOMER.................................................................................................18
IMPACT EVALUATION PROTOCOL..................................................................................19
INTRODUCTION.......................................................................................................................19
AUDIENCE AND RESPONSIBLE ACTORS...................................................................................21
OVERVIEW OF THE PROTOCOL................................................................................................21
Protocol Types...............................................................................................................................................21
Rigor..............................................................................................................................................................22
Key Metrics, Inputs and Outputs...................................................................................................................23
ENERGY AND DEMAND IMPACT PROTOCOLS..........................................................................25
Gross Energy Impact Protocol......................................................................................................................25
Gross Demand Impact Protocol....................................................................................................................32
Participant Net Impact Protocol...................................................................................................................36
INDIRECT IMPACT EVALUATION PROTOCOL...........................................................................40
GUIDANCE ON SKILLS REQUIRED TO CONDUCT IMPACT EVALUATIONS.................................45
SUMMARY OF PROTOCOL-DRIVEN IMPACT EVALUATION ACTIVITIES....................................46
MEASUREMENT AND VERIFICATION (M&V) PROTOCOL........................................49
INTRODUCTION.......................................................................................................................49
AUDIENCE AND RESPONSIBLE ACTORS...................................................................................50
CPUC iii TecMarket Works Team
Evaluators’ Protocols Table of Contents
OVERVIEW OF THE PROTOCOL................................................................................................50
M&V Framework & Language.....................................................................................................................51
Relationship of the M&V Protocol to Other Protocols.................................................................................51
Key Metrics, Inputs and Outputs...................................................................................................................52
SITE-SPECIFIC M&V PLAN.....................................................................................................54
M&V RIGOR LEVELS.............................................................................................................56
MEASURE INSTALLATION VERIFICATION................................................................................56
Measure Existence.........................................................................................................................................56
Installation Quality.......................................................................................................................................56
Correct Operation and Potential to Generate Savings.................................................................................57
M&V PROTOCOL FOR BASIC LEVEL OF RIGOR.......................................................................57
IPMVP Option...............................................................................................................................................57
Sources of Stipulated Data............................................................................................................................58
Baseline Definition........................................................................................................................................58
Monitoring Strategy and Duration................................................................................................................58
Weather Adjustments.....................................................................................................................................58
M&V PROTOCOL FOR ENHANCED LEVEL OF RIGOR...............................................................58
IPMVP Option...............................................................................................................................................59
Sources of Stipulated Data............................................................................................................................59
Baseline Definition........................................................................................................................................60
Monitoring Strategy and Duration................................................................................................................60
Weather Adjustments.....................................................................................................................................60
Calibration Targets.......................................................................................................................................60
Additional Provisions....................................................................................................................................61
M&V APPROACH EXAMPLES.................................................................................................61
OVERALL RESULTS REPORTING..............................................................................................62
SAMPLING STRATEGIES..........................................................................................................63
SKILLS REQUIRED FOR M&V.................................................................................................63
SUMMARY OF PROTOCOL-DRIVEN M&V ACTIVITIES............................................................63
EMERGING TECHNOLOGIES PROTOCOL......................................................................65
INTRODUCTION.......................................................................................................................65
Audience and Responsible Actors.................................................................................................................66
Key Metrics, Inputs, and Outputs..................................................................................................................66
Evaluation Planning......................................................................................................................................67
A Sample of Available ETP Evaluation Methods..........................................................................................68
Protocols Requirements................................................................................................................................69
Integration of Results....................................................................................................................................75
Reporting of Results......................................................................................................................................75
Summary........................................................................................................................................................76
Summary of Protocol-Driven Emerging Technology Evaluation Activities..................................................76
References.....................................................................................................................................................77
CODES AND STANDARDS AND COMPLIANCE ENHANCEMENT EVALUATION
PROTOCOL................................................................................................................................81
INTRODUCTION.......................................................................................................................81
Audience and Responsible Actors.................................................................................................................82
Key Inputs, and Outputs................................................................................................................................83
Evaluation Methods.......................................................................................................................................84
Evaluation Planning......................................................................................................................................84
Technology-Specific Code and Standard Change Theory.............................................................................85
Evaluation Approach.....................................................................................................................................88
CPUC iv TecMarket Works Team
Evaluators’ Protocols Table of Contents
Summary........................................................................................................................................................98
Summary of Protocol-Driven Codes and Standards Evaluation Activities...................................................99
CODE COMPLIANCE ENHANCEMENT PROGRAMS..................................................................100
Definition of a Code Compliance Enhancement Program..........................................................................100
What this Protocol is Designed To Do........................................................................................................100
Joint Staff Responsibilities..........................................................................................................................100
DRAFT EVALUATION PLAN...................................................................................................101
Program Theory Review and Assessment....................................................................................................101
Pre-Program Compliance Rate...................................................................................................................101
Post-Program Compliance Rate..................................................................................................................102
Adjustment For Naturally Occurring Compliance Change.........................................................................102
Net Program-Induced Compliance Change................................................................................................102
Assessment of Energy Savings.....................................................................................................................103
Recommendations for Program Changes....................................................................................................103
Cost Effectiveness Assessment.....................................................................................................................103
REPORTING OF EVALUATION RESULTS.................................................................................103
REFERENCES.........................................................................................................................104
EFFECTIVE USEFUL LIFE EVALUATION PROTOCOL (RETENTION AND
DEGRADATION).....................................................................................................................105
INTRODUCTION.....................................................................................................................105
AUDIENCE AND RESPONSIBLE ACTORS.................................................................................107
OVERVIEW OF THE PROTOCOL..............................................................................................108
Protocol Types.............................................................................................................................................108
Rigor............................................................................................................................................................109
Key Metrics, Inputs, and Outputs................................................................................................................110
RETENTION STUDY PROTOCOL.............................................................................................111
DEGRADATION STUDY PROTOCOL........................................................................................116
EFFECTIVE USEFUL LIFE ANALYSIS PROTOCOL....................................................................119
REPORTING REQUIREMENTS.................................................................................................126
STUDY SELECTION AND TIMING...........................................................................................127
GUIDANCE ON SKILLS REQUIRED TO CONDUCT RETENTION, EUL, AND TECHNICAL
DEGRADATION EVALUATIONS..............................................................................................128
Summary of Protocol-Driven Impact Evaluation Activities........................................................................130
PROCESS EVALUATION PROTOCOL..............................................................................131
INTRODUCTION.....................................................................................................................131
AUDIENCE AND RESPONSIBLE ACTORS.................................................................................132
OVERVIEW OF THE PROTOCOL..............................................................................................132
PROCESS EVALUATION PLANNING........................................................................................133
ANNUAL PROCESS EVALUATION PLANNING MEETING.........................................................134
RECOMMENDATIONS FOR CHANGE.......................................................................................135
KEY ISSUES AND INFORMATION COVERED...........................................................................135
PROCESS EVALUATION EFFORTS..........................................................................................136
Program-Specific Process Evaluation Plans..............................................................................................137
Data Collection and Assessment Efforts.....................................................................................................137
Conducting Investigative Efforts.................................................................................................................138
CPUC v TecMarket Works Team
Evaluators’ Protocols Table of Contents
Independence...............................................................................................................................................140
Selection of Evaluation Contractors...........................................................................................................140
SKILLS REQUIRED FOR CONDUCTING PROCESS EVALUATIONS.............................................140
MARKET EFFECTS EVALUATION PROTOCOL............................................................143
INTRODUCTION.....................................................................................................................143
Overview of the Market Effects Protocol....................................................................................................143
The Market Effects Protocol and Other Protocols......................................................................................146
Key Market Effects Inputs and Outputs.......................................................................................................147
AUDIENCE AND RESPONSIBLE ACTORS.................................................................................148
STEPS IN CONDUCTING MARKET EFFECTS EVALUATIONS....................................................148
Scoping Study..............................................................................................................................................149
Collection of Baseline and Longitudinal Indicators....................................................................................154
Analysis of Market Effects...........................................................................................................................155
Reporting.....................................................................................................................................................158
GUIDANCE ON SKILLS REQUIRED TO CONDUCT MARKET EFFECTS EVALUATIONS...............159
CONSIDERATIONS FOR CONDUCTING A MARKET EFFECTS EVALUATION..............................159
SUMMARY OF PROTOCOL-DRIVEN MARKET EFFECTS EVALUATION ACTIVITIES..................160
SAMPLING AND UNCERTAINTY PROTOCOL...............................................................163
INTRODUCTION.....................................................................................................................163
PRECISION: GROSS AND NET IMPACT, MEASUREMENT AND VERIFICATION, AND VERIFICATION
ACTIVITIES...........................................................................................................................164
DEVELOPMENT OF THE EVALUATION STUDY WORK PLAN...................................................167
PROCESS EVALUATIONS.......................................................................................................169
MARKET EFFECTS.................................................................................................................170
SYSTEM LEARNING...............................................................................................................170
ACCEPTABLE SAMPLING METHODS......................................................................................171
SKILLS REQUIRED FOR SAMPLING & UNCERTAINTY............................................................171
AUDIENCE AND RESPONSIBLE ACTORS.................................................................................171
KEY METRICS AND INFORMATION COVERED........................................................................172
Sample Size and Precision..........................................................................................................................172
Validity and Research Design.....................................................................................................................172
Accuracy......................................................................................................................................................173
Summary of Sampling and Uncertainty Protocol........................................................................................175
EVALUATION REPORTING PROTOCOL........................................................................177
INTRODUCTION.....................................................................................................................177
Report Delivery Dates.................................................................................................................................177
Common Evaluation Reporting Requirements............................................................................................177
Evaluation Type Specific Reporting Requirements.....................................................................................183
SAMPLE REPORTING TABLES................................................................................................195
EVALUATION SUPPORT INFORMATION NEEDED FROM ADMINISTRATORS..205
Storage and Disposal of Customer Information Used in the Evaluation....................................................211
APPENDIX A. MEASURE-LEVEL M&V RESULTS REPORTING REQUIREMENTS
.....................................................................................................................................................213
APPENDIX B. GLOSSARY OF TERMS.............................................................................217
APPENDIX C. PERFORMANCE BASIS METRICS.........................................................247
CPUC vi TecMarket Works Team