Process and Outcome Evaluation of the Alternative and Renewable Fuel and Vehicle Technology

367 Pages·2017·4.67 MB·English

Alternat ive and Renewable Fue l and Vehic le
Technology Program
FINAL PROJECT REPORT

Process and Outcome Evaluation of
the Alternative and Renewable Fuel
and Vehicle Technology Program

Prepared for: California Energy Commission
Prepared by: RAND Corporation
SEPTEMBER 2017
CEC-600-11 -004
For more information on this publication, visit www.rand.org/t/rr1948

Prepared by:

Authors
From RAND Corporation
From Sierra Research
Lloyd Dixon
James Lyons
Tom LaTourrette
From Advent Consulting
David Galvan
Jerry Bowers
Charles Goldman
From Katin Engineering Consulting
Nidhi Kalra
Bob Katin
Christopher Nelson

Flavia Tsang
Paul Steinberg

Agreement Number: 600-11-004

Prepared for:
California Energy Commission

Jim McKinney, Pierre duVair, Aida Escala
Agreement Managers

John Butler
Office Manager
Zero Emission Vehicle and Infrastructure Office

John Kato
Deputy Director
Fuels and Transportation Division

Robert P. Oglesby
Executive Director
DISCLAIMER
This report was prepared as the result of work sponsored by the California Energy Commission. It
does not necessarily represent the views of the Energy Commission, its employees or the State of
California. The Energy Commission, the State of California, its employees, contractors and
subcontractors make no warrant, express or implied, and assume no legal liability for the information
in this report; nor does any party represent that the uses of this information will not infringe upon
privately owned rights. This report has not been approved or disapproved by the California Energy
Commission nor has the California Energy Commission passed upon the accuracy or adequacy of
the information in this report.
ii
ACKNOWLEDGMENTS
We acknowledge the Commission Agreement Managers (CAMs) for this project—Jim McKinney,
Pierre duVair, and Aida Escala—for facilitating the evaluation by providing administrative data,
project documents, and introductions to numerous California Energy Commission (CEC) staff.
We also thank the CAMs for numerous other projects who met with us to discuss their projects
and who responded to our requests for data. We are especially grateful to Elizabeth John, who
led the effort to compile and send us administrative databases and project documentation for
more than 300 projects. We are also indebted to senior Alternative and Renewable Fuel and
Vehicle Technology Program (ARFVTP) staff for meeting with us to discuss program strategy,
implementation, and lessons learned. And we would like to thank Jean Baronas, Pierre duVair,
Bill Kinney, David Nichols, and Matthew Ong at CEC for comments on the draft report.
We also gratefully acknowledge the members of the project’s Evaluation Advisory Committee:
David Greene from Oak Ridge National Laboratory and the Howard H. Baker, Jr. Center for
Public Policy at the University of Tennessee; Harry Hatry from the Urban Institute; Ivor John
from Lloyd's Register Quality Assurance; Dan Kammen from the Energy and Resources Group
and the Goldman School of Public Policy at the University of California, Berkeley; and Sonia Yeh
from the Institute of Transportation Studies at the University of California, Davis. The
committee provided valuable guidance and recommendations at several points during the
evaluation, including reviewing the draft final report.
We thank the experts at Stillwater Associates for their technical support of the in-depth project
reviews and for compiling indicator data for the alternative fuels and fueling infrastructure
markets in California.
At RAND, Karin Liu of RAND’s Survey Research Group provided valuable advice on designing
the instrument for the study survey and did an outstanding job pretesting and fielding it. Katie
Loa analyzed the program investment plans and program opportunity notices. Jim Powers
wrote a constructive peer review of the draft report, Keith Crane provided helpful advice during
the course of the project, and Marjory Blumenthal, director of RAND’s Science, Technology, and
Policy Program, provided much-appreciated comments on the draft report. Paul Sorensen,
previously at RAND, played a lead role in putting the project together, designing the evaluation
approach, and writing background documents on barriers to market viability. Martin Wachs,
also previously at RAND, was instrumental in assembling the Evaluation Advisory Committee.
Linda Theung skillfully edited the document. We thank them for their contributions.
Finally, we would like to thank the 475 ARFVTP awardees, unsuccessful applicants, and
potential applicants who completed a detailed survey about the program and the 38 awardees
who participated in in-depth project reviews. Their input was essential to a comprehensive
evaluation of the program.

iii
PREFACE
Assembly Bill (AB) 118 (Núñez, Chapter 750, Statutes of 2007), created the Alternative and
Renewable Fuel and Vehicle Technology Program (ARFVTP). The statute, subsequently amended
by AB 109 (Núñez, Chapter 313, Statutes of 2008), authorizes the California Energy Commission
(CEC) to develop and deploy alternative and renewable fuels and advanced transportation
technologies to help attain the state’s climate change policies. The CEC has an annual program
budget of about $100 million and provides financial support for projects that
• develop and improve alternative and renewable low-carbon fuels
• enhance alternative and renewable fuels for existing and developing engine technologies
• produce alternative and renewable low-carbon fuels in California
• decrease, on a full-fuel-cycle basis, the overall impact and carbon footprint of alternative
and renewable fuels and increase sustainability
• expand fuel infrastructure, fueling stations, and equipment
• improve light-, medium-, and heavy-duty vehicle technologies
• retrofit medium- and heavy-duty on-road and nonroad vehicle fleets
• expand infrastructure connected with existing fleets, public transit, and transportation
corridors
• establish workforce training programs, conduct public education and promotion, and
create technology centers.
The Energy Commission issued solicitation RFP-600-10-612 to provide technical support to the
Energy Commission with evaluation, measurement, and verification of the ARFVTP. To be
eligible for funding under RFP-600-10-612, the projects must also be consistent with the Energy
Commission’s ARFVT Investment Plan, updated annually. In response to RFP-600-10-612, the
RAND Corporation submitted application 1, which was proposed for funding in the Energy
Commission’s Notice of Proposed Awards March 6, 2012, and the agreement was executed as
600-11-004 on August 21, 2012.
The research was conducted in the Science, Technology, and Policy Program of RAND’s Justice,
Infrastructure, and Environment unit. The findings should be of interest to the California State
Legislature, the CEC, and other state and federal agencies that are involved in efforts to
promote the development of new technologies and products.

iv
v
ABSTRACT
This report presents the results of a process and outcome evaluation of the California Energy
Commission’s (CEC’s) Alternative and Renewable Fuel and Vehicle Technology Program
(ARFVTP), relying on quantitative and qualitative methods. For the process evaluation, we
conclude the following: The process for developing the ARFVTP’s investment plan has been
reasonable; CEC has targeted major barriers to market viability across the main fuel-sector
types funded by the ARFVTP (hydrogen, electricity, diesel substitutes, gasoline substitutes, and
natural gas and biomethane); the ARFVTP has been successful in soliciting proposals; the
criteria and weights that the CEC used to evaluate award applications are appropriate, but there
are some concerns about the amount of time required to review applications and execute
agreements; there are both strengths and weaknesses in how CEC oversees grants once they
begin; completing projects on time has been a challenge for many awardees; and awardees are
largely achieving their projects’ most-central technical objectives. For the outcomes so far, we
conclude that the ARFVTP has made considerable progress reducing many barriers to the
market viability of alternative fuels and vehicles. We also find that the ARFVTP seems to have
allowed awardees to proceed with projects they would not have undertaken otherwise; after
completing their ARFVTP projects, awardees overwhelmingly indicated they had taken steps to
continue to develop, market, install, or produce the funded product or technology; and on the
workforce side, ARFVTP project grants supported some employment both in California and
elsewhere and did result in some hiring for the projects. Stakeholder suggestions on ARFVTP
investment priorities going forward vary by fuel type but generally call for continued program
support. We also note that insufficient consumer awareness of, and knowledge about,
alternative fuels and vehicles was one of the most frequently identified barriers to market
viability across all fuel types and that the ARFVTP should not shy away from research,
development, and demonstration projects simply because their outcomes can be difficult to
quantify.

Keywords: Alternative fuels and vehicles, program evaluation

Please use the following citation for this report:
Dixon, Lloyd, Tom LaTourette, David Galvan, Charles Goldman, Nidhi Kalra, Christopher Nelson,
Flavia Tsang, Paul Steinberg, James Lyons, Jerry Bowers, Bob Katin. 2017. Process and
Outcome Evaluation of the Alternative and Renewable Fuel and Vehicle Technology
Program. California Energy Commission. Publication Number: CEC-600-11-004. RAND
Corporation Publication Number: RR-1948-CEC.

vi
Acknowledgments ........................................................................................................................................ iii
PREFACE .......................................................................................................................................................... iv
ABSTRACT ...................................................................................................................................................... vi
EXECUTIVE SUMMARY ..............................................................................................................................x vii
S.1 Introduction ....................................................................................................................................... xvii
S.2 Key Conclusions ................................................................................................................................... xx
S.2.1 Conclusions from the Process Evaluation ............................................................................... xx
S.2.2 Conclusions from the Outcome Evaluation ......................................................................... xxv
S.3 Observations on ARFVTP Investment Priorities ...................................................................... xxviii
S.3.1 Observations for Hydrogen .................................................................................................... xxix
S.3.2 Observations for Electricity .................................................................................................... xxix
S.3.3 Observations for Diesel Substitutes ...................................................................................... xxx
S.3.4 Observations for Gasoline Substitutes ................................................................................. xxx
S.3.5 Observations for Natural Gas and Biomethane ................................................................... xxx
S.3.6 Observations on Workforce Development .......................................................................... xxxi
S.3.7 Observations on Consumer Awareness ............................................................................... xxxi
S.3.8 Observations on Research, Development, and Demonstration ...................................... xxxi
CHAPTER 1: Introduction ............................................................................................................................ 1
1.1 Background ............................................................................................................................................ 1
1.2 Objectives of Evaluation ..................................................................................................................... 3
1.2.1 Conceptual Framework for Evaluation ..................................................................................... 3
1.2.2 Evaluation Questions .................................................................................................................... 5
1.2.3 Evaluation Categories ................................................................................................................... 5
1.3 Evaluation Approach ............................................................................................................................ 6
1.3.1 Review Project Documentation to Arrive at an Independent Assessment of
the Extent to Which Project Technical Objectives Were Achieved ............................................... 6
1.3.2 Survey Awardees, Unsuccessful Applicants, and Potential Applicants ............................. 7
1.3.3 Conduct In-Depth Reviews of Selected ARFVTP Projects ..................................................... 8
1.3.4 Interview CEC Staff About Processes for Administering the Program .............................. 9
1.3.5 Verify Installation of EVSE for Selected Set of Awards ....................................................... 10
vii
1.3.6 Assemble and Analyze CEC Administrative Data on Program Processes and
Interview CEC Staff on Program Processes ...................................................................................... 10
1.3.7 Collect Additional Information to Inform the Outcome Evaluation ................................ 10
1.3.8 Evaluation Scope ......................................................................................................................... 11
1.4 Organization of This Document ...................................................................................................... 12
CHAPTER 2: The Process for Developing and Executing ARFVTP Investment Plans ............... 14
2.1 Purpose and Key Questions .............................................................................................................. 14
2.2 Overview of the Investment Planning Process ............................................................................. 15
2.3 Stakeholder Involvement in Investment Planning ....................................................................... 16
2.4 Adapting the Strategy Based on Experience ................................................................................. 21
2.5 Types of Funding Mechanisms ........................................................................................................ 22
2.6 Attracting Applicants ........................................................................................................................ 23
2.6.1 How People Learn About the Program .................................................................................... 24
2.6.2 Perceptions About Applicant Requirements ......................................................................... 25
2.6.3 PON Subscription Rates ............................................................................................................. 26
2.6.4 Repeat Applicants ....................................................................................................................... 27
2.7 Evaluating and Selecting Among Applications ............................................................................. 27
2.7.1 Evaluators ..................................................................................................................................... 28
2.7.2 Evaluation Process ...................................................................................................................... 28
2.7.3 Evaluation Criteria ...................................................................................................................... 29
2.7.4 Proposal Success Rate ................................................................................................................ 30
2.8 Match Funding ..................................................................................................................................... 30
2.9 Time to Review Applications and Execute Agreements ............................................................. 32
2.10 Overseeing Agreements .................................................................................................................. 35
2.10.1 Invoicing ..................................................................................................................................... 35
2.10.2 Project Managers ....................................................................................................................... 37
2.10.3 Progress Monitoring and Data Collection ............................................................................ 40
2.11 Conclusions ....................................................................................................................................... 41
CHAPTER 3: Impact of Program on Barriers to Market Viability of
Hydrogen-Fueled Vehicles ......................................................................................................................... 44
viii
3.1 Summary of Data Sources ................................................................................................................ 44
3.2 Technical Overview of Hydrogen as a Fuel and of FCEVs .......................................................... 45
3.2.1 Hydrogen as a Vehicle Fuel ....................................................................................................... 45
3.2.2 FCEV Technologies ...................................................................................................................... 45
3.3 Barriers to Use of Hydrogen as a Transportation Fuel ............................................................... 47
3.3.1 Feedstock Barriers ...................................................................................................................... 48
3.3.2 Production Barriers ..................................................................................................................... 49
3.3.3 Distribution Barriers ................................................................................................................... 50
3.3.4 Vehicle Barriers ........................................................................................................................... 51
3.3.5 Barriers Identified in the Survey .............................................................................................. 53
3.4 Review of CEC’s Solicitations and Awards in Hydrogen ............................................................ 54
3.4.1 Hydrogen Funding and Solicitations ....................................................................................... 54
3.4.2 Hydrogen Applications and Awards ....................................................................................... 56
3.4.3 Barriers Targeted ......................................................................................................................... 57
3.5 Awardee Performance ....................................................................................................................... 59
3.5.1 Project Status ............................................................................................................................... 59
3.5.2 Technical Performance .............................................................................................................. 60
3.5.3 Financial Performance ................................................................................................................ 63
3.5.4 Schedule Performance ................................................................................................................ 63
3.6 Impacts on Awardees ........................................................................................................................ 66
3.6.1 Impact on Undertaking Project ................................................................................................ 66
3.6.2 Impact on Knowledge and Capabilities .................................................................................. 68
3.6.3 Inventions or Products Developed During the Project and Market Prospects ............... 70
3.7 Impacts on Barriers ............................................................................................................................ 70
3.7.1 Impacts on Fueling Infrastructure Barriers ........................................................................... 72
3.7.2 Impacts on Feedstock and Fuel-Production Barriers ........................................................... 75
3.7.3 Impacts on Vehicle Barriers ...................................................................................................... 75
3.7.4 Impacts on Other Barriers ......................................................................................................... 76
3.8 State of the Industry .......................................................................................................................... 76
ix
3.9 Views of Hydrogen Stakeholders on ARFVTP Investment Priorities ....................................... 79
3.10 Conclusions ....................................................................................................................................... 80
CHAPTER 4: Impact of Program on Barriers to Market Viability of
Plug-In Electric Vehicles ............................................................................................................................. 82
4.1 Summary of Data Sources ................................................................................................................ 82
4.2 Technical Overview of Electricity as a Fuel and of Electric and Hybrid Vehicles ................. 83
4.2.1 Electricity as a Transportation Fuel ........................................................................................ 83
4.2.2 Electric Vehicle Technologies ................................................................................................... 86
4.3 Barriers to Use of Electricity as a Transportation Fuel ............................................................... 88
4.4 Review of CEC’s Solicitations and Electricity Awards ................................................................. 92
4.4.1 Barriers Targeted by Electricity Projects ................................................................................ 93
4.5 Awardee Performance ....................................................................................................................... 96
4.5.1 Project Status ............................................................................................................................... 96
4.5.2 Technical Performance .............................................................................................................. 98
4.5.3 Financial Performance .............................................................................................................. 109
4.5.4 Schedule Performance .............................................................................................................. 110
4.6 Impacts on Awardees ...................................................................................................................... 111
4.6.1 Effect of Award on Ability to Undertake the Project ......................................................... 111
4.6.2 Impact on Organizational Capabilities ................................................................................. 113
4.6.3 Market Prospects for the Product or Technology Funded by the Award ...................... 114
4.7 Impacts on Barriers .......................................................................................................................... 115
4.7.1 Fueling Infrastructure Barriers Reduced by ARFVTP Projects ......................................... 115
4.7.2 Vehicle Barriers Reduced by ARFVTP Projects ................................................................... 118
4.7.3 Consumer Awareness Barriers Reduced by ARFVTP Projects ......................................... 121
4.8 State of the EV Industry .................................................................................................................. 121
4.9 Views of Electric Vehicle Stakeholders on ARFVTP Investment Priorities ........................... 124
4.10 Conclusions ..................................................................................................................................... 125
CHAPTER 5: Impact of Program on Barriers to Market Viability of Diesel Substitutes .......... 128
5.1 Summary of Data Sources .............................................................................................................. 128
5.2 Technical Overview of Diesel Substitutes as a Transportation Fuel ...................................... 129
x