Support study for the Evaluation of ENIAC/ARTEMIS/ECSEL JU

155 Pages·2017·5.45 MB·English

Support study for the
Evaluation of
ENIAC/ARTEMIS/ECSEL JU

FINAL REPORT
A study prepared for the European Commission
DG Communications Networks, Content & Technology
by:

Digital
Single
Market

This study was carried out for the European Commission by
Consultores de Automatización y Robótica, S.A

Ramboll Management Consulting A/S

AIT Austrian Institute of Technology GmbH

Agilis SA

This study was written by David Vidal (CARSA), Pär Weström (CARSA), Ieva Savickaite (CARSA), Thibaud
Lalanne (CARSA), Xavier Le Den (Ramboll), Edmund Beavor (Ramboll), Oriane Gaillard (Ramboll), Michael
Dinges (AIT), Michael Barber (AIT), Georg Zahradnik (AIT), Thomas Scherngell (AIT), Anais Santourian (Agilis),
Sonia Chalkidou (Agilis).
We would like to thank to Francisco Ibañez, Nikolaos Kattavenos, Eric Fribourg-Blanc and Inês Reis from the
European Commission who supervised this study.

Internal identification
Contract number: 30-CE-0803220/00-96

SMART 2016/0005

DISCLAIMER
By the European Commission, Directorate-General of Communications Networks, Content & Technology.
The information and views set out in this publication are those of the author(s) and do not necessarily reflect
the official opinion of the Commission. The Commission does not guarantee the accuracy of the data included in
this study. Neither the Commission nor any person acting on the Commission’s behalf may be held responsible
for the use which may be made of the information contained therein.
ISBN 978-92-79-68012-0
doi:10.2759/518091
© European Union, 2017. All rights reserved. Certain parts are licensed under conditions to the EU.
Support study for the Evaluation of ENIAC/ARTEMIS/ECSEL JU

SMART 2016/0005
TABLE OF CONTENTS
Table of Contents .................................................................................................................................... 1
Executive summary ................................................................................................................................. 3
1 Introduction ....................................................................................................................................... 9
1.1 Objectives of the study ............................................................................................................... 9
1.2 Methodological overview ............................................................................................................ 9
1.3 Methodological clarifications .................................................................................................... 10
2 Evaluation of ENIAC and ARTEMIS JU ......................................................................................... 13
2.1 ENIAC and ARTEMIS JUs overview ........................................................................................ 13
2.2 ENIAC and ARTEMIS JUs essential statistics ......................................................................... 22
2.3 Implementation of research agenda and impact ...................................................................... 47
2.4 Participation and funding in projects ........................................................................................ 81
2.5 Mobilising and pooling efforts to increase investments .......................................................... 101
2.6 Synergy and coordination of European R&D including Eureka ............................................. 108
2.7 Involvement of SMEs .............................................................................................................. 115
3 Interim Evaluation of the ECSEL JU ............................................................................................ 116
3.1 ECSEL JU overview ............................................................................................................... 116
3.2 Transition to ECSEL JU.......................................................................................................... 118
Index of Figures ................................................................................................................................... 146
Index of Tables .................................................................................................................................... 150
Index of Boxes ..................................................................................................................................... 150

Final Study Report 1 of 150

Support study for the Evaluation of ENIAC/ARTEMIS/ECSEL JU

SMART 2016/0005
Executive summary
The study “Support study for the Evaluation of ENIAC/ARTEMIS/ECSEL JU” - SMART 2016/0005
provides evidence for and support to the Final evaluation of ENIAC and ARTEMIS Joint Undertakings
(JUs) and it also includes the interim evaluation of ECSEL JU. The study has been carried out by a
consortium composed of: CARSA (Spain, Lead Contractor), Ramboll Management Consulting
(Denmark), AIT−Austrian Institute for Technology (Austria) and Agilis (Greece).
The methodology for data collection and data analysis includes bibliographical review of relevant
sources, portfolio analysis of data bases of involved projects, surveys to JU representatives, project
participants and coordinators, as well as interviews with relevant stakeholders. The findings obtained
by the consortium have been provided to an expert panel in form of evidence to answer to the
evaluation questions and make recommendations. Findings have been obtained from the projects of
the calls from the second term of the ENIAC/ARTEMIS JUs1 and from the first term of the ECSEL
JUs2.
The figures below present the distribution of projects by research area for both ENIAC and ARTEMIS
JUs, respectively:
400,0 20
350,0 340,4 18
16
300,0
14
245,1
250,0 12 stc
sorue n200,0 10 ejorp fo
oilliM 156,9 8 rebm
150,0 uN
93,6 6
100,0
4
54,3 58,6
42,6
50,0
24,4 2
0,0 0
Automotive and Communication & Design Technologies Energy Efficiency Equipment, Materials Health and the Ageing Safety and Security Semiconductor Proces
Transport Digital Life Style and Manufacturing Society and Integration
Total EC & MS Contribution (Million euros) Nr. of Projects

The analysis of the ARTEMIS project portfolio shows that the number of projects is concentrated
around a limited number of ASPs3:

1 Including data from calls : ARTEMIS-2008, ARTEMIS-2009, ARTEMIS-2010, ARTEMIS-2011, ARTEMIS-2012, ARTEMIS-
2013, ENIAC-2008-1, ENIAC-2009-1, ENIAC-2010-1, ENIAC-2011-1, ENIAC-2011-2, ENIAC-2012-1, ENIAC-2012-2, ENIAC-
2013-1, ENIAC-2013-2.
2 Including data from calls : ECSEL-2014-1, ECSEL-2014-2, H2020-ECSEL-2015-1-RIA-TWO-STAGE, H2020-ECSEL-2015-2-
IA-TWO-STAGE.
3 For reasons of simplicity, each project has been assigned to one ASP/AIPP only, although several projects address more than
one ASP.

Final Study Report 3 of 150
Support study for the Evaluation of ENIAC/ARTEMIS/ECSEL JU

SMART 2016/0005
140,0 16
127,4
14
120,0
103,8
12
100,0
10 stc
sorue n 80,0 8 ejorp fo
oilliM 60,0 rebm
47,7 47,7 6 u
42,4 N
40,0 36,8 36,2
22,1 30,0 4
19,8
20,0
2
9,3
0,0 0
ASP1-Methods AIPP1-Critical ASP2-Embedded ASP3-Embedded ASP4-Embedded AIPP-4 ASP5-Computing AIPP-5 ASP6-Embedded ASP7-Embedded ASP8-Human-
and processes Systems Systems for systems in Smart Systems for Production and platforms for Computing Systems for Systems centred design
for safety- Engineering Healthcare and environments manufacturing Energy Systems embedded Platforms for Security and supporting of embedded
relevant Factories Wellbeing and process Automation systems Embedded Critical sustainable systems
embedded automation Systems Infrastructures urban life
systems Protection
Total EC & MS Contribution (Million euros) Nr. of Projects

The conclusions obtained from the final evaluation of ENIAC and ARTEMIS JUs, for each of the
evaluation questions (Implementation of research agenda and impact; Participation and funding in
projects; Mobilising and pooling efforts to increase investments; Synergy and coordination of
European R&D including Eureka; Involvement of SMEs) are the following:
Implementation of research agenda and impact
As perceived by participants and JU representatives, ENIAC and ARTEMIS have contributed to the
visibility of and support for the research and innovation agendas of the communities /
stakeholders on both micro-nanoelectronics and embedded systems research fields. Positive progress
has been made in this respect since the second interim evaluation. Stakeholders also appreciate in a
positive way the JUs ability to react to new challenges and opportunities by modification of the
research agendas, for example through the implementation of ARTEMIS Innovation Pilot Projects and
ENIAC large scale pilot lines in order to better support projects at higher technology readiness level
and bring R&D closer to the market.
Previous evaluations of the JUs and other literature examined revealed an initial critical lack of
coordination between ENIAC and CATRENE and between ARTEMIS and ITEA, however, substantial
improvement overtime were reported in both cases. The efforts of improved cooperation, e.g. by joint
development of strategic vision documents and multi annual research agendas, are indeed recognised
in the comments from surveys and interviews carried out in this study. But, although the overall
assessments from the two surveys are fairly positive around 25% of the respondents (excluding those
with no opinion) assessed the consideration of links between the programmes as low or non-existing.
This indicates that there is room for improvement with respect to coordination and complementarities
between ECSEL and related EUREKA clusters during the continued implementation of the
programmes.
With respect to consideration of links to national programmes the two surveys gave similar results,
with a majority of the respondents (65-70%) assessing them positively (i.e. links taken into account to
some extent or to a high extent), but still a significant share being negative. Interviews with
representatives disclose that the links to national programmes differ from case to case depending e.g.
on the priorities of the national programme or the funding capabilities: Further, the interviews indicate
the national programmes of Netherlands, France and Finland as strongly linked to ARTEMIS, and the
ones of France, Germany and Austria as strongly linked to ENIAC; which is basically in line with the
participation patterns observed.

Final Study Report 4 of 150
Support study for the Evaluation of ENIAC/ARTEMIS/ECSEL JU

SMART 2016/0005
With respect to achievement of the
initial objectives of ENIAC and Extent to which the initial objectives of ENIAC and ARTEMIS
have been achieved
ARTEMIS the overall evaluation is
positive. A vast majority (more than 90% 80%
90%) of surveyed representatives, 80% 69.2%
70% 62.5%
as well as project coordinators,
60%
consider that each part of the
50%
objectives has been achieved at
40%
least to some extent. 25.0%
30% 20% 23.1%
Regarding the socio-economic 20% 12.57%.7%
10%
impact of the JUs the study findings
0%
indicate that the impact of both Not at all Low Fair High Do not know
ENIAC and ARTEMIS is positive on
ENIAC ARTEMIS All
several aspects of the socio-
economic nature. The survey to JU representatives shows that the private representatives are much
more positive than public ones with respect to perceived impact on typical socio-economic indicators,
such as job creation, growth of the economy, reduction of energy consumption and ensuring safety
and security in the related domains.
The relevance of the economic support provided by both JUs has been confirmed during the
study. As indicated by JU representatives and project participants, both ENIAC and ARTEMIS JUs
have enabled research to be performed that would otherwise not have been possible. General
examples of the enabled research activities include: increased overall funding, large projects/scope,
increased international cooperation and platform building.
The most frequent project outputs (direct and indirect results) are: publications, new follow-on
projects and new or significantly improved commercial products or services. Additionally, creation of
common platforms, reference architecture, development of pilot testing facilities and contribution to
standard development, are examples of indirect results that have been identified through the surveys
and interviews performed in the study. In particular, secondary data from ECSEL reveal that the
publications and patents generated by ENIAC projects completed up to and including 2015 led to a
noteworthy 4.7 patents per 10 million euro of EU funding. The corresponding figure for ARTEMIS
projects is 1.5 patents per 10 million euro of EU funding. On the other hand, regarding scientific
publications per 10 million euro of EU funding, ARTEMIS projects scored 80.5, which is significantly
higher than the score of ENIAC projects, 53.6.
Another relevant conclusion is that Technology and Manufacturing Readiness Levels of the
funded activities are substantially increased in both ENIAC and ARTEMIS projects, being slightly
more in ENIAC projects. At the beginning of projects the TRL (and MRL) levels, in mean values, were
2.7 (2.2) and 3.2 (3.2) for ENIAC and ARTEMIS, respectively. At the end of the projects corresponding
values reached 5.9 (5.6) and 5.6 (5.1) for ENIAC and ARTEMIS respectively.
ARTEMIS Centres of Innovation Excellence (CoIEs), ARTEMIS Innovation Pilot Projects (AIPPs)
and ENIAC KET Pilot Lines are relevant and innovative policy instruments widely recognised as
successful in terms of impact, synergy and leverage. This positive view is further confirmed by
surveyed ENIAC and ARTEMIS representatives and project participants. When comparing the three
distinct instruments the representatives’ assessments value ENIAC’s large scale pilot projects most
positively, followed by the ARTEMIS AIPPs. Similarly, ENIAC participants’ overall assessment of the
effects of pilot lines is more positive than ARTEMIS participants’ assessment of the effects of pilot
projects and CoIEs.
Participation and funding in projects
The national/European research community has been appropriately and sufficiently represented
in both ENIAC and ARTEMIS projects, as perceived by participants. Nevertheless, there are
different situations considering each Member State and the participating organisations, when a
detailed analysis of the project portfolio is carried out. The study findings reveal a somewhat skewed
picture for ENIAC, showing that very large amounts of ENIAC funding are allocated to a limited

Final Study Report 5 of 150
Support study for the Evaluation of ENIAC/ARTEMIS/ECSEL JU

SMART 2016/0005
number of countries (DE, FR, and NL). ARTEMIS project portfolio is constituted by a core of 9
countries, which collaborated significantly with each other, while in ENIAC interactions are
concentrated among five countries. In ARTEMIS, the first 10% of all participations (not participants)
account for slightly more than 30% of funding, and correspondingly in ENIAC the first 10% account for
more than 60% of the funding.
The study results indicate the following picture for the participating organisations: Analysis of the
project portfolios shows that structure and volume of funding differ considerably between ARTEMIS,
ENIAC and ECSEL, whereas differences concerning the type of projects funded within the project
lines are less pronounced. In terms of shares of funding allocated to SMEs, ARTEMIS exhibits higher
shares than ENIAC and ECSEL. Also research organisations (HEIs & RES) received a comparatively
higher share of funding in ARTEMIS than in ENIAC. In ECSEL the share of funding for research
organisations further increased.
The analysis of the database has shown that between 2008 and 2013 ENIAC and ARTEMIS displayed
a good level of industry participation, from both large enterprises and SMEs. Over this period SMEs
represented 29% of the participating organisations in ARTEMIS and 23% in ENIAC. In the period of
2014-2015 the stakeholder participation per type of organisation in ECSEL was similar to the
participation in ARTEMIS and ENIAC.
Additionally, the surveyed representatives of both ENIAC and ARTEMIS largely agreed that the mix of
organisations in the projects was suitably balanced and adequate to achieve the objectives. In
coherence with the findings of the project portfolio analysis and the network analysis (see section
2.4.1) the representatives’ assessment was slightly more positive for ARTEMIS than ENIAC. The
network analysis also indicates that several research and industry participants (mainly known for their
research focus on semiconductor technologies) are core participants of both ARTEMIS and ENIAC,
demonstrating that the merge of both JUs was a correct decision.
The evidence base is limited, but the assessment of the level of openness and attractiveness of the
programmes to newcomers is relatively weak (particularly from the point of view of the public ENIAC
representatives), where possible improvements could be considered for the future ECSEL calls.
With respect to the extent of
technology users Role of organisation
involvement, the study results 2.74%
are somewhat inconclusive. On Other 2.19%
3.23%
the one hand surveyed JU 30.82%
representatives largely agree Developer of technology 32.85%
29.03%
that technology end-users were
4.45%
involved to a satisfactory User of technology 1.94%7.30%
degree in both ENIAC and
5.14%
ARTEMIS projects, but on the Industrial partner: Service sectors/support provider 2.58% 8.03%
other hand surveyed
15.75%
Industrial partner: Manufacturing 8.76%
ENIAC/ARTEMIS participants 21.94%
report that users of technology 41.10%
Scientific partner 40.88%
were involved in only 4.5% of 41.29%
the projects. In any case, the
0% 5% 10% 15% 20% 25% 30% 35% 40% 45%
study team consider that a
All(cid:9)responses(cid:9)n=292 ARTEMIS(cid:9)(n=137) ENIAC(cid:9)(n=155)
higher percentage of user
involvement would have been desirable.
With respect to the adequacy of the size of the projects and the funding levels in view of
objectives of the projects and the programmes, innovation pilots are considerably larger than
collaborative projects. The average EU and MS contribution to innovation pilots was 45.6 million euro
in ARTEMIS, 37.6 million euro in ENIAC and is at present 33.4 million euro in ECSEL. The average
EU and MS contribution to collaborative projects was 10.4 million euro in ARTEMIS, 12.2 million euro
in ENIAC and is at present 11 million euro in ECSEL. On the whole, projects funded via the three JUs
are distinctly larger than other ICT related research funded in FP7 or H2020, both in terms of overall
funding provides and number of participating organisations.

Final Study Report 6 of 150
Support study for the Evaluation of ENIAC/ARTEMIS/ECSEL JU

SMART 2016/0005
According to a great majority of the surveyed JU representatives, the size and funding levels of ENIAC
and ARTEMIS projects were considered as adequate. Similarly, project coordinators find project size
adequate in terms of number of participants (88.9%), project budget (77.8%) and total funding
received (59.3%). Furthermore, 85% of the project coordinators consider that the underlying needs of
the research fields of nanoelectronics and embedded systems continue to require support at the JU
level.
The evaluation of the costs budgeted, funding awarded and actual expenditure showed that in
previous evaluations and other literature reveal that allocated funding lagged behind planned targets
in the initial years of the ENIAC programme, however, significant improvements in 2012 and 2013
ensured an overall match with the original ambition for the initiative. ARTEMIS was, despite significant
improvements in the latest years of the programme, not successful in attracting the amount of funding
initially foreseen, in particular due to limited funding commitments from Member States. On project
level, 60% of ENIAC coordinators and 62.5% of ARTEMIS coordinators do not observe a difference
between costs budgeted and actual expenditure. Furthermore, 80% of ENIAC coordinators and
93.75% of ARTEMIS coordinators do not observe a difference between funding awarded and funding
received.
Mobilising and pooling efforts to increase investments
The evidences of the study allow for a positive overall evaluation of both ARTEMIS and ENIAC with
respect to the programmes’ response to the needs of the private members, as well as to the
needs of the Member States and Associated countries. The assessments of both JUs’
representatives are particularly positive as regards the coverage of needs from the private members.
Interviews as well as findings from the literature review confirm that this can be linked to a strong
involvement of the private side in reviewing and drafting the SRAs of both ENIAC and ARTEMIS.
Additionally, the survey to participants reaffirms this positive view also with respect to the coverage of
needs of the participating organisations.
The study findings result in a positive evaluation of the added value facilitated by ENIAC and
ARTEMIS. More than 95% of the representatives of the respective JU thought that the costs of
running the ENIAC programme have been justified at least to some extent by the added value
achieved. The most positively evaluated added value aspects are related to the JUs’ contributions to
“dialogue between researchers and industry” and “achievement of critical mass”. The assessments of
added value aspects of are in general somewhat more positive for ARTEMIS than ENIAC.
Qualitative evidence in terms of examples of increased investment have been identified to assess the
leverage effect. For ENIAC, investments related to the large scale pilot lines are reported most
frequently. Interviews point towards leverage effects in the following application and technology
domains: Sensors; heterogeneous integration; power electronics; as well as in modelling, design,
testing facilities at RTO/academic sites. Similarly for ARTEMIS, domains such as safety critical
systems, automotive, manufacturing, air crafting, telecommunications, industrial software and medical
technologies have benefitted from a leverage effect and facilitated investments.
In total 44% of the surveyed ENIAC and ARTEMIS participants assess that investments to their
organisation have been facilitated due to their participation in the projects funded by the JUs. The
corresponding percentage for ARTEMIS participants, 51%, is higher than for ENIAC participants, 36%.
Synergy and coordination of European R&D including Eureka
The evidence gathered in the study results in a rather weak overall evaluation of the synergies
achieved by ANIAC and ARTEMIS and the related EUREKA clusters. Stakeholders of both JUs
recognise that efforts have been made to increase the synergies, e.g. by joint development of the
research agendas and strategic documents, but overall the perceptions appear to be that more could
be done under ECSEL. The introduction of the ENIAC pilot lines and ARTEMIS AIPPs is assessed
positively for increased synergies in the respective field; and also for adding value e.g. in terms of
cross-border and interregional cooperation.

Final Study Report 7 of 150