ERIC EJ1155565: Determinants of Broader Impacts Activities: A Survey of NSF-Funded Investigators

2016·0.25 MB·English

by  ERIC

68 Nagy Nagy 69
Determinants of Broader Impacts Activities: between academia, industry, and others. According to Arden Bement, former director of the NSF,
“The [broader impacts] criterion was established to get scientists out of their ivory towers and
A Survey of NSF-funded Investigators
connect them to society” (quoted in Lok, 2010, p. 416).
A national review of the effectiveness of these criteria conducted in 2000 revealed that, while most
Dr. Dianne Nagy
researchers had little difficulty identifying the intellectual merit of their plans, many struggled
South Dakota State University
to adequately articulate the broader impacts of their proposed investigations (NSF, 2005). The
study also found that the broader impacts criterion was consistently weighted less than intellectual
Abstract: This study investigated the factors that shape the broader impacts activities of NSF
merit in the proposal review process and that many in the scientific community were resistant
grant recipients. A random sample of NSF grantees was surveyed about the type and quality of to or dismissive of the requirement (NSF, 2005). The NSF responded with efforts to educate
their broader impacts activities, their views on knowledge production and the democratization the scientific community about the agency’s rationale and expectations for broader impacts.
of science, their experience and training, and the existence of a supportive climate and A subsequent national evaluation of the NSF’s review criteria conducted in 2010 found that
resources for community engagement at their home institutions. Respondents indicated that problems with the execution, understanding, and acceptance of the broader impacts criterion
they shared an orientation towards knowledge production that was more democratic than persisted, that assessment was unclear and inconsistent, that there was little variety in the type of
activities performed to address the broader impacts criterion, and that principal investigators (PIs)
technocratic and valued public engagement in science; that they had adequate experience
needed greater institutional support to respond effectively to this requirement (NSB, 2011).
but little training in community engaged activities and lacked confidence in their ability
to evaluate such work; that executive leaders at their institutions encouraged community The purpose of this study was to investigate the factors that shape PI response to the NSF broader
engagement but promotion and tenure policies did not recognize such activities; and that impacts criterion. The following research questions guided the study:
they had little access to training, funding, or infrastructure to support community engaged 1. What are the different types of activity that PIs engage in to meet the broader impacts
activities. A multinomial logistic regression showed that faculty expertise, available resources, criterion of NSF-sponsored research?
and academic discipline were the strongest predictors of type of broader impacts activity 2. What is the quality of the broader impacts activities, as assessed by PIs using evaluation
(p < .001). A multiple regression analysis revealed that faculty expertise, a democratic criteria for the scholarship of engagement?
orientation towards knowledge production, and a supportive climate were the strongest
3. To what extent do PIs’ individual characteristics (expertise, epistemology, academic
predictors of quality of broader impacts activity (p < .001). discipline, and rank) and institutional support (climate and resources) for community
engagement predict the type of their broader impacts activities?
Keywords: Broader Impacts, Sponsored Research, Institutional Support, Community Engagement
4. To what extent do PIs’ individual characteristics and institutional support predict the
quality of their broader impacts activities?
Background and Objectives
The author gathered survey data to identify the types of broader impacts activities that PIs
An enduring image of academia is that of an ivory tower, disconnected from the messy problems of
conduct, and their perceptions of the quality of these activities. The author then determined how
the world. It is a recurring complaint that the primacy of basic research and the academy’s emphasis
much variance in type and quality of activity could be explained by the respondents’ personal
on abstract theory has eroded higher education’s connection to the world, isolating scholars from
characteristics and perceptions of institutional support for community engagement. In the
society and making their work obscure or irrelevant to the general public (Barker & Brown, 2009;
survey, community engagement was defined as “The collaboration between institutions of higher
Gibson, 2006). Decreasing appropriations for higher education heighten the need to convince
education and their larger communities (local, regional/state, national, global) for the mutually
the public about the value of university research. In response to prodding from Congress, federal
beneficial exchange of knowledge and resources in a context of partnership and reciprocity”
funding agencies are increasingly requiring academic research grant proposals to include indicators
(Carnegie Foundation for the Advancement of Teaching, n.d., n.p.).
of public impact that would result in social good.
In 1997, the National Science Foundation (NSF) established a policy that all funding proposals Conceptual Framework
submitted to the agency would be evaluated on two criteria: intellectual merit and broader
impacts. Broader impacts refer to “specific, desired societal outcomes” (NSF, 2012, p. III-2) such as This study is grounded in the theoretical framework of the scholarship of engagement. Such
the participation of underrepresented groups in science, technology, engineering, and mathematics scholarship is academically relevant faculty work that simultaneously meets campus goals and
(STEM); enhancing STEM education; public scientific literacy and engagement; and partnerships community needs, incorporates community issues, and is integrative across teaching, research,
The Journal of Research Administration, (47)1
70 Nagy Nagy 71
and service (Clearinghouse and National Review Board for the Scholarship of Engagement, research was the most essential form of scholarly activity, with publications and teaching flowing
2002). Although there are subtle distinctions between the terms “scholarship of engagement,” from it. He argued that “knowledge is not necessarily developed in such a linear manner. The arrow
“engaged scholarship,” and “community-engaged research,” they are used interchangeably of causality can, and frequently does, point in both directions. Theory surely leads to practice.
throughout this study. But practice also leads to theory” (1990, p. 16). The national dialogue that ensued prompted
academics to identify new models of knowledge creation, including that of use-inspired research
The NSF’s broader impacts merit review criterion has been an ongoing challenge within the
(Stokes, 1997) and Mode 2 science (Gibbons et al., 1994).
scientific community, particularly in terms of conceptual clarity, assessment, and philosophical
resistance. The persistence of the problem may stem, in part, from the paucity of literature on the Linear
broader impacts of academic research (Buxton, 2011). While scant research has been conducted
The dominant, linear model of knowledge production begins with basic research, which may lead
on such impacts, volumes have been written on the scholarship of engagement. The engagement
to applied research and technological development, and on to production or operations. This
movement has spawned multiple journals, institutes, conferences, and consortia dedicated to
model has shaped U.S. science policy since it was first proposed by Vannevar Bush in his 1945
studying engaged scholarship in its many forms. This study drew from this extensive literature
report to President Roosevelt (Mazuzan, 1994). Bush saw an inherent tension between the goals
base to better understand the issues that surround the broader impacts of academic research
of understanding and use, and warned that the creativity of basic research would be stifled by
and connections between the academy and society. Framing the study within the context of the
premature thoughts of practical use.
scholarship of engagement is appropriate not only because broader impacts activities may be viewed
as an expression of engaged scholarship, but also because both struggle to achieve legitimacy and
Use-inspired
support within the academy.
One year after Boyer’s call for engaged scholarship, Stokes challenged the traditional dichotomy
The Scholarship of Engagement
between basic and applied research. Stokes (1997) argued that Bush’s premise that flows between
science and technology are uniformly one way, from scientific discovery to technological innovation,
The concept of engaged scholarship is often credited to Ernest Boyer, a renowned leader of
was flawed and offered a limited understanding of how knowledge is generated and put to use.
educational reform. Boyer (1990) argued that the traditional definition of scholarship as research
Stokes contested Bush’s cannon that the goals of understanding and use are dichotomous, arguing
that advances disciplinary frontiers of knowledge was too limited. He countered the prevailing
that research is often influenced by both goals. In his model, theoretical and practical research and
hierarchical view of scholarship with a more inclusive vision that added to the ranks of conducting
application come together to create a dynamic cycle of innovation driven by changing conditions
original research such activities as identifying connections between concepts, bridging theory
and the competitive landscape.
and practice, and effectively communicating knowledge. Boyer posited that faculty scholarly
work included four “separate, yet overlapping, functions” (1990, p. 16) which he identified as the
Mode 2
scholarships of discovery, integration, application, and teaching. Boyer (1996) proposed the term
scholarship of engagement as interaction across these four realms to address community needs. He Gibbons and his co-authors (1994) offered another alternative to the static linear model of
described the scholarship of engagement as “connecting the rich resources of the university to the knowledge production. They determined that Bush’s old paradigm of discipline-specific,
most pressing social, civic, and ethical problems…creating a special climate in which the academic autonomous, university-grounded scientific discovery (“Mode 1”) was being supplanted by a new
and civic cultures communicate more continuously and more creatively with each other” (pp. 19- approach to knowledge production (“Mode 2”). Gibbons et al. posited that the new mode emerged
20). Boyer asserted that it was time for higher education to renew its covenant with society and to from a parallel expansion in the demand for specialist knowledge and the number of potential
partner with communities in directly addressing problems, with a reciprocal flow of knowledge. knowledge producers, adding government laboratories, industry, consultancies, and others to
the domain formerly ruled by universities. They described this new mode as socially distributed,
Models of Knowledge Production application-oriented, trans-disciplinary, and subject to multiple accountabilities. This evolution,
they argued, occurred in response to significant trends changing the research environment: the
Epistemologies about knowledge production and distribution shape perceptions about the role steering of research priorities, the commercialization of research, and the accountability of science.
of higher education in society, define relationships between campus and community, and govern As a result, the research process was no longer an objective investigation, but rather an intense
the values, norms, and practices of scholars. Boyer raised critical questions of how knowledge is dialogue between research actors and subjects.
constructed and what is accepted as legitimate knowledge in the academy. Deeply embedded in
academia is the traditional linear model of knowledge production and distribution, which prizes Reciprocal
basic research that is disciplinary, removed from influence by outside interests, and conducted
In the dominant, technocratic culture of higher education, knowledge flows in one direction: from
without consideration of use (Bush, 1945). Boyer countered the prevailing belief that basic
credentialed, detached experts ensconced in the university to its place of need and application in
The Journal of Research Administration, (47)1
72 Nagy Nagy 73
the community (Hartley, Saltmarsh, & Clayton, 2010). The scholarship of engagement challenges possible associations between engagement in public scholarship and individual characteristics such
the belief that knowledge is produced by academics and then transferred to the community. Instead, as epistemology (Austin & Beck, 2010) and expertise (Baskurt, 2011), as well as demographic
engaged scholarship builds reciprocal relationships between scholars and community partners, factors including discipline (Lunsford & Omae, 2011) and rank (Glass, Doberneck, & Schweitzer,
dynamic synergies between theory and practice, and research that benefits both the community 2011). The literature also indicates that institutions influence individual behavior by means of
and the academy (Nicoreta, Cutforth, Fretz, & Thompson, 2011). Such reciprocity has been overall climate (Vogelgesang, Denson, & Jayakumar, 2010) and the allocation of resources for
defined as “democratic engagement” and described as “an epistemological shift…that favors mutual administrative support, faculty development, and incentives (Doberneck, Brown, & Allen, 2010).

deference between lay persons and academics” (Hartley, Saltmarsh, & Clayton, 2010, p. 401). Figure 2 outlines the variables investigated in this study.
Study Model Rhetoric

The study’s conceptual model, depicted in Figure 1, illustrates the congruence of NSF goals and
Climate
  Recogni6on

broader impacts activities (bulleted lists) with Boyer’s domains of scholarship and the models of
knowledge production described above. The model aligns these elements along the oppositional
axes of democratic versus technocratic orientations and the goals of improving understanding Rewards

versus improving technology. Ins6tu6onal
 Support

Administra6ve

support

Democratic
 Orientation

••  LImeaprrno
 vther
 SoTuEgMh

 aesdsuecsasm6oenn
 t
  ••  LIninckre
 raessee
 aprucbhl
 iacn
 edn
 geadguecma6eonnt

  Resources
  Faculty
 development

• Increase
 scien6fic
 literacy
  • Build
 partnerships

• Broadly
 disseminate
 results
  • Expand
 par6cipa6on
 of

underrepresented
 groups

Incen6ves

Broader
 Impacts
 Ac6vi6es

Teaching
  Integra6on
  Type
 &
 Quality
  Knowledge

produc6on

(Reciprocal)
  (Mode
 2)

Epistemology

Democra6za6on
 of

science

Improve
 Understanding
  The
 Scholarship
 of
 Engagement
  Improve
 Technology

Experience

Discovery
  Applica6on

(Linear)
  (Use-­‐Inspired)
  Individual

Exper6se
  Training

Characteris6cs

• Advance
 fron6ers
 of

• kEnnoawbllee
 dbgreea
 kthroughs
  ••  DExepvaenlodp
 n
 SaT6EoMna
 wl
 soerckuforirtcye

  Ability

• Nurture
 excellence
 in
 basic
  • Increase
 economic

research
  compe66veness

• Enhance
 infrastructure

Rank

Technocratic
 Orientation

Demographics

Figure 1F:i gAurlieg 1n.m Aelingtn bmeetwnte beent wNeSeFn NgoSaFl sg,o Balosy, Bero’yse cr’os ncocenpcet potf oefn egnaggaegded s cschhoolalarrsshhiipp,, a nd models of Discipline

knowledge production. and models of knowledge production.

Figure 2. Variables that shape broader impacts activities of academic faculty.
Figure 2: Variables that shape broader impacts activities of academic faculty.
Within the context of this conceptual framework, the author explored the relationships between

the individual characteristics of faculty members, support at their home institutions, and the type
and quality of their broader impacts activities. The scholarship of engagement literature suggests
The Journal of Research Administration, (47)1
25
24
74 Nagy Nagy 75
Methods The next step in survey development was a pre-survey evaluation conducted by a panel of field
experts in order to identify and correct any foreseeable problems prior to executing the survey.
Population The panel members were selected based on their expertise in engaged scholarship, survey design,
and sponsored research. The panel provided input on the conceptual accuracy of the items, the
The population for this study was the 3,635 faculty members at institutions of higher education
length and legibility of the scale, and any missing or misrepresented dimensions of the constructs
across the United States who received NSF research grants during Fiscal Year 2009 (October 1,
to be studied. After integrating feedback from the expert panel into the instrument, the survey
2008 – September 30, 2009). This period was selected because it not only follows on the heels
received an exempt review and approval from the Institutional Review Board of the University
of the NSF’s 2007 release of broader impacts criterion guidelines, but it also allows sufficient
of South Dakota.
time for the completion of most projects. Research grant is a term used by the NSF to represent
what may be considered a typical research award, particularly with respect to the award size (NSF,
Data Collection
2009). Education research grants are included in this category. Excluded are large awards such
as centers and facilities, as are equipment and instrumentation grants. Also excluded are grants This cross-sectional study used an online survey to collect data from faculty at academic institutions
for conferences and symposia, grants in the Small Business Innovation Research program, Small across the United States. The primary concern about web-based surveys is that they may introduce
Grants for Exploratory Research, and education and training grants. EAGER and RAPID grants error if online use is not universal. This concern does not apply to the study at hand because the
were also excluded as they are exempt from external review. sample population, university faculty, typically demonstrate competency in the use of email and
other forms of online communication (Shannon, Johnson, Searcy, & Lott, 2002). Consequently,
Sample given considerations of convenience and economy, a web-based survey was deemed the most
expedient approach to data collection.
A stratified random sample of 700 grant recipients was drawn from the population. This sample
size was chosen to provide a pool of respondents large enough to restrict error to 5%, assuming a The survey was formatted online using the survey software QuestionPro. The researcher collected
response rate of approximately 50% (Patten, 2002). A stratified random sample design was selected email addresses for members of the sample from the NSF website and sent each member an email
to reduce normal sample variation and to produce a sample more likely to resemble the total message explaining the project, inviting their participation, and directing them to the survey
population than a simple random sample. Given the importance attributed to academic discipline web link. The survey was distributed on February 4, 2013, and remained open through March 1,
in the literature, the sample was stratified along NSF Directorates to provide representation across 2013. Follow-up with non-respondents consisted of reminder emails. All sample non-respondents
a broad spectrum of disciplines. One hundred grant recipients were selected from each of the were sent a first, second, and final email reminder after weeks one, two, and three, respectively.
following directorates: Biological Sciences; Computer and Information Science and Engineering; QuestionPro’s Respondent Anonymity Assurance was enabled on the survey to ensure that,
Education and Human Resources; Engineering; Geosciences; Mathematics and Physical Sciences; although computer generated identification numbers for individuals were created, the researcher
and Social, Behavioral, and Economic Sciences. The population of grant recipients in each did not have simultaneous access to both a respondent’s email address and response data.
directorate was assigned consecutive numbers and the desired sample was drawn using a table of
random numbers. Data Analysis
Instrument The Statistical Package for the Social Sciences was used for all data analyses and the .05 level of
significance was used for all inferential statistics.
As no instrument existed to collect the data necessary for this study, the author developed a special
purpose survey derived from an extensive review of the literature. The survey items generated Research question 1. What are the different types of activity that PIs engage in to meet
fell into five subsections: broader impacts activities, epistemology, expertise, institutional the broader impacts criterion of NSF-sponsored research? Type of broader impacts activity
support, and demographics. The first section asked participants about the type and quality of was a categorical variable. The categories, identified by the NSF, were education, broadening
their broader impacts activities and community partnerships. The second section attempted to participation, infrastructure, dissemination, and other. Frequencies of responses to the question
capture participants’ epistemologies by inquiring about their views on knowledge production about type of activity were tabulated to derive percentages of participation in each activity.
and the democratization of science. The section on expertise asked participants about their prior
Research question 2. What is the quality of the broader impacts activities, as assessed by
experience, training, and sense of ability with activities similar to those conducted to address the
PIs using evaluation criteria for the scholarship of engagement? Quality of broader impacts
broader impacts criterion, both as graduate students and as faculty members. The next section asked
activity was a summed score of responses to the questions about each area of assessment (goals,
participants about the existence of a supportive climate and resources for community engagement
preparation, methods, results, presentation, critique, and ethics of the activity, as well as the
at their home institutions. The final section of the survey gathered demographic information on
reciprocity, communication, structures, and duration of community partnerships). Means
academic discipline and rank. The questions on discipline, rank, and activity type were multiple
and standard deviations generated from these items, and composite means for each element,
choice; the remaining items used a five-point scale, ranging from 1 (not at all) to 5 (fully).
The Journal of Research Administration, (47)1
76 Nagy Nagy 77
provided information related to the areas of relative strength and weakness of broader Demographics
impacts activities.
Of the 124 respondents who completed the survey, half held the academic rank of full professor
Research question 3. To what extent do PIs’ individual characteristics (expertise, (n = 62, 50.0%), while the bulk of the rest were associate professors (n = 40, 32.3%), with some
epistemology, academic discipline, and rank) and institutional support (climate and resources) assistant professors (n = 14, 11.3%). Table 1 reports the frequencies and percentages of the
for community engagement predict the type of their broader impacts activities? Expertise was academic rank categories.
a summed score of questions about the respondent’s prior experience, training, and sense
of ability related to community-engaged activities. Epistemology was a summed score of
Table 1. Frequencies and Percentages of Academic Rank
questions about the respondent’s views on knowledge production and the democratization
of science. Responses were grouped into two categories: democratic and technocratic Academic Rank n %
orientations. Climate was a summed score of questions about the respondent’s perceptions of Full professor 62 50.4
institutional climate as expressed through rhetoric, recognition, and rewards for community
Associate professor 40 32.5
engagement. Resources was a summed score of questions about the respondent’s perceptions
Assistant professor 14 11.4
of the administrative support, faculty development, and incentives available for community
Research scientist/professor 5 4.1
engagement. Academic discipline was operationalized as the NSF directorate to which the
Other 2 1.6
respondent applied. Academic rank was captured with dummy variables for assistant professor,
associate professor, full professor, research scientist, and other. Note: n = 123; one response was missing.
After reverse coding negatively worded items and removing cases with missing observations
The degree to which the sample is representative of the population of NSF grantees is unknown
on multiple indicators, a multiple logistic regression was conducted to determine how the
because the NSF does not track the academic rank of their grantees. The NSF does, however,
variables ranked in terms of highest to lowest when it came to predicting the type of broader
categorize their grantees as either “Early Career PIs” (investigators within seven years of receiving
impacts activity.
their last degree) or “Later Career PIs” (those who received their last degree more than seven years
Research question 4. To what extent do PIs’ individual characteristics and institutional before the award date). In FY 2009, 78% of NSF grantees were classified as Later Career PIs; 22%
support predict the quality of their broader impacts activities? Quality of activity was a were Early Career PIs (NSB, 2010). The fact that the promotion from Assistant to Associate
summed score of assessment questions based on the scholarship of engagement criteria. The Professor generally occurs within six years on the tenure track suggests that the distribution of the
independent variables were the constructs identified in the analysis of research question 3 sample is reasonably representative of the population.
(individual characteristics and institutional support). Because the dependent variable was
The respondents were fairly evenly distributed across the seven NSF directorates, as Table 2 depicts.
interval-scaled, a multiple regression analysis was employed. A stepwise multiple regression
The Directorate of Biological Sciences had the largest representation (n = 27, 21.8%); the next five
determined how much of the variance in quality of broader impacts activity scores could be
directorates were closely clustered, with the number of respondents ranging from 19 (15.3%) to 15
explained by the respondents’ expertise, epistemology, rank, discipline, institutional climate,
(12.1%). The Directorate of Geosciences had the fewest respondents (n = 11, 8.9%).
and resources.
Table 2. Frequencies and Percentages of NSF Directorates
Results
NSF Directorate n %
Response Rate
Biological sciences 27 21.8
The rate of response to the study’s survey was low. Of the 700 surveys distributed, 15 were mail Mathematics and physical sciences 19 15.3
returns, 246 were viewed, 209 were started, and 124 were completed. The completed response Social behavioral and economic sciences 18 14.5
rate was 18.1%. Although the limited number of cases restricted the options for statistical analysis, Computer science and engineering 16 12.9
the low response rate did not necessarily invalidate the survey. An examination of the results of Engineering 16 12.9
81 national surveys with response rates ranging from 5% to 54% revealed that surveys with much Education and human resources 15 12.1
lower response rates were only minimally less accurate (Holbrook, Krosnick, & Pfent, 2007).
Geosciences 11 8.9
Note: n = 122; two responses were missing.
The Journal of Research Administration, (47)1
78 Nagy Nagy 79
Type of Broader Impacts Activity other group). These respondents assessed their community partnerships in terms of reciprocity,
communication, structures, and duration of the collaboration. The scoring for this area of
Research question 1 asked about the different types of activity that PIs engage in. The categories
assessment was noticeably lower than the others (composite M = 3.07, SD = 1.07). The highest
of activity, identified by the NSF, were education, broadening participation, infrastructure,
scoring items were “Communications between the community partner and I were ongoing and
dissemination, and other (NSF, 2008a). Frequencies of responses to the question about type of
bidirectional” (M = 3.53, SD = 1.13) and “The relationship continued after the project ended”
activity were tabulated to derive percentages of participation in each activity. Education was by
(M = 3.47, SD = 1.43). The items receiving the lowest ratings were “A formal agreement outlined
far the most common activity (n = 71, 57.3%), while dissemination was the second most reported
roles and responsibilities” (M = 2.55, SD = 1.33) and “The community partner and I shared
activity (n = 29, 23.4%). The remaining categories were substantially lower. Descriptive statistics
power and responsibility equitably” (M = 2.74, SD = 1.25).
generated from this item are presented in Table 3.
Predictors
Table 3. Frequencies and Percentages of Type of Broader Impacts Activity
The literature suggests possible associations between engagement in public scholarship and
Type n % personal and institutional characteristics. This study investigated these variables as predictors of
type and quality of broader impacts activities.
Education 62 57.3
Dissemination 29 23.4 Expertise. This variable was a summed score of questions about the respondents’ prior
Broadening participation 12 9.7 experience, training, and sense of ability related to community-engaged activities. Respondents
Infrastructure 9 7.3 indicated that they had a good deal of prior experience with activities similar to their broader
Other 3 2.4 impacts work as faculty members (M = 4.01, SD = 1.01), but they had received little faculty
training for such work (M = 2.72, SD = 1.22). As graduate students, they received even less
training (M = 2.39, SD = 1.35) and had less experience (M = 2.78, SD = 1.39). Respondents
Quality of Broader Impacts Activities
reported that they had a relatively strong sense of ability to implement broader impacts
Respondents assessed the quality of their broader impacts activities in terms of goals, preparation, activities (M = 3.75, SD = 0.90), but they were less confident in their ability to evaluate these
methods, results, presentation, critique, and ethics of the activity. The response scale was 1 (not at activities (M = 3.27, SD = 1.09).
all) to 5 (fully). Respondents expressed the greatest confidence in their selection and use of methods Epistemology. This variable was a summed score of questions about the respondents’
(composite M = 4.18, SD = 0.64). High scoring items included “I used methods appropriate to
views on knowledge production and the democratization of science. Responses were
my goals” (M = 4.36, SD = 0.65) and “I effectively applied the methods selected” (M = 4.11,
grouped into two categories: democratic and technocratic orientations. A technocratic
SD = 0.79). Scores were also relatively high for preparation (composite M = 4.16, SD = 0.82),
orientation is characterized by the valuing of basic, discipline-specific, curiosity-driven
with the leading item being “I brought together the skills and resources necessary to move the
research conducted exclusively by highly specialized academic experts. A democratic
project forward” (M = 4.30, SD = 0.74). Interestingly, the area of ethics included both the highest
orientation values applied, trans-disciplinary, problem-centered, use-inspired research that
and lowest scoring items: “I followed all policies concerning the responsible conduct of research”
welcomes collaboration with partners outside the academy. Survey responses indicated
(M = 4.61, SD = 0.74) and “I subjected my activity to university IRB review” (M = 3.10,
orientations that were more democratic (M = 3.22, SD = 0.68) than technocratic
SD = 1.82).
(M = 2.85, SD = 0.68). Respondents tended to view public engagement in science as a
way to increase acceptance of the results and generate information that meets user needs
The area that showed the greatest weakness was critique (composite M = 3.49, SD = 0.98). The
lowest scoring items in this category were “I used an appropriate breadth of evidence for the (M = 3.85, SD = 0.93), and to take into account public values and contexts as well as local
critique” (M = 3.39, SD = 1.11) and “I used this evaluation to improve the quality of future work” expertise (M = 3.09, SD = 1.13). Respondents did not agree that public ignorance should
be corrected through a one-way flow of information from the academy to the community
(M = 3.51, SD = 1.01). Respondents also rated themselves lower in using the results (composite
M = 3.87, SD = 0.80). The items “I presented the results to both academic and non-academic (M = 2.17, SD = 1.01), or that scientific research would be most productive if left to scientists
audiences” and “The results of the activity added consequently to the field” received the relatively alone (M = 2.49, SD = 1.21).
low mean scores of 3.73 (SD = 1.11) and 3.76 (SD = 0.97), respectively. Climate. This variable was a summed score of questions about the respondents’ perceptions of
institutional climate as expressed through rhetoric, recognition, and rewards for community
Thirty-eight respondents (30.6%) reported that their broader impacts activities involved
collaboration with a specific community agency (such as a school, museum, organization, or engagement. The overall mean for this variable was moderate, at 3.02 (SD = 0.79). Respondents
The Journal of Research Administration, (47)1
80 Nagy Nagy 81
indicated that the executive leadership at their institutions explicitly promoted community The stepwise model revealed that expertise accounted for 20% of the variance, and that the
engagement as a priority (M = 3.42, SD = 1.11), but their promotion and tenure policies did inclusion of climate into the model resulted in an additional 8% of the variance being explained.
not reward such research (M = 2.15, SD = 0.93). Table 5 presents indices of the relative strength of each predictor. These results suggest that faculty
expertise is the greatest predictor of the quality of broader impacts activities. An institutional climate
Resources. This variable was a summed score of questions about the respondents’
supportive of community engagement and a democratic attitude toward knowledge production
perceptions of the administrative support, faculty development, and incentives available
are also predictors of high quality broader impacts activities. The results also indicate that full
for community engagement. The overall mean for this item was low, at 2.31 (SD = 0.82).
professors are more likely to report high quality broader impacts activities, while researchers who
Respondents reported that faculty training was not available in engaged scholarship
submit to the NSF Directorate of Biological Sciences are less likely to do so.
(M = 2.19, SD = 1.04) and that there was little financial support (M = 2.20, SD = 1.10) or
infrastructure to facilitate collaboration (M = 2.38, SD = 1.07) and assessment (M = 2.40,
Table 5. Strength of Predictor Variables on Quality in Stepwise Regression
SD = 1.06) of community engaged activities.
Predictor Variable Beta n
Influence on Type of Activity
Expertise .312 .000*
A multinomial logistic regression analysis was conducted to predict the type of broader impacts
Biology -.256 .002*
activity the respondents engaged in, using expertise, epistemology, institutional support, discipline,
Democratic .247 .003*
and rank as predictors. A test of the full model against a constant only model was significant,
Climate .236 .005*
indicating that the predictors, as a set, reliably distinguished between types of broader impacts
Full professor .199 .016*
activity, χ²(52, n = 102) = 87.61, p = .001. A Nagelkerke’s R² of .644 indicated a moderately
strong relationship between prediction and grouping. The model’s overall prediction success was * Significant predictors at .05.
68.3%. The likelihood ratio tests demonstrated that expertise, resources, and discipline contributed
significantly to the model; expertise emerged as the strongest predictor (based on p value). Table 4 Open-ended Comments
presents the contribution of each variable to the model.
The survey concluded with an invitation for participants who wished to comment on the survey to
do so in an open text box. Feedback from participants covered the following topics: appreciation
Table 4. Likelihood Ratio Test of Each Predictor’s Contribution to the Model
for the study (4); suggestions to improve the study by clarifying the language (5) and allowing
respondents to select multiple options (4); and complaints about the broader impacts criterion in
Predictor χ² df p
terms of how the NSF manages it (2), how researchers respond to it (2), and the need for support
Expertise 22.61 4 .000*
(5). The strongest feedback focused on the need for support with broader impacts activities. This
Resources 17.28 4 .002*
concern was expressed both in the comment box as well as through email responses to the author
Discipline 40.67 24 .018*
from sample members who chose not to complete the survey.
Rank 14.11 8 .079
Technocratic 5.13 4 .274
Discussion
Democratic 0.67 4 .956
Climate 1.30 4 .861 Personal Characteristics
* Significant predictors at .05.
All of the individual traits postulated in the literature to influence faculty community engagement
examined in this study emerged, to varying degrees, as determinants of the broader impacts
Influence on Quality of Activity activities of academic faculty.
A multiple regression analysis revealed that the linear combination of individual characteristics Expertise. Broader impacts activities generally push investigators beyond their particular
and institutional support was significantly related to quality, F(15,84) = 5.16, p = .000. The sample areas of specialization and require additional skill sets. Many scientists believe they do not
multiple correlation coefficient (R²) was .70, indicating that approximately 49% of the variance have the skills necessary to develop appropriate outreach activities (Lok, 2010; Mathieu,
in quality could be accounted for by the linear combination of individual characteristics and Pfund, & Gillian-Daniel, 2009; Tretkoff, 2007). This theme pervading the literature was
institutional support. A stepwise multiple regression presented a more parsimonious model with supported by the study’s finding that respondents had received little training to develop and
an equally significant regression equation, F(5,90) = 14.89, p = .000, and an adjusted R² of .42.
The Journal of Research Administration, (47)1
82 Nagy Nagy 83
implement work similar to their broader impacts activities. Sample members in the survey and needs, accounts for public values and knowledge, and improves the quality of science policy
email responses to the researcher also reiterated this belief. decisions. Participants shared an orientation towards knowledge production that was more
democratic than technocratic. Yet, despite this appreciation for the socially embedded context
The data analysis revealed that expertise was the strongest predictor of the type and quality
of application, respondents favored research that was curiosity-driven rather than use-inspired.
of broader impacts activities. Because this variable emerged as such a significant predictor,
The apparent incongruity between respondents’ valuing of public engagement to increase the
the author repeated the analysis with a more conservative computation of faculty expertise
acceptance of results and meet user needs, while favoring research that is curiosity-driven rather
by removing the items referring to prior experience and training as a graduate student. This
than use-inspired, may indicate an emerging culture shift. Researchers appear to be embracing
narrower perspective on expertise did not diminish the significance of this variable in any of
an alternative value system that acknowledges the need for public engagement while clinging to
the regression analyses. The survey findings and supplemental responses underscore the need
traditional norms regarding what constitutes proper academic science.
for faculty development and training in broader impacts, as well as institutional support to
assist with implementation and evaluation. A similar dynamic tension was evident in questions about how knowledge is generated. Survey
respondents indicated a strong belief that knowledge is generated through trans-disciplinary
Epistemology. The influence of epistemology was manifested in this study in more than one
collaboration that brings together multiple sources of distributed knowledge, a view characteristic
way. Negative bias in the scientific community against the broader impacts criterion appears
of a democratic orientation. Interestingly, they also expressed the concurrent belief that knowledge
to have limited the survey response rate. Several members of the sample opted to reply to the
is generated through objective disciplinary expert-led investigation, suggesting they did not
author with negative comments rather than complete the survey. The following example of a
perceive these opposing approaches to knowledge production as dichotomous.
response to the author’s email invitation to complete the survey provides anecdotal evidence
of persistent resistance to the broader impacts criterion: Rank. While prior research suggested a significant relationship between an investigator’s
professional career status and engagement in community-based research, the direction of
“Most everyone I know that submits to NSF *HATES* to come up with arguments
the relationship was inconclusive. It has been reported that the types of publicly engaged
about broader impacts for NSF proposals. They hate doing it, they hate thinking
scholarship faculty members pursued varied by rank in statistically significant ways, with
they had to do it, etc. So, when a request like yours comes in, most folks can’t delete
full professors more likely than associate professors to report such work (Glass, Doberneck,
your unwelcome spam-like message fast enough.”
& Schweitzer, 2011); that lower ranked faculty were more frequently involved in public
The literature reviewed documents with similar negative responses from researchers, describing service (Antonio, Astin, & Cress, 2000); that faculty members of various ranks reported
the broader impacts criterion as burdensome, counterproductive, and punitive (Tretkoff, 2007); using scholarship to address community needs at similar levels (Vogelgesang, Denson, &
confusing and frustrating (Sarewitz, 2011); irrelevant, ambiguous, and poorly worded (Holbrook Jayakumar, 2010); and that tenured and untenured faculty showed similar levels of frequency
& Frodeman, 2005); and undermining the autonomy of the scientific enterprise (Holbrook & in conducting the scholarship of engagement (Braxton & Luckey, 2010). Some argue that
Frodeman, 2011). junior faculty trained in an environment where engaged scholarship is increasingly viewed as
an important part of the mission of modern universities are more likely to engage than senior
Some of this resistance stems from the fact that the revisions to the merit criteria had little impetus
faculty who were not trained in such an environment (Jaeger & Thornton, 2006). Others
from the scientific community, but were instituted in response to Congressional prodding (Bozeman
contend that successful senior scholars with tenure have the flexibility and freedom to pursue
& Boardman, 2009). The broader impacts criterion is interpreted by some as an introduction of
non-traditional research (Scott, 2007). One finding that is consistently reported is that junior
extraneous political, cultural, or economic concerns into basic research (Holbrook & Frodeman,
faculty members are routinely counseled to avoid service activity that may interfere with their
2007); bringing considerations external to science into the peer review process (Holbrook, 2005),
research productivity (Stanton & Wagner, 2010).
and politicizing the value-neutral pursuit of science (Holbrook & Frodeman, 2007). Holbrook
and Frodeman (2007) suggested that agitation over the broader impacts criterion is to be expected This study found that academic rank was a predictor of the quality of broader impacts activities.
from anyone who believes that beneficial societal impacts will eventually follow from a laissez The finding that being a full professor was a significant predictor of quality aligns with reports
faire approach to basic research. Thus, reaction to the broader impacts criterion may depend on in the literature that full professors are more likely than associate professors to conduct
one’s conception of knowledge production, one’s views on the democratization of science, and the community engaged work (Glass, Doberneck, & Schweitzer, 2011) and that successful senior
degree to which one supports the Bush model of opposition between basic and applied research. scholars with tenure have the flexibility and freedom to pursue non-traditional research
(Scott, 2007).
Survey responses revealed that epistemology shaped participants’ response to the broader impacts
criterion, with a democratic orientation towards knowledge production emerging as a predictor Discipline. The literature suggests that the ways in which faculty members define, design,
of the quality of broader impacts activities. A democratic orientation recognizes that public and enact their community-engaged scholarship are largely driven by discipline (Doberneck,
engagement increases the acceptance of research results, generates information that meets user Glass, & Schweitzer, 2010). This aligns well with the study’s finding that discipline contributed
The Journal of Research Administration, (47)1
84 Nagy Nagy 85
significantly to the predictive model for type of broader impacts activity. In this study, Type of Broader Impacts Activity
discipline was operationalized as the NSF directorate to which the researcher applied. The
NSF guidelines to proposers and reviewers identified general areas of broader impact: education,
results revealed that researchers who submitted to the NSF Directorate of Biological Sciences
broadening participation, enhancement of infrastructure, broad dissemination, and other social
were less likely to report high quality broader impacts activities. This supports various reports
benefit (NSF, 2008a). Despite admonitions from the NSF that PIs “are expected to go beyond their
in the literature that community engagement was less prevalent in the physical sciences
normal teaching duties and faculty commitments” (NSF, 2008b, p. 1), the category of education
(Lunsford & Omae, 2011; Vogelgesang, Denson, & Jayakumar, 2010).
is by far the most common broader impacts activity, often with researchers simply repackaging
Institutional Support what they are already doing (Lok, 2010). The chemistry division director at the NSF reported that
“Overwhelmingly, the number one broader impacts that most people in the chemistry division
One of the key predictors of engaged scholarship is institutional practices and policies that support
are using is ‘training graduate students and postdocs’” (Lok, 2010, p. 417). In 2010, the National
engagement (Lunsford, Bargerstock, & Greasley, 2010). Faculty motivation to engage in public
Science Board (NSB), the agency charged with NSF oversight, commissioned a topic modeling
scholarship is influenced by institutional characteristics including mission, resources, norms,
of the project summaries of approximately 150,000 NSF proposals. The analysis revealed that the
and evaluation policies (Colbeck & Wharton-Michael, 2006). Some theorists have noted that
education category occurred in more than 60% of the proposals (NSB, 2011). This was roughly
institutional characteristics have a greater impact than personal motivation and commitment
three times the size of the next largest category. The results of the present study were congruent
when it comes to scholarly productivity, and that faculty productivity may be stimulated or stifled
with the topic modeling; education was by far the most commonly reported activity at 57.3%.
by organizational culture and administrative structure (Freedenthal, Potter, & Grinstein-Weiss,
2008). A study based on the Higher Education Research Institute’s 2004-2005 national survey of The America COMPETES Reauthorization Act of 2010 (P.L. 111-358), which provided
college and university faculty revealed that the perception of institutional support matters, even reauthorization for the NSF, established new goals and policies for the broader impacts criterion.
above and beyond the individual dispositions of faculty members, and even when disciplinary The Act stipulated that the NSF must apply this criterion to achieve an array of societal goals,
culture is accounted for. Every one-step increase in perceived institutional support was associated including increasing economic competitiveness; developing a globally competitive workforce
with a 10% increase in the likelihood of a faculty member collaborating with the local community in science, technology, engineering, and math (STEM); expanding the participation of women
in research or teaching (Vogelgesang, Denson, & Jayakumar 2010). Institutional support can be and underrepresented minorities in science and engineering; increasing partnerships between
conveyed through both climate and resource allocation. academia, industry, and others; improving STEM education at PK-12 and undergraduate levels,
including teacher development; increasing public scientific literacy; and expanding national
Climate. Campus climate is a function of interpersonal interactions and includes dimensions
security (Sec. 526[a][1]-[8]). The NSF integrated this expanded vision of broader impacts into a
such as “perceptions, attitudes, and expectations” (Cress, 2002, p. 390). A multiple regression
new Proposal and Award Policies and Procedures Guide (NSF, 2012). Congress’ identification of
analysis revealed that an institutional climate supportive of community engagement is a
eight national goals to which broader impacts activities should be directed underscores the need
strong predictor of high quality broader impacts activities, accounting for 8% of the variance
for greater variation in the types of broader impacts activities PIs pursue. The results of this study
explained by the model. The literature identified the university reward system as the greatest
suggest that limited resources and training constrain researchers to engaging in broader impacts
obstacle to community engaged research (Pfirman, Collins, Lowes, & Michaels, 2005).
activities that are the most familiar and least demanding (i.e. education).
Responses to the author’s survey concurred that promotion and tenure policies were the
feature of institutional climate least supportive of community-based research. Quality of Broader Impacts Activity
Resources. While faculty recognition and rewards are important motivators to initiate Hampering effective peer review of broader impacts is the fact that no standards have been set and
change, organizational structures must be in place to sustain the work (Brukardt, Holland, no evaluation target provided (Bozeman & Boardman, 2009). The NSF does not systematically
Percy, & Zimpher, 2004; Stanton, 2007). When the NSF asked the scientific community track how its broader impacts requirements are being met, nor does it have a system in place to
what role their institutions should play to ensure that broader impacts are realized, the evaluate project effectiveness (Lok, 2010). The first comprehensive analysis of the NSF’s review
majority response was that home institutions should increase support for broader impacts criteria, conducted in 2000, recommended that the NSF develop an evaluation strategy based
by establishing mechanisms for those activities, increasing financial or in-kind support, or on measures and performance indicators to track the objectives and implementation of the new
encouraging involvement through changes in value systems and incentives (NSB, 2011). criteria supported by both qualitative and statistical data collection methods capable of measuring
The allocation of resources for administrative support, faculty development, and incentives incremental movement towards achieving the NSF’s strategic goals (National Academy of Public
is the second largest barrier to engaged scholarship (Pfirman et al., 2005). Although resources Administration, 2001). One of the six major themes of the second national review of the NSF
emerged as a strong predictor of type of broader impacts activity, participants reported limited criteria, conducted 10 years later, was that post-award assessment of broader impacts activities was
access to training or infrastructure to facilitate collaboration and assessment of community- weak and should be improved (NSB, 2011). Given the variation in size and scope of projects, the
engaged activities.
The Journal of Research Administration, (47)1
86 Nagy Nagy 87
report indicated that the effectiveness of broader impacts would be more meaningful if they were their professional identities, extending training to graduate students and postdoctoral researchers
aggregated at a higher level than the individual project. will help create a cadre of professionals who conduct high quality broader impacts activities as a
natural part of their research programs.
Without established metrics and data collection, it is impossible to objectively assess the broader
impacts of sponsored programs. Instead, this study asked PIs to evaluate the processes with which While there is clearly a need for training, faculty may be reluctant to pursue it. This disinclination
they devised and implemented their broader impacts activities using the standards of scholarship was expressed both via email responses and through the survey’s open comment box. Participants
(clear goals, adequate preparation, appropriate methods, significant results, effective presentation, asserted that they lack both the time and desire to learn how to implement broader impacts
reflective critique, and ethical conduct). Given the subjectivity of self-assessment, better objective activities; that it did not make sense to require scientists to do such work; that trained experts are
metrics of the quality of broader impacts activities are clearly needed. The criteria used in this better suited to public outreach than research scientists; and that it would be preferable to partner
study appeared to engender a social desirability bias. Respondents showed little variation in their with experts in broader impacts than to learn how to conduct such activities. These comments
high ratings of the quality of their broader impact activities, presumably because they wanted to point to the need for campuses to provide professional staff trained in implementing broader
believe that they were practicing good scholarship. It is likely that those who reported working impacts activities to collaborate with and support researchers.
with a specific community agency were more critical of these partnerships because such activity
Survey respondents reported limited infrastructure available to assist with their broader impacts
is traditionally less highly regarded in the academy and thus subject to a more frank assessment.
work. Existing service-learning and civic engagement centers, or other campus-wide coordinating
Identifying valid and feasible ways to assess research impact more objectively should be a priority
units, can fill that gap by providing researchers with practical assistance; links to community
of future studies. Self-reported information needs to be balanced with multiple kinds and sources
partners; information about established, well-aligned projects with which to connect; and a venue
of data. An important contribution to the field would be the development of multiple well-defined
for interdisciplinary partnerships. Technical assistance would be especially important in assessment
measures, consisting of both hard and soft data, which can be systematically analyzed.
and evaluation of the outcomes of broader impacts activities.
Implications Beyond investing in training and infrastructure, institutions can also enhance and expand the
quality and type of their broader impacts activities by providing incentives (such as stipends,
This study makes several contributions to the field of research administration. By providing a
release time, and seed grants) for faculty to build engagement into their work. An institutional
snapshot of the type and quality of broader impacts activities conducted by PIs, it enhances
climate supportive of community engagement was a strong predictor of high quality broader
understanding of the training and support faculty members need to successfully meet the broader
impacts activities. An institution’s climate is shaped by values expressed through rhetoric, public
impacts requirements of federally sponsored research. In addition to data collection, the study
recognition, and reward systems. Respondents indicated that the executive leadership at their
examined the associations between relevant personal, professional, and institutional factors,
institutions explicitly promoted community engagement as a priority, but their promotion and
and broader impacts activities, identifying patterns and highlighting barriers and facilitators to
tenure policies did not recognize anything other than peer-reviewed publications and grants
guide the decision making of higher education administration as well as federal funding agencies
as measures of productivity. If researchers are to be motivated to conduct high quality work in
seeking to expand or enhance the broader impacts of university research. This study was designed
the community, tenure and promotion guidelines must be revised to ensure that they receive
to stimulate discussion among higher education administrators about community engagement,
professional recognition and advancement for engaged scholarship.
inform initiatives to expand interactions between campuses and external entities, generate strategies
to improve the quality of broader impacts activities, and identify faculty development objectives Overhauling the faculty-reward system will not work without a broad shift in the attitudes of
that are scholarship-based. Linking broader impacts activities with forms of engaged scholarship professors, especially those on promotion and tenure committees. Simply adding new metrics
reinforces the value of such work and establishes new indicators for the institutionalization of or categories of work to evaluate as part of the faculty reward systems is not enough. Academics
engagement at research universities. must be encouraged to think differently. Executive leaders can inculcate a supportive climate by
publicly promoting community engagement as a priority and ensuring that engagement activity
The results of the study also hold practical implications. Expertise emerged as the strongest
is publicly recognized through award programs, university publications, and events that showcase
predictor of broader impacts activity type and quality, yet respondents reported receiving little
the work of engaged scholars. By creating new traditions, rituals, and symbols that reinforce the
training in such work. The study identified the need for faculty professional development in
value of community-based work, institutions can foster an epistemological shift toward democratic
effective evaluation, putting the results to use, establishing clearly defined partnerships with
engagement and ease the dynamic tension accompanying the transition from traditional to
community agencies, and equitably distributing power and responsibility with community
alternative value systems, creating enduring and pervasive change.
partners. Such training need not be limited to faculty. Respondents reported that they had
received little to no training and experience in community engagement as graduate students. With decreasing appropriations for public higher education and shrinking budgets, academics
Because graduate school is where researchers are socialized as scholars and where they develop are under growing pressure to secure external funding for their research. In response to prodding
The Journal of Research Administration, (47)1