ERIC EJ918448: Stakeholder Input in Establishing an Evaluation Plan for Tobacco Counter-Marketing Campaigns

2011·0.28 MB·English

by  ERIC

Research Article
Stakeholder Input in Establishing an Evaluation
Plan for Tobacco Counter-Marketing Campaigns
Rebecca Murphy-Hoefer, Marco S. Andrade, Dorean E. Maines, and Maurice Martin
ABSTRACT
Background: Maine was one of eight states to consistently meet funding recommendations for tobacco control from
the Centers for Disease Control and Prevention (CDC) and one of three states to experience 45%-60% reductions in
youth smoking rates since 1999. Purpose: The state’s tobacco control coalition, Partnership for a Tobacco-Free Maine,
sought to develop an evaluation plan based on the framework from the CDC’s Introduction to Program Evaluation for
Comprehensive Tobacco Control Programs and the integration of the CDC’s Designing and Implementing an Effective
Tobacco-Counter-Marketing Campaign. Methods: The coalition conducted key informant interviews and then met
with stakeholders in Maine to address the design, management, implementation and evaluation of the state’s tobacco
counter-marketing campaigns. Results: The priorities identified included the need to improve communication among
all stakeholders, to synchronize local and state campaigns, to elicit audience insights among specific populations and
to link program inputs with outcomes through evaluation. Meeting participants noted that lessons learned should be
shared through internal mechanisms and external publications. Discussion: We describe the practical application of
state and national expertise and resources to the development of Maine’s tobacco counter-marketing campaign evalu-
ation. Translation to Health Education Practice: This article may help other public health programs to work with
stakeholders to identify program and evaluation needs in the development of a health communication evaluation.
Murphy-Hoefer R,. Andrade MS, Maines DE, Martin M. Stakeholder input in establishing an evaluation plan for tobacco counter-marketing
campaigns. Am J Health Educ. 2011;42(2):66-73. This paper was submitted to the Journal on May 29, 2009, revised and accepted for pub-
lication on October 20, 2010.
BACKGROUND comprehensive state tobacco control pro- Rebecca Murphy-Hoefer is a research associate at
In Maine, one-fifth of the adult popula- grams, including mass media campaigns.4 the A.L. Burruss Institute of Public Service and
tion smokes cigarettes1 causing an estimated Mass media campaigns can reduce smoking Research, Kennesaw State University, Kennesaw,
2,235 premature deaths each year.2 For every prevalence when combined with other effec- GA 30144; E-mail: [email protected].
person who dies from tobacco use, 20 others tive community-based interventions such as Marco S. Andrade was a research associate at the
suffer from tobacco-related illnesses.3 These smoking bans and restrictions, higher prices Maine Center for Public Health, Augusta, ME
health issues cost the state of Maine more for tobacco products, smoking cessation 04330. Dorean E. Maines is a program man-
than $534 million in smoking-attributable interventions, and telephone quitlines.5 ager at the Maine Centers for Disease Control
productivity losses and $550 million in The Centers for Disease Control and and Prevention, Augusta, ME 04330. Maurice
health care expenditures.2 Prevention (CDC) recommends that health Martin is the chairperson of the Department
The Institute of Medicine’s Ending the communication interventions, also de- of Community Health and Recreation at the
Tobacco Problem: A Blueprint for the Na- scribed as tobacco counter-marketing University of Maine at Farmington, Farmington,
tion outlines recommendations to support campaigns, should be considered for ad- ME 04938.
66 American Journal of Health Education — March/April 2011, Volume 42, No. 2
Rebecca Murphy-Hoefer, Marco S. Andrade, Dorean E. Maines, and Maurice Martin
dressing the agency’s goals for comprehen- media evaluation with future campaigns. members), target population (e.g., target
sive tobacco control programs, including: Using the conceptual framework from the population for the media campaign, policy
(1) preventing initiation among youth and CDC’s Introduction to Program Evaluation makers, advocacy groups), and the primary
young adults, (2) promoting quitting among for Comprehensive Tobacco Control Programs intended users (main users of the evaluation
youth and adults, (3) eliminating exposure and integrating the CDC’s Designing and findings or those who make decisions about
to secondhand smoke, and (4) identifying Implementing an Effective Tobacco-Counter- the media campaign). These groups are not
and eliminating tobacco-related disparities Marketing Campaign, the Maine Center for mutually exclusive. The Maine Center for
among specific populations.6 These cam- Public Health interviewed key informants in Public Health identified key informants in
paigns can also influence public support the state about their health communication Maine: program staff, members of the target
for tobacco control funding and contribute programs, and key informants outside the population (including policy makers) and
to a supportive climate for school, commu- state with experience in media evaluation.8,9,10 in-state and local-level partners. In addition,
nity, state and national policies to prevent These key informants identified stakehold- they identified key informants from outside
tobacco use.7 ers who could help design Maine’s tobacco of Maine by reviewing health communica-
Maine was one of eight states to meet the counter-marketing campaign evaluation. tion literature, primarily in tobacco control,
CDC’s minimum funding recommendation and querying authors of papers delivered
and the only state to have met the minimum PURPOSE at the most recent national tobacco control
funding recommendation every year since This article shares the strategies that and CDC health marketing conferences. For
the release of the first Best Practices for were generated during key informant in- Phase 1 of this effort, the evaluation team
Comprehensive Tobacco Control Programs in terviews and meetings with stakeholders developed a discussion guide9,10 to conduct
1999.6 Since then, Maine is one of three states to guide future efforts in tobacco counter- four conference calls and individual calls
to have experienced 45%-60% reductions in marketing campaigns in Maine. This with state and national tobacco control
youth smoking rates with sustained com- information may be helpful to other state program and heath communication experts
prehensive statewide programs.6 In 2005, programs that plan to evaluate their health (both with and without tobacco control
the state was the first to receive “straight A’s” communication programs. experience). Of the 21 experts identified,
on the American Lung Association’s State 16 participated in the conference and indi-
Tobacco Control Report Card. METHODS vidual calls in the summer of 2008. The calls
Arizona, California, Florida, Massa- In the spring of 2008, the Maine Center lasted one-to-three hours and were recorded,
chusetts, Minnesota, New York City and for Public Health’s evaluation team began to transcribed by Cambridge Transcriptions,
Oregon have documented the effectiveness develop a new media evaluation plan (Fig- and summarized according to the stage of
of statewide tobacco counter-marketing ures 1 and 2).9,10 The first step in the CDC’s evaluation, as defined by the CDC.9,10 These
campaigns.6 To explore the best practices in Framework for Program Evaluation is to en- internal documents were distributed to the
health communication efforts for tobacco gage stakeholders.8 The CDC has identified key informants for review.
control further, the Maine Center for Public three main groups of stakeholders necessary The second phase, a one-day meeting
Health on behalf of the Partnership for a for evaluation: program staff (e.g., manage- in the late summer of 2008, guided the
Tobacco-Free Maine sought to integrate ment, partners, advertising agency, coalition process of integrating evaluation into the
Figure 1. CDC’s Framework Figure 2. CDC’s Designing & Implementing an
for Program Evaluation8 Effective Tobacco Counter-Marketing Campaign10:
Managerial Aspects of Campaigns
1. Engage Stakeholders
1. Planning Your Counter-Marketing Program
2. Describe the Program
2. Managing and Implementing
3. Focus the Evaluation Design
3. Gaining and Using Audience Insights
4. Gather Credible Evidence
3. Reaching Specific Populations
5. Justify Conclusions
4. Evaluating the Tobacco Counter-Marketing Program
6. Ensure Use and Share Lessons
Learned
American Journal of Health Education — March/April 2011, Volume 42, No. 2 67
Rebecca Murphy-Hoefer, Marco S. Andrade, Dorean E. Maines, and Maurice Martin
next five-year tobacco counter-marketing level organization has work plans and objec- For evaluation purposes, an Indiana state
plan. Twenty-two stakeholders directly tives (i.e., strategic plans) that are developed representative works with the advertising
involved with the day-to-day operations of in accordance with indicators of state and and public relations contractor to obtain
the tobacco counter-marketing campaigns community excellence. In California, the the information needed to conduct media
and evaluation met with local and state state program has implemented a campaign tracking surveys.
partners, program staff and contractors, about the effects of secondhand smoke in Describing the Program
as well as three national experts who were multi-unit housing to support local-level The second step identified in the CDC’s
key informants during Phase 1. Forty stake- efforts. In New York, when the state conducts Framework for Program Evaluation is to
holders who were not directly responsible a campaign, it provides templates for media describe the program.8 We found that inte-
for day-to day operations also participated. materials and talking points to local partners grating components of the CDC’s Design-
This meeting format ensured that the to increase the likelihood that they will use ing and Implementing an Effective Tobacco
evaluation would be designed to answer a similar message. These same states noted Counter-Marketing Campaign worked well
questions important to stakeholders, ensure that when partners speak with one voice the for describing the program in relation to the
that the evaluation would be used, increase message is more readily comprehended and development of a media evaluation (Figure
support for a long-range plan and build the role of the state is to perform that inte- 2).10 Specific tobacco counter-marketing
capacity in evaluation. gration. For example, Indiana has been able program components include planning a
to mobilize state and local grantees quickly.
program, implementing and managing a
RESULTS The stakeholders at the local level know who
program, gaining and using audience in-
their policy makers are and they are ready to
Engaging Stakeholders sights and reaching specific populations.10
make calls when needed with a single health
During Phase 1, conference calls and One of the main findings of evaluation in
communication message.
one-on-one interviews with key informants New York shared by a key informant was the
In regard to target populations, the
emphasized the importance of communica- importance of having a good mix of adver-
tions among stakeholders. Key informants following key points were generated by tisements, not just those with high emotional
agreed that stakeholders need to understand the key informants: transparency through content. The mix of advertisements depends
roles, communication systems and decision information sharing is needed with stake- on resources and the focus of the campaign
making at each stage of evaluation. They holders at every stage, program goals and as noted by several key informants. Cam-
added that communication should honor objectives should be clearly articulated, paigns in New York have focused on smokers
the expertise of each stakeholder group (e.g., stakeholders should be educated about and tend to have three or four campaigns per
advertising agencies, state and local health the process of developing a campaign and year, with several advertisements for each
department staff, community partners, formative evaluation, and a plan should not campaign. In California, the focus has been
evaluation contractors, state and local-level be implemented until all stakeholders agree at the community-level and typically five or
coalition members, and funding agencies) on program direction for program sustain- six advertisements will run each year. “Some
given that each group has unique and valu- ability. In New York State, the decision of advertisements have longer staying power,”
able expertise and should invest the time which and how many advertisements to run as noted by key informant David Cowling,
necessary to build bridges to other groups. each year must be cleared at several levels. PhD, California Tobacco Control Program.
Several issues were raised about meeting This process typically takes up to one year. He added that evaluators should evaluate the
face-to-face on a regular basis. Although it is If an advertisement is not approved in time, level of saturation and determine the cutoff
preferred, it is not always possible because of existing and pre-approved advertisements point for effectiveness. One method used to
time, distance and weather conditions. are used. Several key informants noted that prolong the staying power of advertisements
Key informants stressed the importance decision makers (e.g., health commissioner, is to have a group of advertisements that are
of state and local partners working collab- health department officials and legislators) shown during a specific period of time, also
oratively to integrate statewide campaigns should be educated about tobacco counter- known as media flights, rather than showing
with local activity. They elaborated that the marketing campaigns. the same advertisement continually.
interaction between the multiple compo- The primary intended users of the Key informants said that gaining and
nents is more effective than the individual ef- evaluation are those in a position to make using target audience insights, identifying
forts alone. For example, California, Indiana decisions about a paid media campaign, a sound theoretical foundation for each
and New York recognize the pivotal role of such as the project director from the health campaign and using health promotion
local partners, particularly in reaching out department or foundation, agency staff and processes are important parts of formative
to the public. These states have contracts evaluators.9 For example, New York State research and evaluation.10 Several key in-
with local health departments and other has weekly meetings with its advertising formants stressed that insight from the
community-based organizations. Each local- contractor, evaluator and program manager. target audience, not the health department
68 American Journal of Health Education — March/April 2011, Volume 42, No. 2
Rebecca Murphy-Hoefer, Marco S. Andrade, Dorean E. Maines, and Maurice Martin
staff or political entities, should carry the ate for the population. Dr. Cowling shared needed at once to evaluate multi-component
most weight when identifying what types of that evaluations completed in California media campaigns, as noted by key informant
advertisements will have the most influence show that most people, regardless of their Dr. David Sly of Florida State University. He
on behavior change. The consensus of the primary language, consume media in Eng- added that evaluation needs must then be
key informants was that even if a budget lish – particularly through television. Media prioritized. Key informants discussed that
is limited, formative research needs to campaigns in California have been more while independent (i.e., outside) evalua-
be conducted to have confidence in the successful among high and middle socio- tion is unbiased; intimate knowledge of the
messages being used. economic status groups than among low program, goals of a campaign, theoretical
In Colorado, a guiding principle is socioeconomic status groups. In addition, foundation of a campaign and media mes-
to be true to the formative research. If different races/ethnicities consume different sages are also advantageous.
an administrator does not approve a types of media. According to an informant Key informants discussed how different
concept generated by the focus groups, from California, African Americans tend to types of evaluation are used, such as for-
then program staff members revisit the consume more television, while Hispanics mative, process and outcome (short-term,
research and find a way to use another consume more radio. Radio may be more intermediate and long-term outcomes).
concept from the target audience. For cost-effective than television to reach non- California has been using online panels to
example, focus groups of pregnant native speakers in their own language, and test advertising concepts as part of their
smokers said that money weighed more other specific populations, especially during formative research and evaluation. before
heavily than the health of the unborn the summer. Another participant high- the advertisements are produced, screen
baby. However, a message focused on lighted the importance of seasonal timing, shots and voice-overs (called animatics)
finances was not accepted by decision noting that during election time, television are streamed through the Internet to panel
makers. After revisiting the results of and other media may be more expensive and participants. Their feedback is used to decide
the formative research, stress was noted placement may be a challenge. which advertisements to produce. A web-
as another issue for the focus group In rural areas, key informants stressed based process evaluation system is used
participants. Decision makers approved that local partnerships are important in in California, Indiana and New York to
this topic, and the campaign resonated deciding what works on a local level, such as monitor local activity. Local partners are
well with the target audience. As the free weekly newspapers, signs at local post required to have an evaluation plan for each
former media director of Colorado’s offices and shopping centers. In some rural objective in their work plan and to submit an
State Tobacco Education and Prevention areas of New York, for example, residents evaluation report at the end of each funding
Program said, “If you’re going to conduct must use one road. The local partners know cycle. The evaluation plans and subsequent
the research, you should listen to it.” that if a billboard is placed there, the entire reports may include a media component.
Key informants said that focus groups community will see it. Indiana tobacco Partners record the cost of paid and earned
allow for in-depth discussion about audi- control health communication specialists media. This type of system provides a
ence insights and campaign components. use earned media to reach areas not covered mechanism for assessing expenditures and
Conducting focus groups may be time- through paid media. These specialists in accountability. With these data, Indiana
consuming and expensive, informants said, Indiana have extended media reach through program officials have been able to respond
but they provide valuable information and news releases tailored to local communities, to legislative requests about specific local
help decision makers understand the issues sponsorships and creating new or using ex- activity. A process evaluation method that
from the target audience’s point of view. isting events to gain earned media coverage. wasn’t effective in one of the key informant’s
Ideally, specific campaigns should be Indiana monitors state and local coverage states was to analyze specific sites in terms
developed for every targeted population. If of the news by contracting with a clipping of identify specific sites in terms of their
this is not possible key informants said that it service to clip every tobacco-related article in current progress with tobacco control (i.e.,
is necessary to prioritize specific populations daily and weekly papers. These clips are then “the weather”) instead of the overarching
and the campaign decision makers need to coded by topic and the coverage is analyzed question of how they had changed over time
be willing to make tough decisions about at least monthly. and what roles the local partners played in
where a campaign can have the most effect. Designing the Evaluation that change (i.e., “the climate”).
Nevertheless, the same campaign often can All key informants agreed that the evalua- Key informants discussed that a broad
be used by multiple populations by changing tion of health communication efforts should design can capture two or more individual
the actors or the language, but keeping the be coordinated with the evaluation of the designs, that cross-sectional designs can be
overarching theme the same, as noted by a overall tobacco control program from the conducted over time with a longitudinal
key informant from California. beginning of a campaign. Several different component and that a cohort design can
The media channel should be appropri- logic models and evaluation designs may be explain variations between those who re-
American Journal of Health Education — March/April 2011, Volume 42, No. 2 69
Rebecca Murphy-Hoefer, Marco S. Andrade, Dorean E. Maines, and Maurice Martin
member versus those who do not remember the panel are recruited through knowledge informant interviews, literature reviews, and
various advertisements. They also said that networks, which tend to offer them free identified the following priorities.
designs must feature some type of variation Internet service as an incentive. Sampling
Engaging Stakeholders
in exposure, such as in the media buy ac- and period of recruitment are two significant
Participants recommended that stake-
cording to geographic region, in campaign variations in panel designs. Sampling may
holders should create a work group to ad-
cycles over time, in audience segments, or be population- or convenience-based (such
dress stakeholder involvement, including
between those who remember versus those as the opt-in method of recruitment). The
defining roles, responsibilities, and common
who do not remember advertisements. As population-based method is more rigor-
terminology. For example, they recom-
noted by Dr. David Sly, “Four components ous, but often must be sacrificed to sample
mended using monthly phone calls to keep
are required to attribute effects to a media enough smokers. The period of recruitment
stakeholders informed.
campaign and provide results for program may be a single panel (New York) or a panel
Subgroups of stakeholders, such as the
accountability: appropriate level of exposure cohort that completes surveys during a two-
advisory committee, advocates, evaluators,
and awareness among the target population, year period (California). A two-year cohort
targeted populations and legislative com-
program outcomes change, association with allows for longitudinal data.
mittee members, have unique issues. For
exposure to the campaign, and other pos- California uses population-based sam-
example, advocates should be involved in
sible explanations (e.g., tax increases, secular ples that are followed for two years. The
ongoing communication through timely
trends).” Additional key points discussed by sample initially included approximately 540
meetings. Advocates must have a means to
the key informants was that data collection smokers and now has added opt-in panels
share the action steps and progress of the
should be carefully monitored, match the to bring the sample up to 900 smokers and
media campaigns. Participants discussed
target audience, start with baseline data 1,200 nonsmokers. New York uses an opt-in
the importance of planning how to share
and use consistent survey items over time sample of participants who are recruited
health communication campaign briefings,
to allow comparisons with follow-up sur- based on criteria the evaluators choose
tobacco-related stories of constituents and
veys. The New York Adult Tobacco Survey (e.g., male smokers). In 2010, the sample
program outcomes throughout the year.
includes short and specific questions about was approximately 1,500. Disadvantages
advertisements, based on three criteria: of online surveys noted by the informants Describing the Program
confirmed awareness, personal meaning to are that a large percentage of the popula- Program staff said that district tobacco
the respondent and personal action (i.e., did tion, particularly low-income households, coordinators should be included in the
the respondent take action, such as talking to does not have a computer in the household planning process, and regional media plans
another person about the advertisement). or access to the Internet and the random should be integrated into the statewide plan.
A few key informants believe that random component of population–based samples They discussed making the most of media
digit dialing (RDD) may be inefficient for is absent. buys (e.g., two-for-one media buys), working
certain target audiences and that purchased The evaluation design should include with programs around the country for ideas
lists may be more cost-effective than RDD. detailed dissemination plans. 8 The annual and campaigns (e.g., the American Legacy
A few key informants have also purchased report and independent evaluation report Foundation’s Great Start campaign to work
sampling frames from vendors or have used are still necessary, but the emphasis is mov- with pregnant smokers), and creating media
the registrars’ lists from universities. An ad- ing toward developing a series of topical attention by sharing information about the
ditional point was that incentives may be reports and management briefs for differ- program (e.g., evaluation findings).
needed to motivate the target audience to ent stakeholders as identified by several key The development of a communications
participate in an evaluation. informants. For example, one key infor- systems map was discussed to show meetings
Historically, California and New York mant said that a legislative report should and participants, decision-making points,
used the telephone interview method to be shorter than other types of publications and roles and responsibilities of decision
conduct their outcome surveys. However, and should be packaged into “sound bites” makers. This kind of map can strengthen
both have recently changed to an online and personal stories and quotes from the the focus on stakeholder involvement and
media tracking survey. This web-based quitline. Publication in peer-reviewed pro- streamline decision making. Convening
method is less expensive and allows for fessional journals adds to the sustainability regular meetings to debrief, monitoring
different types of evaluation. Panelists can of health communication interventions as progress and proposing adjustments were
screen the whole or part of an advertisement agreed upon by the key informants. described as ways of providing timely com-
and report their reaction. Screen shots may Phase 2 munication among stakeholders.
also help jog the panelist’s memory before In Maine, 65 stakeholders met for Phase To ensure a common understanding
questions are asked to confirm awareness 2 of the state’s effort to design an evaluation about the importance of the target audi-
and campaign evaluation. Participants on plan. They discussed issues from the key ence, a few stakeholders said that they
70 American Journal of Health Education — March/April 2011, Volume 42, No. 2
Rebecca Murphy-Hoefer, Marco S. Andrade, Dorean E. Maines, and Maurice Martin
should be educated and invited to attend tion were identified. Participants requested research may provide additional guidance
focus groups. In addition, understanding reports that should address stakeholders. For about the effectiveness of tobacco counter-
the target audience improves the rationale example, getting timely campaign feedback marketing campaigns among audiences of
for targeting harder-to-reach populations to program planners is essential and inform- interest and appropriate theories to use. In
and why changes to a campaign are needed ing stakeholders about how the program is particular, research and evaluations have
to be effective. making a difference for the common good, shown that stronger, hard-hitting, emotion-
Participants said that the strategic plan not only about program sustainability. Par- ally powerful and disturbing advertisements
should include a mix of statewide and tar- ticipants also said that reporting positive are effective in encouraging smokers to
geted media to address specific populations. anecdotal stories should be balanced with quit,11-14 but these kinds of advertisements
An example, in relation to reaching specific reporting program outcomes. Partners are must not be overused. “Softer” advertise-
populations, was the consideration of using needed for program outcome dissemination. ments are aimed at ensuring that smokers
weekly and target audience specialty papers Meeting participants said that eventually understand that services exist to help them
in campaign development. Maine should consider establishing a data quit, and that quitting can be difficult. The
The integration of earned media into and information archive. “quitting Takes Practice” advertisements
the strategic plan was also indentified as a by the California Department of Health
program need. DISCUSSION are examples of softer advertisements.9 In
Designing the Evaluation With the goal of guiding the process of general, mass media campaigns are most
Participants described the need for an in- integrating evaluation with future media effective when they:11
dependent evaluation contractor and for ad- plans, the Partnership for a Tobacco-Free • Are part of broader, comprehensive to-
equate funding for evaluation. For example, Maine intends to use the valuable informa- bacco control programs designed to change
Partnership for a Tobacco Free Maine needs tion gained from the key informants’ and a community’s prevailing attitudes toward
more information to inform campaign stakeholders’ input. The framework from tobacco use.
development in the areas of the stagnating the CDC’s Introduction to Program Evalu- • Convey simple messages.
decline in adult smoking rates, cessation ation for Comprehensive Tobacco Control • Are visually exciting.
treatments, especially for smoking relapse Programs and the integration of the CDC’s • Include advertisements with strong
and quit-line use. Conducting assessments Designing and Implementing an Effective negative emotional appeal that produce, for
for new health communication campaigns, Tobacco-Counter-Marketing Campaign example, a sense of loss, disgust, or fear.
including paid media, will help identify key provided the structure necessary to move • Introduce persuasive new information
ideas to incorporate into media campaigns. forward the evaluation needs of the Part- or new perspectives about health risks to
Focusing on intermediate outcomes as part nership for a Tobacco-Free Maine.9,10 The smokers and nonsmokers.
of the Partnership for a Tobacco Free Maine’s overriding message from stakeholders was • Use personal-testimony or graphic-
strategic planning process will enhance the that communication among state and local depiction formats to reach smokers and
existing evaluation plan. health departments, partners, media cam- that youth find emotionally engaging but
Another evaluation priority that was paign planners, and evaluators is important not authoritarian.
identified was the tracking and reporting throughout every stage of evaluation. • Feature multiple message strategies,
of earned media—letters to the editor, Key informants and stakeholders dis- advertising executions and media channels
editorials, and feature stories—because they cussed the need to synchronize local and to consistently attract, engage and influence
are useful to program planners, advocates state efforts and to gain audience insights diverse youth with varying levels of suscep-
and legislators. Tracking of other non-paid among specific populations. Gaining and tibility to smoking.
placements, such as brochures, posters, and using target audience insights, identifying a • Provide adequate exposure to media
collateral items (e.g., key rings) was also sound theoretical foundation for each cam- messages over substantial periods of time.
identified as useful. paign and using health promotion processes • Avoid placement of advertisements
The final step in the CDC’s Framework are important parts of formative research meant to teach parents about talking with
for Program Evaluation is to ensure that the and evaluation.10 Conducted in the early “your kids” when they may be viewed by
evaluation results are used.8 Participants stages of campaign development, formative adolescents.
identified the need to include a plan for research involves analysis of the audience In addition, many processes are available
regular and predictable dissemination of by reviewing health status measures (e.g., for understanding audiences such as social
lessons learned in the overall Partnership behavioral Risk Factor Surveillance System, marketing,16-20 and community-driven par-
for a Tobacco-Free Maine strategic plan. Youth Risk behavior Surveillance System), ticipatory models.21,22 Formative research
The need for role clarification of evaluators, demographics, psychographics, media habits through focus groups provides an op-
and the responsibilities regarding dissemina- and attitudes about certain issues. Published portunity to elicit the needs, desires
American Journal of Health Education — March/April 2011, Volume 42, No. 2 71
Rebecca Murphy-Hoefer, Marco S. Andrade, Dorean E. Maines, and Maurice Martin
and expectations of the target audience. observational, post-only, pre-post only, Control and Prevention, National Center for
Specifically, existing advertisements multiple pre-post, repeated cross-sectional, Chronic Disease Prevention and Health Promo-
can be tested for effectiveness among and longitudinal.10 tion, Office on Smoking and Health; 2007.
target audiences. The CDC’s Media Maine intends to work with an inde- 7. U.S. Department of Health and Human
Campaign Resource Center facilitates pendent evaluator, develop a survey linking Services. Reducing tobacco use: a report of the sur-
the use of existing advertisements, key outcome indicators, and share lessons geon general. Atlanta: U.S. Department of Health
including some with evaluation data.11 learned through internal mechanisms and and Human Services, Centers for Disease Control
Key informants noted that it may be easier external publications. and Prevention, National Center for Chronic
to obtain permission from state agency Disease Prevention and Health Promotion, Office
management and stakeholders to use an TRANSLATION TO HEALTH on Smoking and Health; 2000.
existing advertisement if other states have EDUCATION PRACTICE 8. Centers for Disease Control and Preven-
already done so and require fewer financial The process outlined in this article may tion. Framework for program evaluation in pub-
and time-related resources than creating help other public health programs to use a lic health. Morb Mortal Wkly Rep. 1999;48(No.
new advertisements. In addition, the use of similar process to work with stakeholders RR-11).
existing advertisements may appeal to deci- and to identify program and evaluation 9. MacDonald G, Starr G, Schooley M, Yee SL,
sion makers who want to see their state doing needs. Communicating and synchronizing Klimowski K, Turner K. Introduction to program
at least as much as other states to reduce the with state and local stakeholders, gaining evaluation for comprehensive tobacco control
burden of tobacco use. audience insights, developing an evalua- programs. Atlanta, Georgia: Centers for Disease
The next step for Maine’s program is to tion with an independent evaluator that Control and Prevention; 2001.
develop the evaluation that links program links program activities with outcomes, and 10. Centers for Disease Control and Preven-
activities with outcomes. A logic model sharing lessons learned through internal tion. Designing and implementing an effective
should be used to measure the progress mechanisms and external publications are tobacco counter-marketing campaign. Atlanta,
toward program goals.8 Comprehensive all priorities that other programs may deem Georgia: U.S. Department of Health and Hu-
evaluation includes all three types of appropriate for their health communica- man Services, Centers for Disease Control and
evaluation: formative, process, and out- tion programs. Prevention, National Center for Chronic Disease
come. For example, formative evaluation Prevention and Health Promotion, Office on
provides information about program REFERENCES Smoking and Health; 2003.
design and development, such as 1. Centers for Disease Control and Pre- 11. Schar E, Gutierrez K, Murphy-Hoefer R,
qualitative communication checks vention. Cigarette smoking among adults— et al. Lessons learned from youth tobacco use pre-
of advertising concepts before pro- United States, 2006. Morb Mortal Wkly Rep. vention mass media campaigns in nine countries.
duction or quantitative copy tests to 2007;56(44):1157-1161. Atlanta, GA: Centers for Disease Control and
verify that produced advertisements 2. Centers for Disease Control and Preven- Prevention; 2006.
communicate effectively to the target tion. State tobacco activities tracking and evalua- 12. Hyland A, Wakefield MA, Higbee C, et al.
audience. Process evaluation provides tion (STATE) system. Available at: http://apps. Anti-tobacco television advertising and indica-
information about the integrity of nccd.cdc.gov/statesystem. Accessed September tors of smoking cessation in adults: a cohort
program implementation.8,22 It mea- 29, 2010. study. Health Educ Res. 2006;21(2):296-302.
sures relationships among partners, 3. Centers for Disease Control and Preven- 13. Netemeyer R, Andrews C, burton S. Ef-
how an advertisement component fits tion. Cigarette smoking-attributable morbidi- fects of antismoking advertising-based beliefs
into an overall program and message ty—United States, 2000. Morb Mortal Wkly Rep. on adult smokers’ consideration of quitting. Am
tracking. Outcome evaluation pro- 2003;52(35):842-844. J Public Health. 2005:95(6);1062-1066.
vides information about the progress 4. Institute of Medicine. Ending the tobacco 14. Murphy-Hoefer R, Hyland A, Higbee
toward program goals through short- problem a blueprint for the nation. Washington, C. Perceived effectiveness of tobacco counter-
term, intermediate and long-term D.C.: The National Academies Press; 2007. marketing advertisements among young adults.
outcomes. 8 These include indicators 5. Zaza S, briss PA, Harris KW, editors. The Am J Health Behav. 2008;32:725-734.
such as changes in attitudes or beliefs guide to community preventive services: what 15. Centers for Disease Control and Preven-
about tobacco, intentions, behaviors, the works to promote health? New York: Oxford Uni- tion. Media campaign resource center. Available
environment, and disease prevalence.10 The versity Press; 2005. at: http://www.cdc.gov/tobacco/mcrc. Accessed
following outcome evaluation designs meet 6. Centers for Disease Control and Preven- September 1, 2010.
the primary goals of media evaluations to tion. Best practices for comprehensive tobacco con- 16. Andreasen AR. Marketing social change.
address short-term, intermediate and trol programs—2007. Atlanta: U.S.Department of San Francisco: Jossey-bass;1995.
long-term outcomes: field experiment, Health and Human Services, Centers for Disease 17. Kotler P, Zaltman G. Social marketing:
72 American Journal of Health Education — March/April 2011, Volume 42, No. 2
Rebecca Murphy-Hoefer, Marco S. Andrade, Dorean E. Maines, and Maurice Martin
an approach to planned social change. J Mark 20. Lefebvre RC, Rochlin L. Social marketing. tion. Introduction to process evaluation in tobacco
1971;35:3-12. In: Glanz K, Lewis FM, Rimer bK, eds. Health use prevention and control. Atlanta, GA: U.S.
18. New York State Department of Health. behavior and health education: theory, research, Department of Health and Human Services,
Social marketing and public health lessons from and practice, 2nd ed. San Francisco: Jossey- Centers for Disease Control and Prevention,
the field: a guide to social marketing from the bass;1997:384-402. National Center for Chronic Disease Prevention
social marketing national excellence collaborative. 21. Hennessey Lavery S, Smith ML, Es- and Health Promotion, Office on Smoking and
Albany, NY: Turning Point National Program parza AA, et al. The community action model: Health; 2008.
Office; 2003. a community-driven model designed to ad- 23. McAlister A, Morrison TC, Hu S, et al.
19. Siegel M, Doner L. Marketing public dress disparities in health. Am J Public Health. Media and community campaign effects on
health: strategies to promote social change. New 2005;95:611–616. adult tobacco use in Texas. J Health Comm.
York: Aspen Publishers;1998. 22. Centers for Disease Control and Preven- 2004;9:95-109.
American Journal of Health Education — March/April 2011, Volume 42, No. 2 73