Integrating Evaluation into Tobacco Programs for Asian American and Pacific Islander Communities

16 Pages·2001·0.72 MB·English

M H OL
Integrating Evaluation into
Tobacco Programs for Asian American
and Pacific Islander Communities
This evaluation kit is designed to help people who are developing
or implementing tobacco control programs for Asian American
and Pacific Islander (AAPI) communities.
The goals of this kit are:
• to help program planners understand the value of evaluation
for their program
• to identify ways to plan and incorporate evaluations
• to provide different methods for conducting evaluations
• to provide case studies of how evaluations can be planned
for specific AAPI tobacco control programs.
This kit is divided into:
the following sections: 1) a question and answer section that provides
basic evaluation information, 2) case studies of specific evalua¬
tion planning and implementation examples, 3) a checklist of
steps to plan your own evaluation, 4) a worksheet and 5) a list
of resources for further information.
4^~ 4 Ml
Answers
and
O What is evaluation and why is it important ? programs because they help us to identify and
Evaluation helps you to determine the effects of document what works for our communities. There
your activity or program on the participants and on are very few AAPI tobacco control programs that
the community and to document the successes and have had adequate time and resources to plan and
challenges. The goal of evaluation is to help you conduct extensive evaluations. While this is a reality
(and others) to understand what your program is of community-based programs, the lack of evaluation
actually accomplishing. For example, is your pro¬ has resulted in the lack of proven models that work.
gram being implemented the way you planned? Is it We must conduct evaluations of our own programs
reaching the kinds of people you wanted? Is it in order to document efforts and measure change.
achieving the kinds of short-term changes (like
increased knowledge about the harm of tobacco use, O How do you overcome fear of evaluations ?
or increased skills in refusing to start using tobacco)
Most often, people are intimidated by evaluation
and long-term changes (such as tobacco use preven¬
either because they don’t know what it is for, or
tion or cessation) that you expected? Is it making a
they do not know how to do it. The reasons for
difference in the long run? Evaluation can help you
doing evaluation are to benefit the program and
to answer these kinds of questions and more.
document the successes and challenges. Program
Evaluation is also important because it can help you evaluations are less concerned about whether an
to make changes to your program during imple¬ individual did a "good" or "poor" job (in fact, eval¬
mentation, as well as areas for improvement in the uations that assess job performance are different
future. For instance, evaluation of a youth tobacco from program evaluations). However, many people
use prevention program could identify that the cur¬ do not understand this difference, and thus believe
rent program participants were not those who are at that the purpose of evaluation is to find fault with
high risk for smoking. If you found this out mid¬ how they did their job.
way through your program implementation, you
Since people may be intimidated by evaluations,
could change your program outreach to reach the
they believe that it takes a lot of training and
kinds of youth you originally intended.
expertise to execute. Many people, however, are
Finally, evaluations are important for Asian already doing evaluation-type activities as part of
American and Pacific Islander tobacco control their program! For instance, do you keep docu-
(Case #1)
Using Interviews of Youth Leaders to
Evaluate Program Outcomes in the Pacific
On an island in the Pacific, local governmental
leaders have developed a successful youth for
youth leadership program.The long-term goal
of the program is to prevent youth from using
tobacco, including not smoking or chewing.
The program recruits middle school and high
school students, and requires them to sign a
pledge to not use tobacco during the month
of their training. During training, teen leaders
train other teens in tobacco control, including
health effects and policies, as well as in skills
ments like sign-in sheets, meeting notes, photo¬
to stay tobacco free.The program is so suc¬
graphs of events in the community, or newspaper cessful that it has become a model for other
clippings of articles on your program? These docu¬ like programs across the Pacific and on the
ments can all be used to evaluate your program! A U.S.The major evaluation question was
review of sign-in sheets of youth who attended your whether the training program "worked" to
tobacco prevention education sessions, lor example, help prevent more teens from not smoking
could help you to understand the types of people than other types of educational programs. In
order to answer this, staff decided to inter¬
who were involved — how many, what ethnicity,
view six randomly identified teen participants
what gender, what age, etc. If your original program
each year at the beginning of their training
goal was to reach middle school Cambodian stu¬
and one year later. Each interview took about
dents, a review of your sign-in sheets could help you
half an hour to complete, and asked each par¬
to determine whether you actually reached those
ticipant three questions: I) why they believe
types of students.
they are at risk for tobacco use, 2) what they
It is important to de-mystify evaluation for everyone, have learned to help prevent them from using
in order to show them that evaluation can be based tobacco, and 3) how they think they may or
may not be able to stay tobacco-free. From
on common-sense approaches to answering basic
these interviews, the staff learned that the
questions about the process or outcomes of a program.
more at-risk a teen feels he or she is, the
more things they learned from the program,
O What are the different types of evaluations ?
and the more they believed they could stay
There are three basic types of evaluation: process,
away from tobacco.These interviews are con¬
outcome, and impact. tinuing to be conducted, sometimes modified
to include other questions of importance for
• Process evaluation tells you about how your pro¬
that specific year.The "findings" from these
gram was implemented. Process evaluation looks
interviews have been included in the yearly
at what your program did, what was distributed to
reports of progress to the Department of
the community, who and how many people were
Health Services, which continues to give
involved, when the event took place, and what
money each year to sustain the program.
they were involved in. For instance, the example
of reviewing sign-in sheets (in the previous sec-
tion) is a form of process evaluation. Process O How do I start to plan an evaluation of
evaluation is often the most frequently used type my program ?
of evaluation that is conducted for a program.
To start to plan a program evaluation, you should
• Outcome evaluation looks at the immediate effects first look at your program’s goals and objectives, and
of your program, such as changes in program par¬ answer the following questions:
ticipants’ tobacco use knowledge, attitudes and
behavior. For example, post-test surveys of youth O Who is my program trying to reach?
who attended a prevention education session is a Prevention education programs focusing on youth
form of outcome evaluation. Photographs of vendors are very different from cessation programs focusing
who change the placement of tobacco advertisements on adults, or business education programs focusing
and promotional items could be another way to on vendors. The answer to this question will help
document the effects of a business education pro¬ you to eventually determine if your program
gram. This type of evaluation is very important reached the intended audience.
because it allows you to understand whether your
program made the kind of difference you expected. O What kinds of change is my program trying
to accomplish?
• Impact evaluation looks at the long-term effects of
If you program is successful, what would your
our program, usually on the reduction in disease
results look like? The answers to these questions
and/or deaths due to tobacco use in your commu¬
will help you to define how to assess the outcomes
nity. This type of evaluation is often the hardest to
ol your program.
do because it often takes years for someone who uses
tobacco to get sick or die. Most likely, your program
O What are the main ways that my program tried
will have ended long before we know whether it
to change the community?
saved anyone’s life. However, many researchers are
Understanding where your program priorities are
interested in this type of evaluation, and can com¬
will help you to tailor the evaluation to focus on
mit the time needed to track long-term changes.
the most important aspects, rather than on your
For the purposes of this tool kit, we will primarily
entire program.
focus on process and outcome evaluation.
Evaluation Methods,Advantages/Chal
and Resources Needed
Table I
Method Purpose Advantages and Challenges Resources Needed
To gather informa¬ + Most common method for + Can be done by program

t
n tion from pre-exist¬ process evaluation staff themselves
e w ing written materials,
m + Good method for reviewing - Takes a lot of time to
e
such as meeting
u i materials on how a program review existing files,
v
minutes, reports,
c e was implemented especially if program is
o r budgets, newspaper
complex and of long
D + Uses information that is
articles, etc.
duration
already available
To gather informa¬ + Good for process evaluation, + Can be done by program
tion on how the because you can get an staff themselves
program operates, immediate impression about
- Takes a lot of time to
or on visual changes how the program is progressing
review existing files,
that have resulted
— Information can be difficult especially if program is

n
to interpret complex and of long
o
duration
i — May be biased by how staff
t
a
interpret the information
v
r
e — Events that are seen can be
s
time-dependent, and not
b
O representative of what is
really going on
— If people know they are
being observed, they may
act differently than usual
To obtain informa- + Excellent for both process - Can take a lot of time
tion from individuals and outcome evaluation, to conduct
</i about their experi- because you can get in-depth - May require data inputting
£ ences, or to learn information from participants and analysis programs
oj more about their
• m>m + Can ask sensitive questions - May require assistance
answers to surveys
&- that require confidentiality from a consultant to
A\
- Data can be difficult and design the interview
c
complex to analyze protocol and develop
the database program
- Interviewer can bias
the responses
KEY: + advantages
- disadvantages
Table I: Evaluation method, advantages/challenges and resources needed, continued
Method Purpose Advantages and Challenges Resources Needed
To hold discussions Good for outcome evaluation, + Can be done relatively
</) with groups of because you can ask people quickly (1-2 hours per
Q_ people (10-12) to do explain how the program focus group)
3
understand in-depth affected them
0 Requires a good facilitator
i- what they believe
Can identify a lot of issues
w> Takes time to analyze and
were the effects of
and effects
(/) interpret the discussion
the program, or how
3
Can give staff better under
U they saw the program May require extra resources
O standing of the program from
implemented. for facilitator’s time and
Li.
participants’ own words
participant incentives
To describe a + Good for combining process Can take a lot of time
program or and outcome evaluation to collect information,
v) experience in depth, + Can be a powerful way to organize and describe
0) 0) often using some or the program
(/> describe the program
all of the other
3 + Can depict personal experiences,
U 4-» evaluation methods
</) quotes, and unique program
to construct a case
processes
study.
To get information + Excellent for outcome Takes time to develop
from individual evaluation questionnaire
people about their
+ Can get information from a May require database
changes in tobacco
lot of people program to manage and
use knowledge,
analyze a lot of data
+ Can be done confidentially
IS) attitudes, and
CJ or anonymously, so may be May require extra resources
behavior.
L.
more valid for participant incentives
and data analysis
</) c + Can be used as pre/post tests
X<v oc to measure changes from
> program educational sessions
1- </>
3 V + Can use questions from
l/> 3
cr existing surveys
— More effective when using
-o
c yes/no or true/false type
rd
questions
— Are more impersonal for
participants, and usually not
good for getting quotes
in participants’ own words
KEY: + advantages
- disadvantages
O What evaluation methods work best if I don’t know. The following table provides an overview of
have many resources ? the different kinds of evaluation methods, which
are loosely ordered from more simple (and requiring
There are many methods that can be used in an
less outside resources) to more complex.
evaluation, some require more resources in terms
of time and from the perspective of what you want
O Do all evaluations have to have a "before" and
to find out — some methods take less time and
"after" design?
money, but may not give you the kind of informa¬
Many evaluators believe that evaluations must be
tion you need to answer your evaluation questions.
based on measures of what was going on before the
For instance, conducting observations of a teen
program, and what happened after the program. A
leadership education program and writing about the
typical example is the use of pretests and posttests
outcomes can take much less time than conducting
during educational sessions, where session partici¬
a survey of what the youth participants’ believed
pants are asked to complete a survey at the start of
were their outcomes.
the session, followed by another one at the end of
However, the observational method may be criti¬ the session. Changes between the beginning (i.e.,
cized as biased (on the part of the person who con¬ pretest) and the end (i.e., posttest)are assumed to
ducted the observations), while the interviews might be due to the effects of the education itself. A more
be considered more valid. You must balance consid¬ complex example is the use of baseline measures
erations of resources with what you really need to (such as surveys) at the start of your program, and
Hypothetical Case Study Example of
How Evaluation Can Help!
(Case #2)
Using Program Documents to
Develop Case Studies of an Alternative
Sponsorship Education Project
A community-based organization decided to
tackle the difficult issue of educating Asian
American and Pacific Islander community groups
about the importance of refusing money from
tobacco companies.The long-term goal was to
eliminate all tobacco industry money support
for AAPI community groups in three years,
and identify alternative non-tobacco resources
instead.Their primary evaluation question was
“What steps should programs take if they want
to conduct tobacco sponsorship efforts in other
areas?” Staff started by identifying a list of
community groups that sponsored annual AAPI
cultural events and fairs, like the Vietnamese
Tet festival, Korean fall celebration, and Pacific
Islander festival. About one year before each
event, the staff also developed an in-depth log
then repeating the measures at the end to assess
file for each community group, in which each
community-wide changes. [If you are interested in
staff documented the dates, times, and content
how to conduct baseline measures, see APPEAL’S
of conversations with the group, any barriers
publication, "Conducting Needs Assessments lor
and concerns, possible alternative funding
sources to approach (such as local foundations Tobacco Control in Asian American and Pacific
that could give $5000 to print a festival pro¬ Islander Communities."]
gram, instead of the program, being printed by
However, it is often difficult to get information from
a tobacco company), and ultimate outcomes
participants or from the entire community before
of the effort.These conversations with local
the start of the program. Think about the example
groups occurred over the course of the year,
of a restaurant education program in the Cambodian
and by the end the staff had very large files on
how they worked with each community group. community — it would be very difficult to count
Finally, on the day of the community event, staff all of the restaurants that support anti-smoking laws
attended and documented through photographs before your program began. But in this example, it
the presence and the lack of tobacco company would be possible for people to record how many
advertisement and sponsorship. During the
restaurants they spoke to throughout the program,
last two months of the program, staff took
how many were initially resistant to a new law
all the information that they collected and
(e.g., believed that anti-smoking laws would harm
constructed "case studies" of each community
their business), and how many were willing to
group (without using the group’s name or any
change to support a new law alter the stalf talked
specific ethnic community identifiers in order
to them. This kind of documentation of the pro¬
to maintain confidentiality) — whether or not
they were successful in changing the group’s gram uses both "observation" and "interview"
sponsorship acceptance. These case studies methods of evaluation, which could be used to
were then distributed to all agency staff to develop case studies of program successes. In this
train them on how to go about doing anti¬ case, the "power" of evaluation is to document the
sponsorship type activities for the future.
kinds of things that stall discussed with the restau¬
rant owners, the kinds of barriers to any new laws,
and the kinds of information that helped convince • Also, good evaluations include incentives for com¬
owners of the importance of a new law. munity members to participate. If you are planning
to do a focus group or survey, you should include
Therefore, there is no hard and fast rule about need¬
resources for "thank you" incentive items, trans¬
ing "before" and "after" designs for your evaluation.
portation, etc. If your evaluation will be based
upon observations, you can enlist youth to help
O What resources are available or who can help
you take pictures of what they see has changed in
with our evaluation?
their community (such as less tobacco promotional
In general, there are also some general rules you may
items at their local stores or restaurants). You
want to follow to maximize the resources you have
should also provide participants with the final
for an evaluation:
evaluation report, or at minimum, a summary of
• First, start with an assessment of the capacities of the findings.
your staff. Someone who is a good listener might • Finally, do you have a university nearby from which
make an effective interviewer. Someone who likes you can recruit students, faculty or researchers to
to take photos and document events might make assist you with your evaluation? Oftentimes they
a good observer. Someone who knows programs may not understand the complex issues of tobacco
like Microsoft Excel or Access might be able to control for AAPIs — in other words, they may
design a database program for survey data. not have the cultural competence to develop an
• Think about how many resources you have avail¬ entire evaluation plan. However, they might have
able to evaluate your program. A basic rule is to specific skills that can help you conduct your own
allocate about 10% of your program’s budget to evaluation — skills like putting questions into a
evaluation activities. These resources may be more survey format, designing a database program, and
or less — just like when planning a program, you analyzing data for presentations or reports.
need to think about who will be responsible for
planning, designing, conducting, and compiling
the evaluation information.
How to Integrate Evaluation
Tobacco
- into
Control for AAPIs
Now that you have a basic background on the value
of evaluations for your tobacco control program,
here is a checklist of steps that you can take to
incorporate evaluation into your program. The
evaluation steps you take do not necessarily have
to occur in this order, but this checklist should help
you to get started in your effort!