T.A.G+ Technical Assistance Guide... Employment and Training Administration... U.S. Department of Labor... 1994

124 Pages·1994·37.9 MB·English

by  United States Department of Labor

TAG TM ce}
a - | IK =Yela lallet=] | _ ‘ ,
\
ee
SOU” ln
BLANK PAGE
Mat. rial contained in this publication is in
the public domain and may be reproduced,
fully or partially, without permission of the
Federai Government. Source credit is requested
but not required. Permission is required
only to reproduce any copyrighted material
contained herein.
This material will be made available to
sensory impaired individuals upon request.
Voice phone: [phone]
TDD* phone: [phone]
*Telecommunications Device for the Deaf.
Technical
T.A.G.+ Aasialanes Guide
U.S. Department of Labor
Robert B. Reich, Secretary
Employment and Training Administration
Doug Ross, Assistant Secretary
Office of Strategic Plannina and Policy Development
Raymond J. Uhalde, Acting Deputy Assistant Secretary
1994
T.A.G. +
| TABLE OF CONTENTS |
MODULE 1 - ABOUT TAG. + Page 1
MODULE 2 - TO SAMPLE OR NOT TO SAMPLE -- THAT
MODULE 3 - TERMINEE CALL RECORDS Page 17
MODULE 4 - PREPARING THE RESPONDENT FOR
FOLLOW-UP Page 25
MODULE 5 - THE INTERVIEWER IS A KEY PLAYER Page 33
MODULE 6 - DATA LINKAGES Page 41
MODULE 7 - UTILIZATION -- UNLEASHING THE LIONS Page 49
MODULE 8 - COLLECTING DATA IN AN IMPERFECT WORLD:
UNDERSTANDING ERROR AND BIAS Page 57
MODULE 9 - POSTPROGRAM DATA AND PERFORMANCE
MANAGEMENT Page 65
MODULE 10 - LEVEL 3: THE PAST INTO THE FUTURE Page 73
MODULE 11 - EVALUATION QUESTIONS FOR POSTPROGRAM
DATA: A SAMPLER Page 81
MODULE 12 - A STUDY GUIDE FOR FOLLOW-UP AND
EVALUATION Page 89
T.A.G. + MERGED INDEX
1990 1.A.G. INDEX
ON
a
Prepared for the U. S. Department of Labor, ETA by Appropriate Soksions, inc., Columbus, Ohio. Contract @99-1-36S4-72-078-01
I
eee
T.A.G. +
Supplement to the Technical Assistance Guide for JTPA Follow-up and Validation — MODU1L - E199 2
Published by the Employment and Training Administration, U.S. Department of Labor, Fn NS310, 200 Constitution Avenue, N.W.,
WashingtonD,.C . 20210 - [phone]
In 1991, the U.S. Departmem of Labor funded The second phase of the project was to prepare a
Appropriate Solutions, inc. (ASI) to carry out a supplement te the 1990 Technical Assistance Guide
technical assistance project on JTPA participant (1980 TAG.) on follow-up which would not only
follow-up. presem new information, but would also enhance the
accessibility and utility of the material in the 1990
The first phase of the project involved visits to a TAG.
number of States and SDAs. These visits had two
purposes: 1) to assist the organizations visited; and Instead of producing a single document, i was
2) to gather information on the data collection and decided to divide the supplement into twelve parts
utilization problems being experienced in the job that are sensitive to the research and planning cycles
training system. of Staatnde SsDAs .
The sites represented the diversity within the JTPA The supplement has been named T.AG.*, which
system. in some of the States visited, each SDA tells the reader that this publication Is intended to go
colleitcs towne ddat a. Other States had centralized beyond the 1990 TAG. and provide value-added
systems and contracted with a survey organization information. Its twelve modules will present materials
for data collection services. One State had a
centralized system, but interviews were being
conducted in-house. In spite of these differences,
sites tended to experience similar problems and were
concaboeut rthen samee idssu es.
FLAGS T.A.G.* is a twelve module supplement to the 1990 Technicai Assistance Guide for JTPA Follow-Up and
Validation. To better meet our objectives, we have adopted two conventions.
You will find that we will consistently “key” you to relevant information in the 1990 T.AG. and “flag* related matorial
in other T.A.G.* Modules with the following symbols.
a
T.AG.: pp.
a ee
Information in the 1990 T.A.G. will be “keyed” by this information in the T.A.G.* Modules will be “flagged”
symbol. by this symbol.
Prepafrore ‘nde U.S . Departmoef Lnabtor , ETA by Appropriate Sohaions, inc., Columbus, Ohio. Con@t99- r1-38a54-c72-0t786- 01
T.A— MoGdule. 1 —* Pag e 2
The primary emphasis of T.A.G.* will be on helping T.AG.* should be disseminated to anyonaen d
States and SDAs get the most cut of their follow-up everyone who wants to know more about follow-up,
and evaluation systems. TAG.* should be postprogram evaluation, performance management,
circulated and discussed among the people who and policy analysis. information conceming how to
contribute to making follow-up work at all levels. We request additional copies of the 1990 TAG. and
T.A.G.* modules can be found on page & We
are dealing with serious material, but we hope to
present hti n a fashion that you will not only find encowuider Gisasemginaetio n. 3 3
helpbuft eunjloya,ble .
The original 1990 T.A.G. is an essential reference for
people involved in follow-up, and we will refer to it
often.
| DEFINING THE JTPA SYSTEM
Regardless of how data collection is organized within does work end informatifolonw ? Follow-
a State, JTPA participant follow-up is a complex up is an orderly process, involving a
system. When any of the elements which make up definable sequence of events. When steps in
that system do not operate effectively, or when that process do not occur, occur late, or are
information does not flow smoothly between the carried out badly, the entire system suffers.
elements, the entire system is less effective.
a What kinds of feedback are available to
The elements which comprise follow-up systems are the membof ether sysste m? Feedbacki s
people who fill specialized roles within their imponant because it enables the system to
organizations. In sinaller SDAs, one person may fill adjust Rself to improve Rs performance.
several roles. Typically, however, follow-up involves Follow-up systems can generate lots of
many people, often in more than one organization. It information about their own functioningb,u t
is not surprising, therefore, that communicatioann d feedback Is rot effective unless it is received
coordinaiion issues were important to the sites and undebyr sosmeotne owhoo hdas the
visited by the Foliow-up Technical Assistance project. capanad voclitiion tto reyspo nd.
States and SDAs that want to strengthen their a Wheare rautehori ty, responsibilitayn,d
programs may want to consider the following accounlotcataed bwitihinl tihe tsysyte m?
questions as @ starting point for defining their Typically, ditieremt people are responsible for
systmoere mClesarl y: Gifleremt stages of the follow-up process, but
there is also a need for someone who is
B Who is included in the follow-up system? respfoor tnhe ssysitem basl a ewho le. in
Whdoa thtey do? What resourdco etshe y either case, Ff & is not clear where the
need to carry out their functions? Oata authority to make specific decisions is
collection is only one part of the follow-up located, decisions will not be made; and the
system. Folblegions aws c-ounsueloprs a nd status quo will be perpetuated.
instructors collect contact information and
encourage participants to cooperate. itG oes The diagrarn on the next page shows that the system
not end until the information collected is has a great deal of interaction, and, # operating
used by decision makers. properly, a great amount of communication. There
are no unimppaortsr itn tahe msytste m. Because
a What are the connectiboetnwesen everyone in this systen) has an inherent inerest in
membof ether sysste m? That is, how the success of the JTP/ program. they also have an
X
T.A— .ModuGle .1 -* Pag e 3
interest in the quality of the follow-up and
postprogram evaluation.
renowenTs
The audiences at the bottom of the diagram are ra \
many and include USDOL, the Governor, Congress, COUNSELORS,
the media, and the general public. Each eleme:tt in aw -_
the diagram is a potential user of follow-up
information. They are also potentia) advisers to o her
parts of the system. Ciear communication between PROGRAM ¢ > wey
all elements leads to better and more responsive _ ya
programs. 6 3 al
ideally, data collection and utilization go hand in
hand. Organizations collect what they need and use JTPA follow-up systems have generally been
what they collect. Too often, howevteher d,at a designed to meet federal requirements for data
collection process is designed without much thought colleacndt riepoortning . At the same time, States
as to how the data collected will be used. When that and SDAs are discovering that those systems do not
happens, the potential value of the data decreases always fully meet their own informationneeadls.
dramatically. T.A.G.* is intended to address these problems.
The three prerequisites to utilization are content, We view the issue of utilization in terms of three
qualaindt aycce,ss . Although collecting data which levels. These levels represert increasing
are relevant to our information needs is essential to sophiin susintg iinfocrmataiont ini perofornman ce
utiliit izs noat tenoiugho bny ,itse lf. We cannouts e managaned pmoliecy nanatlys is. Each level of
what we collect unless we are confident that what it utilization builds and expands upon the
teuls iis svali d. And, regarocf iconetesnt san d level.
(quality, we cannot use our data unless we can get i
when we need it, the way we need it.
LEVEL3 : Program evaluation and policy analysis, including
support of longer term planning and policy development
and measuof rproegramm eeffecntivtenes s.
af program strengths and weaknesses and continuous
feedback on the quality of the placements made.
LEVEL1 : Compliance with federal requiremiencnlutdisn g
annual reporting and implemenotf apetrfioromannc e
standards.
TAG.* — Modu1 —l Paege 4
Regardiess of your Local or State resources, we a Data is not Power; it's just data. When you
believe that everybody is capable of working at Level use Data, f becomes informatWiheonn y.o u
2 and Level 3 if that is what they want. TA.G.* wil
study the information, & becomes
shoyouw ho w. Knowledge. When you ieam from that
Knowledge, k& becomes Wisdom.
Here are some key things to remember. inforKnomwleadget anid oWisndom, a re
Power.
B if you don't do anything with the data except
fill in torms, then foliow-up Is @ cost. TAGpp.. 1:3-1 5.
a
S When you go beyond calculamtanidantogr y
Utilization of your data will be a recurrent theme in
perfmeoasurres,m anad bengin cto euse the T.AG.*. We wil be looking at all aspects of the
data in performance management and policy
follow-up and survey process to provide you with
analysis, then follow-up becomes an
tools and guidance to help you achieve the level of
investment.
utilitzhaat tyoiu odesnir e. That procbeegisns swi th
your survey design. % 3
“As long as | have you on the phone..."
A shorn GUIDE TO QUESTIONNAIRE DESIGN
Follow-up is a required activity for collecting There are many good reasons to ask additional
postprogram data that are used in calculating questions. The important thing is to have a reason
perfmeaosurers anmd ina awanrdingc inceenti ves. for each question asked.
The major cost in acquiring follow-up data lies in Avoid the common trap of asking too many questions
locating the respondents and getting them on the just because “i would be nice to know.” If you never
telephone. For a modest additiona! cost, States and get around to analyzyoiurn dgat a, you have wasted
SDAs can add other questions which will help them time and resources.
understand what is happening in their programs and
with their participants. This is not as simple as it
sounds. interviewing is more than just asking
The preferred method for follow-up surveys is
questions. Likewise, designing a questionnairet s
telephone interviewing. Becyaou uwills be easki ng
much more than just writing questions.
the questions orally, you must listen to how they
sound — not just in your head as you read them, but
TAG.: pp. 8-9.
aloud. Your design is not finished unti you are
a ee
satisfied that the questionnaire sounds right and does
whyaout wan t. You are after clear and unarnbiguous
ANALSTAYRTSS wiIth SDESI GN
communication.
The first task is io determine what you want to know
ard what you are going to do with the answers. ONE THOUGHT
What is this going to tell us? Will this be of help to
A well Gesigned question contains a single thought.
State or SDA program p.anning and design? Will this
You may be tempted to construct a question with
help us identify gaps in our program or needs we
very complicated instructions or multiple thoughts.
may have overlooked?
Keep 8 simple. Understand what you are asking and
TA—G Modu.le 1* — Pa ge 5
what the responisd aenswnertin g Consider the you feel are the 2 or 3 best sources of
following: inforamboautt niewo jonbs ?
How ‘wouyolu drat e the instructors OndeEtr FEC T
and training facility — excellent, good,
Writing a good question is only one element in the art
faoir proor ?
of design. The order in which you put those
questions will also affect the accuracy of the data you
How do respondents answer ff they think the
collect.
instructors were superb, but were cursed with a
faciity that should have been razed in the 1800's?
All things being equal, your questionnasihorueld
You cannot design good questions unless you can
move from the general to the specific. A broad,
look at them through the respondente'yses . The
open-ended question should precede specific close-
above example should have been two questions:
ended questions to prevemt suggesting answers to
respondents.
How would you rate the instructors—
excelgoold, efainr otr p,oo r?
An open-ended question has 10 specific categories
How would you rate the training given for responses. An ex mople is the above
facility - excellent, good, fair or poor? Question on sources for employn °nt leads. A close-
ended question requires a specific response, such as
Although you now have two questions instead of one, the question on evaluating instructors.
the ansawre ierierrprestabl e. The two questionwisl l
probably go faster than the one because the A high school studenti n ColumbOhiuo sdi,d a
interviewer will not be stopped by respondents science fair experiment on order effects in 1969. He
asking for clarification or qualifying their responses. asked all of the Freshman class two sets of
questions. For half of the group he asked a general
One CLEAR THOUGHT question about problems facing a high school
student today followed by three specific problems.
Yes, t was clear to you. No, it was not clear to
For the other half, he reversed the order asking the
anybody else. As you design questiyoou nfalsl ,in to
specific questions first.
mind sets. The following is a real example of a failure
of this kind.
When the data were analyzed, the second group,
which was asked specific before general,
There are a lot of things people can
dispropogavre tonie oof nthae tthreee lspeycif ic
do when they are looking for a job. if
problems as a response to the general question. The
you were looking for a job, what are
respondent had been “cued” to possible respcnses
the 2 of 3 most effective things you
by hearing the specific questions first. Opinions had
would do to find a job?
been created.
The designers of that question thou:/t they would
find things such as checking help wanted ads,
registering with an employamgeencyn, tet c.
The respondents, however, looked at it differently
and related things they did such as buying a sult and
gettai hanirgcut . This is a better question.
There are a lot of things people can
do when they are looking for a job. if
you were looking for a job, what do category (usually 9 or blank) should also be included
S