ebook img

T.A.G+ Technical Assistance Guide... Employment and Training Administration... U.S. Department of Labor... 1994 PDF

124 Pages·1994·37.9 MB·English
Save to my drive
Quick download
Download
Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.

Preview T.A.G+ Technical Assistance Guide... Employment and Training Administration... U.S. Department of Labor... 1994

TAG TM ce} a - | IK =Yela lallet=] | _ ‘ , \ ee SOU” ln BLANK PAGE Mat. rial contained in this publication is in the public domain and may be reproduced, fully or partially, without permission of the Federai Government. Source credit is requested but not required. Permission is required only to reproduce any copyrighted material contained herein. This material will be made available to sensory impaired individuals upon request. Voice phone: 202-219-7664 TDD* phone: 1-800-326-2577 *Telecommunications Device for the Deaf. Technical T.A.G.+ Aasialanes Guide U.S. Department of Labor Robert B. Reich, Secretary Employment and Training Administration Doug Ross, Assistant Secretary Office of Strategic Plannina and Policy Development Raymond J. Uhalde, Acting Deputy Assistant Secretary 1994 T.A.G. + | TABLE OF CONTENTS | MODULE 1 - ABOUT TAG. + Page 1 MODULE 2 - TO SAMPLE OR NOT TO SAMPLE -- THAT MODULE 3 - TERMINEE CALL RECORDS Page 17 MODULE 4 - PREPARING THE RESPONDENT FOR FOLLOW-UP Page 25 MODULE 5 - THE INTERVIEWER IS A KEY PLAYER Page 33 MODULE 6 - DATA LINKAGES Page 41 MODULE 7 - UTILIZATION -- UNLEASHING THE LIONS Page 49 MODULE 8 - COLLECTING DATA IN AN IMPERFECT WORLD: UNDERSTANDING ERROR AND BIAS Page 57 MODULE 9 - POSTPROGRAM DATA AND PERFORMANCE MANAGEMENT Page 65 MODULE 10 - LEVEL 3: THE PAST INTO THE FUTURE Page 73 MODULE 11 - EVALUATION QUESTIONS FOR POSTPROGRAM DATA: A SAMPLER Page 81 MODULE 12 - A STUDY GUIDE FOR FOLLOW-UP AND EVALUATION Page 89 T.A.G. + MERGED INDEX 1990 1.A.G. INDEX ON a Prepared for the U. S. Department of Labor, ETA by Appropriate Soksions, inc., Columbus, Ohio. Contract @99-1-36S4-72-078-01 I eee T.A.G. + Supplement to the Technical Assistance Guide for JTPA Follow-up and Validation — MODU1L - E199 2 Published by the Employment and Training Administration, U.S. Department of Labor, Fn NS310, 200 Constitution Avenue, N.W., WashingtonD,.C . 20210 - (202) 535-0687 In 1991, the U.S. Departmem of Labor funded The second phase of the project was to prepare a Appropriate Solutions, inc. (ASI) to carry out a supplement te the 1990 Technical Assistance Guide technical assistance project on JTPA participant (1980 TAG.) on follow-up which would not only follow-up. presem new information, but would also enhance the accessibility and utility of the material in the 1990 The first phase of the project involved visits to a TAG. number of States and SDAs. These visits had two purposes: 1) to assist the organizations visited; and Instead of producing a single document, i was 2) to gather information on the data collection and decided to divide the supplement into twelve parts utilization problems being experienced in the job that are sensitive to the research and planning cycles training system. of Staatnde SsDAs . The sites represented the diversity within the JTPA The supplement has been named T.AG.*, which system. in some of the States visited, each SDA tells the reader that this publication Is intended to go colleitcs towne ddat a. Other States had centralized beyond the 1990 TAG. and provide value-added systems and contracted with a survey organization information. Its twelve modules will present materials for data collection services. One State had a centralized system, but interviews were being conducted in-house. In spite of these differences, sites tended to experience similar problems and were concaboeut rthen samee idssu es. FLAGS T.A.G.* is a twelve module supplement to the 1990 Technicai Assistance Guide for JTPA Follow-Up and Validation. To better meet our objectives, we have adopted two conventions. You will find that we will consistently “key” you to relevant information in the 1990 T.AG. and “flag* related matorial in other T.A.G.* Modules with the following symbols. a T.AG.: pp. a ee Information in the 1990 T.A.G. will be “keyed” by this information in the T.A.G.* Modules will be “flagged” symbol. by this symbol. Prepafrore ‘nde U.S . Departmoef Lnabtor , ETA by Appropriate Sohaions, inc., Columbus, Ohio. Con@t99- r1-38a54-c72-0t786- 01 T.A— MoGdule. 1 —* Pag e 2 The primary emphasis of T.A.G.* will be on helping T.AG.* should be disseminated to anyonaen d States and SDAs get the most cut of their follow-up everyone who wants to know more about follow-up, and evaluation systems. TAG.* should be postprogram evaluation, performance management, circulated and discussed among the people who and policy analysis. information conceming how to contribute to making follow-up work at all levels. We request additional copies of the 1990 TAG. and T.A.G.* modules can be found on page & We are dealing with serious material, but we hope to present hti n a fashion that you will not only find encowuider Gisasemginaetio n. 3 3 helpbuft eunjloya,ble . The original 1990 T.A.G. is an essential reference for people involved in follow-up, and we will refer to it often. | DEFINING THE JTPA SYSTEM Regardless of how data collection is organized within does work end informatifolonw ? Follow- a State, JTPA participant follow-up is a complex up is an orderly process, involving a system. When any of the elements which make up definable sequence of events. When steps in that system do not operate effectively, or when that process do not occur, occur late, or are information does not flow smoothly between the carried out badly, the entire system suffers. elements, the entire system is less effective. a What kinds of feedback are available to The elements which comprise follow-up systems are the membof ether sysste m? Feedbacki s people who fill specialized roles within their imponant because it enables the system to organizations. In sinaller SDAs, one person may fill adjust Rself to improve Rs performance. several roles. Typically, however, follow-up involves Follow-up systems can generate lots of many people, often in more than one organization. It information about their own functioningb,u t is not surprising, therefore, that communicatioann d feedback Is rot effective unless it is received coordinaiion issues were important to the sites and undebyr sosmeotne owhoo hdas the visited by the Foliow-up Technical Assistance project. capanad voclitiion tto reyspo nd. States and SDAs that want to strengthen their a Wheare rautehori ty, responsibilitayn,d programs may want to consider the following accounlotcataed bwitihinl tihe tsysyte m? questions as @ starting point for defining their Typically, ditieremt people are responsible for systmoere mClesarl y: Gifleremt stages of the follow-up process, but there is also a need for someone who is B Who is included in the follow-up system? respfoor tnhe ssysitem basl a ewho le. in Whdoa thtey do? What resourdco etshe y either case, Ff & is not clear where the need to carry out their functions? Oata authority to make specific decisions is collection is only one part of the follow-up located, decisions will not be made; and the system. Folblegions aws c-ounsueloprs a nd status quo will be perpetuated. instructors collect contact information and encourage participants to cooperate. itG oes The diagrarn on the next page shows that the system not end until the information collected is has a great deal of interaction, and, # operating used by decision makers. properly, a great amount of communication. There are no unimppaortsr itn tahe msytste m. Because a What are the connectiboetnwesen everyone in this systen) has an inherent inerest in membof ether sysste m? That is, how the success of the JTP/ program. they also have an X T.A— .ModuGle .1 -* Pag e 3 interest in the quality of the follow-up and postprogram evaluation. renowenTs The audiences at the bottom of the diagram are ra \ many and include USDOL, the Governor, Congress, COUNSELORS, the media, and the general public. Each eleme:tt in aw -_ the diagram is a potential user of follow-up information. They are also potentia) advisers to o her parts of the system. Ciear communication between PROGRAM ¢ > wey all elements leads to better and more responsive _ ya programs. 6 3 al ideally, data collection and utilization go hand in hand. Organizations collect what they need and use JTPA follow-up systems have generally been what they collect. Too often, howevteher d,at a designed to meet federal requirements for data collection process is designed without much thought colleacndt riepoortning . At the same time, States as to how the data collected will be used. When that and SDAs are discovering that those systems do not happens, the potential value of the data decreases always fully meet their own informationneeadls. dramatically. T.A.G.* is intended to address these problems. The three prerequisites to utilization are content, We view the issue of utilization in terms of three qualaindt aycce,ss . Although collecting data which levels. These levels represert increasing are relevant to our information needs is essential to sophiin susintg iinfocrmataiont ini perofornman ce utiliit izs noat tenoiugho bny ,itse lf. We cannouts e managaned pmoliecy nanatlys is. Each level of what we collect unless we are confident that what it utilization builds and expands upon the teuls iis svali d. And, regarocf iconetesnt san d level. (quality, we cannot use our data unless we can get i when we need it, the way we need it. LEVEL3 : Program evaluation and policy analysis, including support of longer term planning and policy development and measuof rproegramm eeffecntivtenes s. af program strengths and weaknesses and continuous feedback on the quality of the placements made. LEVEL1 : Compliance with federal requiremiencnlutdisn g annual reporting and implemenotf apetrfioromannc e standards. TAG.* — Modu1 —l Paege 4 Regardiess of your Local or State resources, we a Data is not Power; it's just data. When you believe that everybody is capable of working at Level use Data, f becomes informatWiheonn y.o u 2 and Level 3 if that is what they want. TA.G.* wil study the information, & becomes shoyouw ho w. Knowledge. When you ieam from that Knowledge, k& becomes Wisdom. Here are some key things to remember. inforKnomwleadget anid oWisndom, a re Power. B if you don't do anything with the data except fill in torms, then foliow-up Is @ cost. TAGpp.. 1:3-1 5. a S When you go beyond calculamtanidantogr y Utilization of your data will be a recurrent theme in perfmeoasurres,m anad bengin cto euse the T.AG.*. We wil be looking at all aspects of the data in performance management and policy follow-up and survey process to provide you with analysis, then follow-up becomes an tools and guidance to help you achieve the level of investment. utilitzhaat tyoiu odesnir e. That procbeegisns swi th your survey design. % 3 “As long as | have you on the phone..." A shorn GUIDE TO QUESTIONNAIRE DESIGN Follow-up is a required activity for collecting There are many good reasons to ask additional postprogram data that are used in calculating questions. The important thing is to have a reason perfmeaosurers anmd ina awanrdingc inceenti ves. for each question asked. The major cost in acquiring follow-up data lies in Avoid the common trap of asking too many questions locating the respondents and getting them on the just because “i would be nice to know.” If you never telephone. For a modest additiona! cost, States and get around to analyzyoiurn dgat a, you have wasted SDAs can add other questions which will help them time and resources. understand what is happening in their programs and with their participants. This is not as simple as it sounds. interviewing is more than just asking The preferred method for follow-up surveys is questions. Likewise, designing a questionnairet s telephone interviewing. Becyaou uwills be easki ng much more than just writing questions. the questions orally, you must listen to how they sound — not just in your head as you read them, but TAG.: pp. 8-9. aloud. Your design is not finished unti you are a ee satisfied that the questionnaire sounds right and does whyaout wan t. You are after clear and unarnbiguous ANALSTAYRTSS wiIth SDESI GN communication. The first task is io determine what you want to know ard what you are going to do with the answers. ONE THOUGHT What is this going to tell us? Will this be of help to A well Gesigned question contains a single thought. State or SDA program p.anning and design? Will this You may be tempted to construct a question with help us identify gaps in our program or needs we very complicated instructions or multiple thoughts. may have overlooked? Keep 8 simple. Understand what you are asking and TA—G Modu.le 1* — Pa ge 5 what the responisd aenswnertin g Consider the you feel are the 2 or 3 best sources of following: inforamboautt niewo jonbs ? How ‘wouyolu drat e the instructors OndeEtr FEC T and training facility — excellent, good, Writing a good question is only one element in the art faoir proor ? of design. The order in which you put those questions will also affect the accuracy of the data you How do respondents answer ff they think the collect. instructors were superb, but were cursed with a faciity that should have been razed in the 1800's? All things being equal, your questionnasihorueld You cannot design good questions unless you can move from the general to the specific. A broad, look at them through the respondente'yses . The open-ended question should precede specific close- above example should have been two questions: ended questions to prevemt suggesting answers to respondents. How would you rate the instructors— excelgoold, efainr otr p,oo r? An open-ended question has 10 specific categories How would you rate the training given for responses. An ex mople is the above facility - excellent, good, fair or poor? Question on sources for employn °nt leads. A close- ended question requires a specific response, such as Although you now have two questions instead of one, the question on evaluating instructors. the ansawre ierierrprestabl e. The two questionwisl l probably go faster than the one because the A high school studenti n ColumbOhiuo sdi,d a interviewer will not be stopped by respondents science fair experiment on order effects in 1969. He asking for clarification or qualifying their responses. asked all of the Freshman class two sets of questions. For half of the group he asked a general One CLEAR THOUGHT question about problems facing a high school student today followed by three specific problems. Yes, t was clear to you. No, it was not clear to For the other half, he reversed the order asking the anybody else. As you design questiyoou nfalsl ,in to specific questions first. mind sets. The following is a real example of a failure of this kind. When the data were analyzed, the second group, which was asked specific before general, There are a lot of things people can dispropogavre tonie oof nthae tthreee lspeycif ic do when they are looking for a job. if problems as a response to the general question. The you were looking for a job, what are respondent had been “cued” to possible respcnses the 2 of 3 most effective things you by hearing the specific questions first. Opinions had would do to find a job? been created. The designers of that question thou:/t they would find things such as checking help wanted ads, registering with an employamgeencyn, tet c. The respondents, however, looked at it differently and related things they did such as buying a sult and gettai hanirgcut . This is a better question. There are a lot of things people can do when they are looking for a job. if you were looking for a job, what do category (usually 9 or blank) should also be included S

See more

The list of books you might like

Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.