DOCUMENT RESUME ED 460 307 CG 029 097 AUTHOR Presley, Cheryl; Austin, S. Bryn; Jacobs, Judith Selecting the Right Tool: A Compendium of Alcohol and Other TITLE Drug Assessment and Evaluation Instruments for Use in Higher Education. INSTITUTION Higher Education Center for Alcohol and Other Drug Prevention, Newton, MA. SPONS AGENCY Department of Education, Washington, DC. PUB DATE 1997-00-00 NOTE 19p. CONTRACT SS95013001 AVAILABLE FROM Higher Education Center for Alcohol and Other Drug Prevention, Education Development Center, Inc., 55 Chapel St., Newton, MA 02458-1060; Tel: 800-676-1730 (Toll Free); Fax 617-928-1537; e-mail: HigherEdCtr@edc:org; Web site: http://www.edc.org/hec PUB TYPE Guides Non-Classroom (055) EDRS PRICE MF01/PC01 Plus Postage. DESCRIPTORS College Students; *Drinking; Evaluation Methods; Higher Education; *Measurement Techniques; *Prevention; Program Evaluation; *Substance Abuse ABSTRACT The purpose of this compendium is to assist administrators, staff, evaluators, and alcohol and other drug (ADD) prevention specialists in identifying and selecting feasible, informative, and appropriate evaluation instruments to assess AOD use and monitor program effectiveness on campus. The use of qualitative methods in the evaluation of grant-funded projects is discussed. Section 1, "Choosing the Right Instrument," presents the primary issues that must be considered when choosing an instrument, which range from format and feasibility to technical concerns about the quality of the measure. Section 2, "A Compendium of Measures," has short descriptions of seven instruments selected by a panel of experts in AOD prevention. These tools are among the best in the field and are designed to cover a variety of areas including frequency and amount of AOD use, consequences of AOD use, student perceptions of AOD use on campus, fraternity and sorority AOD environment, faculty and staff perceptions of AOD use, and community coalition involvement in prevention efforts. Section 3, "Annotated Evaluation Bibliography," covers resources on research design, statistical analysis, evaluation, and other instruments. (EMK) Reproductions supplied by EDRS are the best that can be made from the original document. http://www.edc.org/hec/pubs/selecting-right-tool.txt Selecting the Right Tool A Compendium of Alcohol and Other Drug Assessment and Evaluation Instruments for Use in Higher Education Cheryl Presley/S. Bryn Austin/Judith Jacobs This article discusses the use of qualitative methods in the evaluation of grant-funded projects. CONTENTS Preface Introduction Section 1: Choosing the Right Instrument U.S. DEPARTMENT OF EDUCATION Office of Educational Research and Improvement EDUCATIONAL RESOURCES INFORMATION Section 2: A Compendium of Measures CENTER (ERIC) Alcohol and Other Drug Surveys 0 This document has been reproduced as received from the person or organization Environmental Assessment originating it. Fraternity and Sorority Assessment 0 Minor changes have been made to improve reproduction quality. Other Instruments Points of view or opinions stated in this docu- ment do not necessarily represent official Section 3: Annotated Evaluation Bibliography OERI position or policy. PREFACE The Higher Education Center for Alcohol and Other Drug Prevention was established by the U.S. Department of Education in 1993 to assist institutions of higher education in developing and carrying out alcohol and other drug (AOD) prevention programs that will promote campus and community safety and help nurture students' academic and social development. To accomplish this mission, the Center seeks to increase the capacity of postsecondary schools to develop, implement, and evaluate programs and policies that are built around environmental management strategies. Environmental management means moving beyond general awareness and other education programs to identify and change those factors in the physical, social, legal, and economic environment that promote or abet alcohol and other drug problems. Clearly, stemming the use of alcohol and other drugs is not something that college administrators alone can achieve. Top administrators, especially presidents, must exercise leadership, but their success will depend ultimately on their ability to build a strong coalition of both on-campus and community interests. The better AOD prevention programs are campuswide efforts that involve as many parts of the college as possible, including students, staff, and faculty. For this reason, the Center emphasizes team-focused training and technical assistance work. Building coalitions with local community leaders is also key. College campuses do not exist in isolation. AOD prevention planners need to collaborate with local leaders to limit student access to alcohol, prevent intoxication, and support the efforts of local law enforcement. The Center therefore seeks to motivate and train academic leaders to work with local community representatives, while also joining with national organizations that urge local coalitions to increase their outreach to academic institutions. This publication represents one piece in a comprehensive approach to AOD 2 AVAILABLE BEST COPY 2/18/99 10:22 AM 1 of 18 http://www.edc.org/hec/pubs/selecting-right-tool.txt prevention at institutions of higher education. The concepts and approaches it describes should be viewed in the broader context of prevention theory and the approaches affirmed by the U.S. Department of Education and promoted by the Center in its training, technical assistance, publication, and evaluation activities. For information on Center services, please contact: The Higher Education Center for Alcohol and Other Drug Prevention Education Development Center, Inc. 55 Chapel Street Newton, MA 02458-1060 (800) 676-1730 Tel.: (617) 928-1537 Fax: Website: http://www.edc.org/hec/ E-mail: [email protected] INTRODUCTION The aim of this compendium is to assist administrators, staff, evaluators, and AOD prevention specialists in identifying and selecting feasible, informative, and appropriate evaluation instruments to assess AOD use and monitor program effectiveness on campus. Since passage of the federal Drug-Free Schools and Communities Act of 1986, colleges and universities have been required to maintain alcohol and other drug (PLOD) prevention programs and policies and to conduct biennial reviews of their effectiveness. Partly in response to the federal mandate, institutions of higher education across the country have launched a myriad of ambitious programs and policy changes. Increasingly, school officials have begun to devote a significant amount of staff time and resources to systematic, campus-based assessment of AOD use at their individual institutions. They recognize that the available national data on college AOD use, while informative, are not alone a sufficient basis on which to make programming decisions. Assessing AOD use and measuring the effects of prevention efforts can be a daunting task, especially when there are a number of programs on campus that promote prevention on multiple levels. A single institution may have in place AOD prevention programs designed to address individual student behavior, interpersonal behavior, perceptions of social norms, institutional change, and community-level change. High demands on resources and staff time also pose challenges to evaluation work. It may be unrealistic to expect every campus to conduct a rigorous, experimental evaluation of its programming and policies. But even so, all schools need to use key indicators to monitor progress and suggest needed modifications. This document will help in either case. Often the most basic piece of information collected in campus assessments is the prevalence of AOD use. The proportion of students drinking alcohol or using other drugs, the frequency of use, and the amount of use are all critical to understanding the AOD use patterns at a college or university and to monitoring program impact. But there are many other factors that evaluations can assess to help improve understanding of the ADD climate on campus. Selecting the right data collection instruments is a vital component of evaluation. This compendium covers the important issues to consider when making those selections and identifies the leading instruments used in the postsecondary AOD prevention field. The instruments included in this compendium not only assess student AOD use but also cover consequences of AOD use; student, staff and faculty perceptions of AOD use; environmental factors that encourage use (for instance, tailgate parties) or discourage use (such as substance-free housing); and other issues related to the college or university AOD climate. The chief criteria for inclusion of the instruments in this compendium are 3 2/18/99 10:22 AM 2 of 18 http://www.edc.org/hec/pubs/selecting-right-tool.txt that they are * designed specifically for a postsecondary student population or environment * well-tested, valid (a term discussed in section 1 meaning that the instrument actually measures what it purports to measure), and reliable (meaning that the instrument provides consistent measurements over time) * intended for general survey research and not for the diagnosis of alcoholism or alcohol abuse problems in individuals Section 1 of the compendium presents the primary issues that must be considered when choosing an instrument, which range from format and feasibility to technical concerns about the quality of the measure. Section 2 has short descriptions of seven instruments selected by a panel of experts in AOD prevention. These tools are among the best available in the field and are designed to cover a variety of areas relevant to campus AOD prevention, including assessment of * frequency and amount of ADD use * consequences of AOD use * student perceptions of AOD use on campus * fraternity and sorority AOD environment * faculty and staff perceptions of AOD use * community coalition involvement in prevention efforts Section 3 contains an annotated bibliography of resources on research design, statistical analysis, other aspects of evaluation, and additional instruments not included in this compendium. While informed and careful instrument selection is central to quality evaluation, it is only one piece of a comprehensive.evaluation strategy. It is hoped that the information included in this compendium will serve as a valuable resource for administrators, staff, evaluators, and prevention specialists as they develop their evaluation plans and work to increase the effectiveness and efficiency of prevention programming and policy on America's college and university campuses. SECTION 1: CHOOSING THE RIGHT INSTRUMENT Key Features of Instruments * Instruments vary a great deal, not only in terms of the information they assess but also in the way they assess it and how well they do so. Within the AOD prevention field, instruments have been designed to collect data on a range of factors, from student ADD use patterns to perceptions of campus norms to environmental conditions that promote or deter AOD use. Before choosing an instrument it is important to determine what kind of data is needed. If it has not been clearly established which questions about AOD use on campus are the most pressing, there is a risk of selecting an instrument that focuses on irrelevant information or that omits important data. In section 2 of this compendium, the domain, or range of factors covered by an instrument, is described in the summaries under Application and Outcomes. * A second consideration in selecting an instrument is the target population. Instruments are designed with a particular group of people in mind. For instance, a measure intended for adults may be too difficult for children to understand. Also, a measure surveying college students about drinking on campus would not be appropriate for young adults in the same age range if they were not attending a 4 2/18/99 10:22 AM 3 of 18 http://www.edc.org/hec/pubs/selecting-right-tool.txt college or university. All the instruments presented in this compendium are appropriate for the college community, but the intended respondents may be students, faculty and staff, or another subgroup on campus, depending on the measure. In section 2, the target population is described under Informants. * A third consideration is the format of the instrument. Many use the self-report, forced-choice format typical of standardized tests such as the SAT. Others are self-report but use open-ended questions to allow respondents more freedom in answering. Still others use an interview format, in which a trained staff person conducts a semistructured interview with a respondent or a group of respondents and records their comments. A self-report, forced-choice format is the most feasible for surveying a large number of people but is more restricted than the other formats in the detail of information that can be gathered. The interview format tends to be more labor intensive for staff compared with other survey techniques, but it also has the potential to collect more detailed and complex information from respondents. Format and length determine the time and effort required of respondents and may limit the settings in which an instrument can be administered. For instance, if students are to be surveyed during class time set aside by a cooperating professor, then a 20-minute, forced-choice questionnaire may be most feasible. Information about the format of the instruments is detailed in section 2 under Format and Administration. * A fourth and arguably the most important consideration is the technical quality of instruments. Experts in evaluation refer to technical quality as the validity and reliability of an instrument. Validity is an indication of an instrument's ability to measure what it is intended to measure. Reliability indicates the consistency with which the instrument provides those measurements. Consider, for example., a ruler, an everyday type of measuring instrument. If a ruler has high validity, then we know that the space between lines marking an inch truly is one inch. If the ruler has high reliability, it will measure one inch as the same amount of distance every time we use the ruler. Determining validity and reliability is essential to establishing the technical quality of an instrument and ultimately to producing credible evaluation results. Since validity and reliability testing is time consuming, the instrument selection process should always include a careful review of existing instruments that have been fully tested. Section 2 of this compendium includes information on the validity and reliability of each of the featured instruments. Because an understanding of these concepts is so critical to well-informed instrument selection, they are described in greater detail below. Validity There are three types of validity used in the field: content validity, construct validity, and predictive validity. In general, validity is not an all-or-nothing characteristic; rather, it is described in degrees from low to high. The more evidence gathered through multiple studies that shows that an instrument performs well in terms of the three types of validity, the more confidence evaluators can have in the measure and, most important, in the study results. Content Validity. included called items Content validity refers to whether the questions in an instrument cover the whole domain of factors it is intended to address. This form of validity is often determined by agreement among experts. For example, an instrument intended to measure patterns of alcohol use on a college campus would need to include items covering at least three areas: the proportion of students who drink, how frequently they drink, and how much they drink. If a measure addressed only one of these aspects of 5 2/18/99 10:22 AM 4 of 18 http://www.edc.org/hec/pubs/selecting-right-tool.txt college student alcohol consumption, experts would consider it to have poor content validity. Construct Validity. Construct validity refers to the extent to which an instrument successfully measures a theoretical concept-called a construct-such as anxiety, peer pressure, or perceptions of drinking norms. Scores from one instrument can be compared with scores from others that are intended to measure the same construct to assess whether the instruments perform comparably. Moderate or high correlation between instruments designed to measure the same construct suggests that the instruments are performing as intended. Predictive Validity. Predictive validity indicates an instrument's ability to provide meaningful patterns of results. For instance, student performance on the SAT is used by admissions committees to predict academic performance in college. How well SAT scores actually do predict success in college would be an indication of the SAT's level of predictive validity. Predictive validity is also used to refer to an instrument's ability to discriminate among groups of respondents who would be expected to score differently on a particular measure. For example, a researcher could conduct interviews with students about their alcohol consumption and then divide them into two groups, one made up of heavy drinkers and the other of light drinkers. If the researcher then had the students complete a questionnaire on AOD use, the expectation would be that the heavy drinkers would score differently on the instrument than light drinkers. If the questionnaire could not discriminate between the two groups, it would be considered to have low predictive validity. Reliability Reliability, which refers to a measure's consistency, can be divided into three main types: test-retest reliability, alternate forms reliability, and internal consistency. Test-Retest Reliability. An instrument's ability to measure a construct the same way at repeated testing times is referred to as its test-retest reliability. To assess this type of reliability, an instrument may be administered twice to a group of people. If the time in between testing is not too long and if the construct being measured is relatively stable, the scores at the first test would be expected to be very similar to the scores at the second test. For instance, a group of students could be surveyed about their AOD use over the past year and surveyed again two weeks later. Their reports of AOD use over the past year should not have changed much over the two weeks, so if an instrument has a high test-retest reliability, each student's score on the first test should highly correlate with his or her score on the second test. The magnitude of the test-retest correlation can be represented by a number ranging from 0 to 1, where 1 indicates perfect correlation and 0 represents no correlation. In the context of test-retest reliability, a correlation of .85 or above is generally considered high. Alternate Forms Reliability. Reliability can also be established if two alternate forms of the instrument produce highly correlated scores when administered to the same group of people. The alternate forms of the instrument must be comparable, that is they must include the same types of questions in the same format. Evaluators may want to use more than one version of a questionnaire in a situation, for instance, where surveys are repeated in a short time period, raising concern that respondents' memory of their answers may influence subsequent responses. Internal Consistency. Internal consistency is described by the correlations among items within an 6 2/18/99 10:22 AM 5 of 18 http://www.edc.org/hec/pubs/selecting-right-tool.txt instrument. If an instrument measures a single construct, all the items would be expected to be highly correlated. Likewise, if an instrument is really measuring two related but distinct constructs, the items might cluster into two groups, where those relating to one construct correlate highly with each other but less so with items in the other group and vice versa. For instance, if an instrument included questions about both binge drinking and consumption of alcohol at meals, scores on all the items about binge drinking would be expected to be highly correlated with each other but perhaps only moderately correlated with items on drinking at meals. Internal consistency is frequently represented by a statistic called Cronbach's alpha, which also can range from 0 to 1. Cronbach's alpha above .85 is considered high but a value above .70 usually is considered an indication of adequate internal consistency. Another number used to represent internal consistency is the item-to-total correlation, which between the score indicates the correlation again, ranging from 0 to 1 on any one item and the total score on the test as a whole. Generally, scores for each item should fall between .3 and .7. SECTION 2: A COMPENDIUM OF MEASURES Alcohol and Other Drug Surveys Core Alcohol and Drug Survey Key Contact: Core Institute Student Health Programs Kesnar Hall MC 6802 Southern Illinois University Carbondale, IL 62901 (618) 453-4366 Phone: Website: http://www.siu.edu/-coreinst/ Application and Outcomes: The Core Alcohol and Drug Survey was developed in 1989 to assess the nature, scope, and consequences of alcohol and other drug use on college campuses. This survey was expanded in 1994 to include assessment of students' perceptions and beliefs about alcohol and other drug (AOD) use and AOD-related sexual behavior and violence. For each behavior measured, an item score can be obtained. Quantity and frequency of use is measured by annual prevalence of use for all drugs and alcohol, 30-day use for all drugs and alcohol, average number of drinks per week, binge drinking within the last two weeks, and change in drinking and drug use in the past 12 months. Problem-related questions measure 19 consequences of drinking and drug use by frequency of occurrence. Another set of questions measures AOD-related incidents of violence. Several items also measure perception of risk, perception of others' use, perceptions of others' feelings regarding use, and perception of campus environment. Social beliefs about the effects of alcohol are measured by 14 items using a forced-choice yes/no format. There are questions that assess other campus variables, such as the extent of participation in campus activities, frequency of refusal behavior, and interference from others' drinking in a student's life on or around campus. Informants: Postsecondary student population Format and administration: The long-form version of the self-report Core Alcohol and Drug Survey Includes 39 questions on use and consequences, as well as demographics. The short-form version included 23 questions. Individual questions have between 2 and 19 response options. The most common methods of administration are to 7 2/18/99 11:38 AM 6 of 18 http://www.ecle.org/hec/pubs/selecting-right-tool.txt mail surveys to a randomly selected subset of students or to administer surveys in randomly selected classrooms. Each method is discussed in the Core Survey User's Manual, which is distributed upon purchase of the instrument. Colleges can have the results of the survey analyzed by the Core Institute. The long form of the survey takes from 20 to 35 minutes for respondents to complete. Technical Assistance: Technical assistance for those administering the Core Alcohol and Drug Survey is available from the Core Institute through a 15-minute video ($25) that describes how to conduct the survey; through the Core Institute's Website (http://www.siu.edu/departments/coreinst/public_html/index.html);or via telephone. Costs: The Core Alcohol and Drug Survey is distributed from and analyzed by the Core Institute. The costs for obtaining the long form of the survey and having the results compiled and analyzed are as follows: Purchase of each survey copy: $ 0.22 Scanning of each returned survey: $ 0.17 Executive Summary:$20.00 Abbreviated Analysis:$45.00 Disk with Raw Data:$13.00 Use in Evaluation: The Core Alcohol and Drug Survey is one of the most rigorously developed survey instruments for postsecondary populations. Its original intent was to assess only the frequency and consequences of AOD use. After hundreds of two- and four-year institutions of higher education had used the Core Survey, many requests were received to expand the content of the survey to include other aspects of campus life related to substance issues. Consequently, the survey was expanded to include questions on sexuality, campus violence, institutional climate, perceptions of AOD use, and extracurricular activity involvement. Development of both the original and expanded Core Survey followed strict American Psychological Association (APA) guidelines for test development. This instrument is not used to diagnose alcohol dependency in individuals but rather to assess the level and impact of alcohol and other drug use on campus. It is a valuable tool for determining how to target populations for prevention programming, designing social marketing and media advocacy campaigns, and assessing the impact of these prevention efforts. Validity and Reliability: Considerable evidence supports the validity of the Core Survey. To establish content validity, existing instruments and literature were reviewed to ensure that important aspects and consequences of alcohol and other drug use were adequately covered. A panel of experts then reviewed the items to assess whether they sampled the domain of interest. The level of agreement for item Inclusion among the experts was very high (interrater reliability was .90). Test-retest reliability was estimated using Pearson product-moment correlation coefficients. For items on AOD use and consequences of use, test-retest reliability was high, with the majority of item correlations falling above .80. Items on campus AOD norms showed moderate test-retest reliability, with most correlations falling between .30 and .80. Item-to-total correlations were calculated to assess the internal consistency of the survey. Correlations for the majority of items on AOD use, consequences, and campus norms fell between .30 and .70, as recommended by Henryssen. Comments: The Core Alcohol and Drug Survey is easy to administer and has substantial 8 2/18/99 11:38 AM 7 of 18 http://www.edc.org/hec/pubs/selecting-right-tool.txt validity and reliability data to support its use with postsecondary populations. Optical scan scoring of the survey is done at the Core Institute, and a report of results is provided to the institution administering the survey, generally within a week. In addition, the Core Institute can provide a computer disk with the institution's survey data in a format that will allow further data analysis. Another benefit of using the Core Survey is that institutions can compare results with those of similar institutions from across the country that have also used the instrument. Compiled results from other colleges and universities are published in Core Institute monographs every two years and are available on the Core Institute's Website. Alcohol and Other Drug Surveys Campus Survey of Alcohol and Other Drug Norms* Key Contact: Core Institute Student Health Programs Kesnar Hall MC 6802 Southern Illinois University Carbondale, IL 62901 (618) 453-4366 Phone: Website: http://www.siu.edu/-coreinst/ Applications and Outcomes: The Campus Survey of Alcohol and Other Drug Norms was developed in 1997 to assess respondents' AOD use and their perceptions of campus norms and substance use by other students. The survey was developed in response to the almost universal finding that students overestimate ADD use by their peers. This misperception is of concern to the extent that students then try to emulate the perceived norm. The first set of questions asks how frequently the respondent believes students at their institution use alcohol and marijuana and other illicit drugs. The second set of questions asks students to estimate the percentage of students who do not drink alcohol and of those who binge drink. Students are asked to report their perceptions of how much students typically consume and of places where alcohol is consumed. There is also a section that elicits respondent attitudes regarding other aspects of drinking and drug use. Finally, respondents are asked about their perceptions of campus policy and policy enforcement. Informants: Postsecondary student population Format and Administration: The self-report Campus Survey of Alcohol and Other Drug Norms consists of 17 behavioral, perceptual, and attitudinal questions and nine demographic questions. Items range from 2 to 10 response options. Respondents average 12 minutes to complete the survey. The campus survey can be combined with the Core Alcohol and Drug Survey or the Faculty and Staff Environmental Alcohol and Other Drug Survey to elicit a broad view of AOD problems on the campus. The Core Institute is available to optically scan questionnaires and analyze survey results for individual colleges and universities. Technical Assistance: Technical assistance for those administering the Campus Survey of Alcohol and Other Drug Norms is available from the Core Institute through a 15-minute video ($25) that describes how to conduct the survey; through the Core Institute's Website (http://www.siu.edu/departments/coreinst/public_html/index.html); or via 9 2/18/99 11:38 AM 8 of 18 http://www.edc.org/hec/pubs/selecting-right-tool.txt telephone. Costs: The Campus Survey of Alcohol and Other Drug Norms is distributed from and analyzed by the Core Institute. The costs for obtaining the survey and having the results compiled and analyzed are as follows: Purchase of each copy: $ 0.27 Scanning of each returned survey: $ 0.17 Executive Summary: $25.00 Cross-Tab Analysis: $30.00 Disk with Raw Data: $13.00 Use in Evaluation: While the Campus Survey of Alcohol and Other Drug Norms was not introduced on college campuses until 1997, various items on the survey have been used extensively in research and to develop social marketing programs for correcting campus misperceptions regarding AOD use and changing campus norms. The items were based on theory in AOD prevention research. Validity and Reliability: Content validity was assessed based on the level of agreement among the experts for item content. There was unanimous agreement among the experts on the choice of items. Many of the items have been evaluated for use in other instruments. Comments: The campus survey has a strong basis in theory, and many of its items have been tested in earlier research studies. In addition, results were found to be stable over a four-week test-retest period. The survey's design enhances ease of administration, scoring, and interpretation of the results. Environmental Assessment Faculty and Staff Environmental Alcohol and Other Drug Survey Key Contact: Core Institute Student Health Programs Kesnar Hall MC 6802 Southern Illinois University Carbondale, IL 62901 (618) 453-4366 Phone: Website: http://www.siu.edu/-coreinst/ Application and Outcomes: The Faculty and Staff Environmental Alcohol and Other Drug Survey, developed in 1993, consists of five subscales developed to assess faculty and staff perceptions of AOD problems on campus, awareness of policy and policy enforcement, support for programming efforts to combat AOD problems, awareness of university assessment efforts, and faculty and staff's perceived ability to identify students who are experiencing a problem and refer them for help. The basis for this instrument was the assumption that faculty and staff attitudes, perceptions, and beliefs about the extent of alcohol and other drug problems are a major influence on the campus environment. Informants: Postsecondary faculty and staff Format and Administration: The Faculty and Staff Environmental Alcohol and Other Drug Survey consists of 35 items in a forced-choice format, plus seven demographic questions. 10 2/18/99 11:38 AM 9 of 18