Table Of Content

DOCUMENT RESUME ED 471 654 TM 034 683 AUTHOR Tanner, David E. To Err Is Human: Decision Errors and Candidates' Competency. TITLE 2002-00-00 PUB DATE NOTE 13p. Descriptive (141) PUB TYPE Reports EDRS Price MF01/PC01 Plus Postage. EDRS PRICE Cutting Scores; Decision Making; State Programs; Teacher DESCRIPTORS Certification; Teacher Competencies; *Teacher Competency Testing; Teacher Evaluation ABSTRACT The competency testing of teacher candidates has become almost universal in the United States. This paper explores some of the issues associated with competency testing, especially as they relate to teacher testing in California. The practice has its origin in concern that some of those who choose to pursue teaching careers in the elementary and secondary schools may lack the competencies that effective teaching requires. Stated in terms of decision errors in scoring competency tests, the fear has been that some of those moving into full-time teaching may represent a population of false positives, teachers who lack necessary skills and abilities but who are certified nevertheless. The contingencies under which the competency test is actually administered in California, however, suggest greater worry over the possibility of false negatives, a concern that some of those who are denied certification may actually be qualified. (Contains 2 figures and 10 references.) (Author/SLD) Reproductions supplied by EDRS are the best that can be made from the original document. competency To err is human: Decision errors and candidates' U.S. DEPARTMENT OF EDUCATION PERMISSION TO REPRODUCE AND Office of Educational Research and Improvement DISSEMINATE THIS MATERIAL HAS ATIONAL RESOURCES INFORMATION ED BEEN GRANTED BY CENTER (ERIC) This document has been reproduced as received from the person or organization originating it. _D.E._Tanner Minor changes have been made to improve reproduction quality. TO THE EDUCATIONAL RESOURCES Points of view or opinions stated in this INFORMATION CENTER (ERIC) document do not necessarily represent 1 official OERI position or policy. David E. Tanner, Ph.D The Department of Educational Research, Administration, and Foundations California State University, Fresno BEST COPY MULE 2 Abstract become almost universal in the United The competency testing of teacher candidates has teaching that some of those who choose to pursue States. The practice has its origin in concern competencies the effective teaching and secondary schools may lack the careers in the elementary full-time fear was that some of those moving into requires. Stated in terms of decision errors, the positives, teachers who lack necessary skills and teaching may represent a population of false contingencies under which the competency test is abilities but are certified nevertheless. The over the possibility of false actually administered in California, however, suggests greater worry denied certification may actually be qualified. negatives, a concern that some of those who are 3 3 in university admissions procedures and The ubiquitous element in college and selection testing. Although employing test scores as a certification activities, is some sort of uniformly condemn educational research organizations criterion is often criticized and major selection criterion, test scores remain a fixture in many using a test score as a sole selection classify and colleges and universities, face the need to decisions. Many institutions, particularly efficiency. applicants, and do so with accuracy and screen large numbers of candidate's what they reveal about the level of a The value of test data, of course, is in traits, essential skill or knowledge. Because mental aptitude, ability, or command of some risk of directly, quantifying them involves some abilities, and knowledge cannot be measured for errors in the measurement increase the potential measurement error. Errors in mental characteristic, the requisite level of the measured decisions that are made regarding who have successful study, or who will become the most who will most likely succeed in advanced decision examine some of the issues associated with candidates. The point of this paper is to although they generalize to relate to teacher certification in California, error, particularly as they screening situations. many other selection and Error in Assessment identify what is relevant to decisions about The point of collecting candidate data is to which predetermination of some sort of reference to candidates' qualifications. This requires the candidate must order to reach a decision. When what the assessment data can be compared in defined. They discrete terms minimal competencies are know or be able to do can be specified in criteria. The processes for formal standards or some less formal set of may take the form of setting.' determining them are the domain of standard ability the minimal level of some required Particularly when it is difficult to determine applicants when there are greater numbers of qualified (What level of reasoning is sufficient?), or adopted. Norm- them, a normative reference is usually than there is opportunity to accommodate defined ranking within the group of all who fit a references specify as a required standard, some 4 'applicants scoring in the upper quartile,' are category; 'the top 20% of high school seniors,' or examples. gathered so that an Whether the standard is normative or criterion-based, data are qualify. Both approaches impose the informed decision can be made regarding which candidates criteria are used since the difficulty of risk of error, but the risk may be greater when objective be compounded by any error that might occur accurate assessment of the candidate's ability can increasing popularity of assessment in identifying the appropriate standard. In spite of the be truly comprehensive. grounded in authentic outcomes, assessment procedures can rarely regarding the larger universe of Inferences must be drawn from a sample of candidates' responses be some component of what is their skills, abilities, and knowledge. Furthermore, there may characteristic of interest. The measured that has its origin in a characteristic other than the their analytical ability. Their talent students' command of the language becomes confused with termed problem-solving ability. for calculation becomes a component of what is inadvertently candidates are classified. Such occurrences increase the potential for decision errors when reflect measurement Components of a score that are irrelevant to the named construct for example, cannot be assessed the way one assesses height, or error. Because problem-solving, Lucky guesses on a multiple choice exam, a age, there is a great potential for measurement error. test-takers to detect correct grader's scoring mistakes, a level of test-wiseness that allows which is the focus of the exercise, even the candidate's responses when they lack the construct compound the effects of errors in health can contribute to measurement error. These situations of poor quality or insufficient quantity, measurement that might occur because the test items are data are tabulated improperly. because test-takers don't understand the directions, or because the doing so, they may give rise to decision Any of these factors may inflate, or deflate a score and in errors. 5 5 Decision Errors the population of competent hi an ideal assessment world where there is no error, (Figure 1 a). This is candidates would be entirely separate from those who are incompetent competent from those who actually the inference when a cut-off score separates those who are the scoring distributions of the competent and the are not. But because of measurement error, incompetent can overlap, giving rise to decision errors. also called alpha (a) errors Decision errors fall into two categories. False positive errors, who lacks the relevant characteristic is judged to possess it. or type I errors, occur when one competentoften those at the lower end Some of those judged to be competent are actually not called beta (f)or type II errors, occur of their distribution (Figure lb). False negative errors, also level required are judged instead to when candidates who possess a necessary characteristic at the actually competenttypically those be wanting. Some of those judged to be not competent are at the upper end of that distribution. Place Figures la and lb About Here only those scores While error inflates and deflates the scores of members of either group, the candidates who are most likely nearest the cut-off score are of concern, since they represent the scores of those candidates in the lowest to be misjudged. In particular, errors that deflate of those in the highest ranks of the ranks of the competent and the errors that inflate the scores incompetent provide the greatest probability of misclassification. The Horns of a Dilemma false positive and false Part of the difficulty in making candidate decisions is that negatives can be minimized negative classifications are inherently related. The number of false false positive classifications. by lowering the cut-off score, but the companion will be more but only by Correspondingly, false positives can be minimized by raising the cut-score, 6 6 Therefore, a critical element in standard- increasing the likelihood of false negative decisions. positive or false negative errors pose the greater setting procedures, is deciding which of false threat. Choosing One's Error depends upon circumstances and is Which of type I or type II error is more acceptable the competency of prospective surgeons are manifest in the rigor of the standard. Those judging patients II than type I errors. The safety of prospective more willing to accommodate type seek On the other hand, when landed immigrants requires that standards be fixed relatively high. given an oral examination of their understanding naturalization as United States citizens they are determine that the occasional false positive of U.S. government. Decision-makers may understand the system) is preferable to denying classification (the individual really does not quite does not reasonable understanding of the political system, but someone citizenship who has a of the questions. speak English well enough to respond to all of the nuances clear-cut and questions about which type of In many other instances, the issues are less issues. But the grounded in politics as much as in measurement error is more palatable are in the rigor of the standard. Teacher preference for one type of error over another is implicit Twenty years ago, much of the impetus for certification in California represents a case-in-point. candidates are among the least educational reform came from criticisms that teacher Weaver, 1981). In the wake of the academically talented of university students (Nelli, 1984; with particular attention on candidates' charges, nearly every state adopted competency tests academic, rather than their pedagogical ability. continues to be a driving force behind the As long as concern about teacher competency and (Barth, 2000; Hextall, Mahoney, and Menter, 2001; Song use of screening instruments less objectionable. For the Christiansen, 2001) the suggestion is that false negative errors are exclude someone who may actually sake of the learner, it would seem better to occasionally skills, than provide a credential to someone who does not. possess the required literacy 7 7 objectives, standpoint of educational reform Besides being less objectionable from the false negative decision: allows the possibility of correcting a there is a compensation built in that be rectified, it also Besides allowing for a decision error to the test-taker may repeat the test. interim to candidates who may have worked in the allows a second chance opportunity to standards they retesting, candidates may be able to meet develop the measured competencies. On failed to meet in an initial trial. competency test can correct for false Although allowing repeated attempts to pass a option continuing possibility of a false positive? One negatives, what adjustment is made for the that when repetition of the test. Millman (1989) explained is to adjust the cut-off score with each false positive decision over successive the required score remains fixed, the risk of a point that, as repetitions increase, there is a very administrations of the test accumulates to the practical demonstration of this potential, Huynh high risk of false positive classifications. In a show that multiple retakes with a fixed cut-off score (1990) used high school exit exam data to positives to a probability of nearly 1 with repeated reduce false negatives to near 0 and false attempts at the test. is witness to concerns that some of Although the very appearance of competency tests errors), a is, they represent false positive classification those who teach are not competent (that modified. that the original decision may have been fixed cut-off score for the CBEST suggests fact that is for first time test-takers, in spite of the The criterion is the same for repeaters as it with the testing the instrument and have greater familiarity repeaters have seen some version of the first time.2 procedure than those who are taking the test for underscore this position. First: Only the failed Two other elements of CBEST procedure false negative can be corrected by repeating portion need be repeated suggesting that although a fact for a false positive. Second: In spite of the the test, there is no parallel opportunity to correct provide independent measures of discrete skills, to some that the three sub-tests are intended to allowed to compensate for scores below degree, higher-than-minimum scores in one area are criterion in another. 8 8 Implications graduation from high Whether intended for those who aspire to teach, to the bar, or to decisions are made. Although our school, assessments provide the data from which classification of measurement data focus has been on testing data and classroom teacher certification, any type decisions at risk. Decision errors are is subject to error which places the resulting classification potential for the other. The type related in that one type can only be minimized by increasing the is a substantial risk to some that is less objectionable depends upon circumstances. When there the education innocent party, for example, the tendency is to favor false negatives. However, the rationale for the reform movement suggests that there is sometimes a discontinuity between tests were imposed assessment and the mechanics of its implementation. Teacher competency in terms of decision because of concerns that teachers lacked basic literacy skills. Expressed who become credentialed teachers represented false errors, the concern was that some of those adjust selection positive classifications and the point of the testing appears to have been to administration and procedures to be more tolerant of false negatives. But the conditions of test decisions. data interpretation indicate a relatively greater sensitivity to false positive for what At this point it seems unlikely that procedures will be adjusted to compensate false positive classifications, but such procedures are are almost certainly significant numbers of that it is certainly available. Besides sliding the cut-off score, Millman (1989) has noted original score, or designate a "no-decision" band in common to average the repeat scores with the the area where most decision errors are made. been In teacher candidates we have an assessment situation in which procedures have of this adjusted to minimize false negatives. Although their discussion is beyond the scope that correlate with the candidate's ethnic group paper, teacher shortages, and scoring patterns Competency testing may allow decision appear to be the driving issues in those decisions. reflect literacy, makers to determine whether candidates have the skills and understandings that 9 9 positive decision errors may undermine but what appears to be an institutional bias favoring false the original intent.