Cognitive task analysis: Think alouds & Difficulty Factors Assessment PDF
Preview Cognitive task analysis: Think alouds & Difficulty Factors Assessment
Unpacking & repacking Cognitive Task Analysis: expertise: Chick sexing Think Alouds and Difficulty • Experts don’t know, what they know Factors Assessment – 98% accurate after years of on-the-job training Ken Koedinger • Interviews led to design of “pictures in which critical HCI & Psychology features of various types were indicated” CMU Director of Pittsburgh • After just minutes of instruction, novices Science of Learning Center brought to 84% accuracy! Biederman & Shiffrar (1987). Sexing Day-Old Chicks: A Case Study and Expert Systems Analysis of a Difficult Perceptual-Learning Task. JEP: Learning, Memory, & Cognition. 7/12/10 Pittsburgh Science of Learning Center 1 7/12/10 Pittsburgh Science of Learning Center 2 Overview Cognitive Task Analysis • Cognitive Task Analysis • Techniques to specify cognitive –What is it? Why do it? structures & processes associated with task performance • CTA methods –Structured interviews of experts –Difficulty Factors Assessment –Think alouds of experts & novices –Think Aloud performing tasks • Hands-on exercise –Computer simulations of human reasoning 7/12/10 Pittsburgh Science of Learning Center 3 7/12/10 Pittsburgh Science of Learning Center 4 Cognitive Task Analysis Isn’t knowledge analysis Improves Instruction done for long-standing Studies: Traditional instruction vs. CTA-based academic domains? • Med school catheter insertion (Velmahos et al., 2004) – Sig greater pre to post gain • Hasn't all this been worked out? – Better with patients on all four measures used – Example: Sig fewer needle insertion attempts! • Other examples • Surely by now we understand the – Radar system troubleshooting (Schaafstal et al., 2000) content of, say, Algebra? – Spreadsheet use (Merrill, 2002) • Meta-analysis, 7 studies: 1.7 effect size! (Lee, 2004) 7/12/10 Pittsburgh Science of Learning Center 5 7/12/10 Pittsburgh Science of Learning Center 6 Need for a Knowledge Overview Decomposition Methodology • Cognitive Task Analysis • Good instruction targets the edge of students' knowledge, what is "just-learnable" –What is it? Why do it? • Need a method for decomposing a topic into • CTA methods knowledge components – What components are learners’ missing? –Difficulty Factors Assessment – What order do they acquire these components? – Which components are particularly hard to acquire? –Think Aloud – What “hidden skills” must be acquired? • Hands-on exercise • Knowledge decomposition guides design of: – problem solving activities, tutor interface, cognitive model, hints and bug messages, problem sequence 7/12/10 Pittsburgh Science of Learning Center 7 7/12/10 Pittsburgh Science of Learning Center 8 Knowledge Decomposition through Which problem is hardest for Difficulty Factors Assessment (DFA) beginning algebra students? • Goal: Identify what is "just learnable" for students at different levels of competence Story Problem • The DFA methodology: As a waiter, Ted gets $6 per hour. One night he 1. Identify possible problem difficulty factors made $66 in tips and earned a total of $81.90. - Use think aloud or analytic task analysis How many hours did Ted work? 2. Create test items & forms; Administer Word Problem 3. Analyze results: a. Main effects and interactions Starting with some number, if I multiply it by 6 and b. Strategies and errors then add 66, I get 81.90. What number did I start 4. Create a cognitive model with? 5. Create a “developmental model” or “learning progressions” Equation x * 6 + 66 = 81.90 7/12/10 Pittsburgh Science of Learning Center 9 Algebra Student Results: Part of a larger “Difficulty Story Problems are Easier! Factors Assessment” • Difficulty factors involved 80% 70% –Presentation type ct 61% re 60% •Story, Word, vs. Equation r o 42% –Unknown position C 40% nt •Result-unknown vs. start-unknown e c –Number type r 20% e P •Whole vs. decimal numbers 0% • Multiple quiz forms Story Word Equation • Detailed strategy & error analysis Problem Representation Koedinger & Nathan (2004). The real story behind story problems: Effects of representations on quantitative reasoning. In International Journal of the Learning Sciences. 7/12/10 Pittsburgh Science of Learning Center 12 More Common: Formal, Translate & Solve Strategy Informal Strategies 7/12/10 Pittsburgh Science of Learning Center 13 Algebra equations Expert Blind Spot are like a foreign Algebra teachers worst at recognizing algebra student difficulties language -- takes extensive 100 90 experience to 80 acquire % making 70 correct 60 ranking (equations 50 hardest) 40 30 20 10 0 Elementary Middle High School Teachers School Teachers Teachers Nathan, M. J. & Koedinger, K.R. (2000). Teachers' and researchers' beliefs of early algebra development. Journal of Mathematics Education Research, 31 (2), 168-190. 7/12/10 Pittsburgh Science of Learning Center 16 What’s behind expert blind Using Cognitive Task Analysis to spot? design better algebra instruction • Self-reflection on current cognition, biased memory of past learning • Inductive support strategy • Aware of verbally-mediated reasoning – Help students generalize abstract math – False inference: More words => more thinking from their own intuitive concrete solutions • Not aware of implicit processing & learning – Similar to “progressive formalization” or – Fluent algebra language processing requires “concreteness-fading” (Golstone & Son, 05) extensive implicit learning – Our minds are continually engaged in pattern • Test idea with an in vivo experiment induction, analogy, chunking, strengthening … 7/12/10 Pittsburgh Science of Learning Center 17 7/12/10 Pittsburgh Science of Learning Center 18 Parametric Study: Textbook Parametric Study: Textbook vs. Cognitively-Based Design vs. Cognitively-Based Design 5 5 Learning Due to Tutor Variants Learning Due to Tutor Variants 4 4 3 3 Pre to Post Pre to Post Improvement Improvement Score 2 Score 2 1 1 0 0 Textbook Inductive Support Textbook Inductive Support (Symbolize first) (Solve & then symbolize) (Symbolize first) (Solve & then symbolize) …Plumbing Co. charges $42 per 1. 35 + 42h = d 2. 35 + 42*3 = 161 …Plumbing Co. charges $42 per 1. 35 + 42h = d 2. 35 + 42*3 = 161 hour plus $35 for the service call 3. 35 + 42*4.5 = 224 hour plus $35 for the service call 3. 35 + 42*4.5 = 224 1. …. write an expression … 2. 35 + 42*3 = 161 1. …. write an expression … 2. 35 + 42*3 = 161 2. How much for a 3 hour call? 3. 35 + 42*4.5 = 224 1. 35 + 42h = d 2. How much for a 3 hour call? 3. 35 + 42*4.5 = 224 1. 35 + 42h = d 3. What will bill be for 4.5 hours? 3. What will bill be for 4.5 hours? 4. Find hours when bill is $140 4. 35 + 42h = 140 4. 35 + 42h = 140 4. Find hours when bill is $140 4. 35 + 42h = 140 4. 35 + 42h = 140 Koedinger, K. R., & Anderson, J. R. (1998). Illustrating principled design: The early evolution Koedinger, K. R., & Anderson, J. R. (1998). Illustrating principled design: The early evolution of a cognitive tutor for algebra symbolization. Interactive Learning Environments. of a cognitive tutor for algebra symbolization. Interactive Learning Environments. 7/12/10 Pittsburgh Science of Learning Center 19 7/12/10 Pittsburgh Science of Learning Center 20 Overview What is a Think-Aloud Study? • Cognitive Task Analysis Basically, ask a users to “think aloud” as –What is it? Why do it? they work... • CTA methods ...on a task you want to study –Difficulty Factors Assessment ...while you observe & audio or videotape –Think Aloud ...either in context (school) or in lab ...possibly using paper/storyboard/interface • Hands-on exercise you are interested in improving 7/12/10 Pittsburgh Science of Learning Center 21 7/12/10 Pittsburgh Science of Learning Center 22 The Roots of Think-Aloud The Cognitive Psychology Theory behind Think-Aloud Protocols Protocols • People can easily verbalize the linguistic • Allen Newell and Herb Simon created contents of Working Memory (WM) the technique in 1970s • People cannot directly verbalize: – Applied in ‘72 book: “Human Problem – The processes performed on the contents of WM Solving” • Procedural knowledge, which drives what we do, is outside our conscious awareness, it is “tacit”, “implicit” knowledge. • People articulate better external states & some internal goals, • Anders Ericsson & Herb Simon’s book not good at articulating operations & reasons for choice – Non-linguistic contents of WM, like visual images – “Protocol Analysis: Verbal Reports as Data” • People can attempt to verbalize procedural or 1984, 1993 non-linguistic knowledge, however, doing so: – Explained & validated technique – May alter the thinking process (for better or worse) – May interfere with the task at hand, slowing performance 7/12/10 Pittsburgh Science of Learning Center 23 7/12/10 Pittsburgh Science of Learning Center 24 Example: Think Alouds in How to Collect Data in a Statistics Tutor Development Think-Aloud Study (Gomoll, 1990, is a good guide) • Task: Exploratory Data Analysis 1. Set up observation 5. Explain that you will not – Given problem description and data set provide help – write tasks – Inspect data to generate summaries & conclusions 6. Describe tasks – recruit students – Evaluate the level of support for conclusions 7. Ask for questions before 2. Describe general purpose you start; then begin • Example Problem of observation observation In men’s golf, professional players compete in either the regular 3. Tell student that it’s OK – say “please keep tour (if they’re under 51 years old) or in the senior tour (if they to quit at any time talking” if the are 51 or older). Your friend wants to know if there is a 4. Explain how to “think participant falls silent difference in the amount of prize money won by the players in aloud” for 5 seconds or more the 2 tours. This friend has recorded the prize money of the top – give a demonstration – be sensitive to a severe 30 players in each tour. The variable money contains the money – give an unrelated desire to quit won by each of the players last year. The variable tour indicates practice task, e.g., 8. Conclude the observation which tour the player competed in, 1=regular, 2=senior. The add digits variable rank indicates player rank, 1=top in the tour. Thanks to Marsha Lovett! 7/12/10 Pittsburgh Science of Learning Center 25 7/12/10 Pittsburgh Science of Learning Center 26 Rational Task 1) Translate question into Sample Transcript statistical terms Analysis of Major L# Participants words & actions Annotation 2) Identify relevant Goals variables 60% 1 Oh, okay. 2 So we need to, he wants to know whether there is a Goal 1 3 difference in the amount of prize money, the amount of • Inspired by ACT-R 3) Characterize problem 0% 4 money won by players in the two tours. theory (variable types, etc.) 5 So, I think this is the prize money, uh, money contains the Goal 2 6 prize money won by each of these players. • Break down task: 4) Select appropriate 100% 7 Tour indicates which tour the player competes in. – 7 major goals analysis 8 Well, you don't really need rank, in order to solve this, right? 9 Cause like, well, I don't know. – Each goal has involves multiple steps or 5) Conduct analysis 100% 10 Um... I'm gonna do a boxplot... ... Goal 4 11 [Subject uses statistics package to make a boxplot] Goal 5 subgoals to perform 12 oh, cool (laugh)- I did it. – Key productions react 6) Interpret results of 100% 13 All right, uh, so just looking at the average. Goal 6 to major goals & set analysis 14 It looks like the people in the senior tour get less money. subgoals 15 Um, and there's a lot less variation in the amount of money 7) Evaluate evidence with 60% 16 that, like all the prizes. respect to question 17 A couple little outliers in each which means like, I don't 18 know, like some people won, like a lot of money at a time… Key observations about this Comparing Think 1) Translate question into 20% statistical terms verbal report Aloud Results with 2) Identify relevant Rational Task Analysis variables 60% • No evidence for goal 3 “characterize the problem” • Percentages to the right 3) Characterize problem 0% (variable types, etc.) of each step represent – Line 10: student simply jumps to selecting the percentage of a data representation (goal 4) without students in the think- 4) Select appropriate 100% analysis thinking about why. aloud study who showed explicit evidence of • No evidence for goal 7 “evaluate engaging in that step. 5) Conduct analysis 100% evidence” • Step 3 is totally absent! – A tutor can help students 6) Interpret results of 100% to do & remember to do analysis step 3 7) Evaluate evidence with 60% respect to question 7/12/10 Pittsburgh Science of Learning Center 29 Inspiration for Production Rules (Knowledge Components) Statistics Tutor • Missing production (to set goal 3): Design: Explicitly Characterize problem If goal is to do an exploratory data analysis prompts students to & relevant variables have been identified engage in critical then set a subgoal to identify variable types subgoals • Buggy production (skipping from goal 2 to 4): Select any data representation If goal is to do an exploratory data analysis & relevant variables have been identified then set a subgoal to conduct an analysis by picking any data representation 7/12/10 Pittsburgh Science of Learning Center 31 7/12/10 Pittsburgh Science of Learning Center 32 Overview • Cognitive Task Analysis Students are prompted to –What is it? Why do it? complete critical subgoal • CTA methods #3 from CTA –Difficulty Factors Assessment –Think Aloud • Hands-on exercise 7/12/10 Pittsburgh Science of Learning Center 33 7/12/10 Pittsburgh Science of Learning Center 34 Cognitive Task Analysis Try this ... Exercise • One person think aloud while solving • Use Think Aloud to design a Difficulty this problem. You can use paper. Other person is experimenter. Factors Assessment Experimenter: Remember to say “keep • Find someone next to you to work with talking” whenever participant is silent – I will give two problems – Take turns giving a think-aloud solving • Ready ... these next two problems • What is 5 ÷ 3/4 = ? 7/12/10 Pittsburgh Science of Learning Center 35 7/12/10 Pittsburgh Science of Learning Center 36 Think about student thinking ... Now this ... • Switch roles: • Which will be easier? – Other person think aloud – What’s written on paper is part of TA • Why? – Did the experimenter say “keep talking”? • Strategy & error analysis: • Ready … – What strategies will students use? – Will there be differences in strategy selection • If 5 yards of ribbon are cut into pieces that are between problem types? each 3/4 yard long to make bows, how many – What errors might account for observed differences? bows can be made? 7/12/10 Pittsburgh Science of Learning Center 37 7/12/10 Pittsburgh Science of Learning Center 38 How could you design a DFA “Latin Square” Design to test your hypotheses? 5 ÷ 3/4 = ? 7 ÷ 2/3 = ? • Can you put these two problems on the same quiz form? No Context Form 1 Form 2 – Why not? What can you do instead? • What other factors might be involved? Context Form 2 Form 1 – Size of the numbers--big nums discourage informal strategy • Don’t give problems with same answer on same form – “Tempting” nums like 6 ÷ 3/5 • Can give problems with both values of a difficulty factor – Order: context first vs. context second • Example above – Students using either Form 1 or Form 2 will get both a No- Context & a Context problem – But, two forms “counter balance” the number types 7/12/10 Pittsburgh Science of Learning Center 39 7/12/10 Pittsburgh Science of Learning Center 40
Description:The list of books you might like
