Evaluation of Learning Efficiency and Efficacy in a Multi-User Virtual Environment Doug Hearrington Kennesaw State University Abstract This study evaluated the multi-user Moodle (Modular, Object-Oriented, (a) Promote, support, and model virtual environment (MUVE) known Dynamic Learning Environment) is a creative, innovative thinking and as Second Life, integrated with Moodle free and open source learning manage- inventiveness and SLOODLE technologies, as an ment system similar to Blackboard or (b) Engage students in exploring real- exploratory course delivery platform WebCT. The researcher tested Moodle world issues and solving authentic and for its ability to enable teachers to in this study because it interfaces with problems using digital tools and meet elements of NETS•T. Graduate Second Life via the plugin known resources student participants (N = 17) inter- as SLOODLE (Simulation-Linked, (c) Promote student reflection using acted, constructed simulated schools, Object-Oriented, Dynamic Learning collaboration tools to reveal and and attended classes in the MUVE. Environment). Using these two tech- clarify students’ conceptual under- The researcher used pre- and posttest nologies with Second Life allows users standing and thinking, planning, measures of self-efficacy and learning to blog, take quizzes, submit three-di- and creative processes efficiency to understand the effects of mensional objects to drop-boxes, and (d) Model collaborative knowledge con- the MUVE on participants and their much more. The ability to integrate struction by engaging in learning rate of learning to make educational the Second Life MUVE with a learn- with students, colleagues, and others use of the environment. Findings imply ing management system (Moodle) in face-to-face and virtual environ- that the technologies have potential as via SLOODLE provided an additional ments (ISTE, 2008) a distance-learning platform and as a reason to test the functionality of these tool to meet elements of NETS•T. Pre- technologies to serve as a distance The second NETS•T standard is: “De- paring teachers to use the MUVE for education platform. sign and develop digital age learning ex- these purposes is likely to require a sig- Another compelling reason to use a periences and assessments.” Two of this nificant amount of scaffolding. (Key- MUVE in teacher education programs standard’s performance indicators clearly words: MUVE, Second Life, distance is that the technology may facilitate lend themselves to using a MUVE: education, self-efficacy) the implementation of specific program standards. ISTE has, in conjunc- (a) Design or adapt relevant learning I n recent years, an increasing number tion with a wide variety of professional experiences that incorporate digital of colleges, universities, and edu- education organizations, established tools and resources to promote cational institutions have started to the National Educational Technology student learning and creativity use multi-user virtual environments Standards for Teachers (NETS•T). (b) Develop technology-enriched (MUVEs) to host online classes and Teacher-education institutions ac- learning environments that enable provide online content and provide credited by the National Council for all students to pursue their indi- virtual educational interactions. Second Accreditation of Teacher Education vidual curiosities and become active Life, a popular free and commercially (NCATE) use the NETS•T as the participants in setting their own available MUVE, has attracted the standards by which to measure teacher educational goals, managing their attention of educational institutions readiness to use information and com- own learning, and assessing their worldwide and is being used by at munication technologies (ICTs). A own progress (ISTE, 2008) least 142 colleges and universities, 41 MUVE may help a teacher implement for profit and nonprofit educational the first two NETS•T standards and A MUVE is a form of virtual reality organizations, eight libraries, and four several of the performance indicators (VR), a computer-based technology that museums (SimTeach, 2008). Second for those standards. The first NETS•T provides the visual, aural, and tactile Life is also finding increasing favor standard is: “Facilitate and inspire stimuli of a virtual world generated in among secondary schools, and schools student learning and creativity.” This real time (Sanchez, Lumbreras, & Silva, across the globe are using it for educa- standard contains four performance 2001). A MUVE is an example of inter- tional purposes. This study utilized the indicators, each of which may be ac- active multimedia in which experiences MUVE provided by Second Life. complished using a MUVE: are made possible by dynamic elements Volume 27 Number 2 | Journal of Digital Learning in Teacher Education | 65 Copyright © 2010, ISTE (International Society for Technology in Education), 800.336.5191 (U.S. & Canada) or 541.302.3777 (Int’l), iste@iste.org, www.iste.org. All rights reserved. Hearrington under the user’s control (Rieber, 2005). Barriers to learning in the Second relationship with her or his ability to For example, rather than reading about Live MUVE include a “high learning learn to build a simulation in a MUVE. the surface of the Moon, one can repli- curve,” which is reflected in its compli- This study advances the field in this area cate the environment in a MUVE and cated user interface, high difficulty level by providing a first step in this direc- students can control their own explora- of building new objects for novice users, tion. In addition, this study represents tion of a virtual Moon, thus providing a perceptions that working in the Second an attempt to pilot-test an instrument to form of experiential learning. Addition- Life MUVE was too time-consuming, measure self-efficacy for using a MUVE. ally, users may interact with objects and and technical difficulties (Sanchez, 2009). events in simulations in MUVEs and Some users have expressed disappoint- Self-Efficacy similar three-dimensional environments ment with the environment because of Perceived self-efficacy is one’s belief in (Cobb & Fraser, 2005). their expectations that it should be more one’s ability to complete actions required Users of a MUVE experience three like a playing a game. When Second Life to produce a result or to accomplish a “presence layers” in a 360-degree, three- turned out to not be a game, these users given task (Bandura, 1997). Self-efficacy dimensional environment, with the became bored and frustrated (Sanchez, influences the careers people pursue, effect of combining physical and virtual 2009). Due to the limitations of moni- the level of effort they invest in a given realism in the virtual space to produce tors, three-dimensional MUVE envi- endeavor, their resilience to adversity, an immersive experience that “conveys ronments are viewed on a flat screen, and the level of accomplishments they a feeling of being there and a strong creating a lack of peripheral vision. This achieve (Bandura, 1997). Therefore, it is sense of co-presence when other avatars lack of peripheral vision detracts from possible that self-efficacy will be related are present” (Warburton, 2009, p. 6). the representation of the environment to learning efficiency or the perceived The three layers of presence are physi- (Cobb & Fraser, 2005). Time delays difficulty of using the environment. For cal, communication, and status (War- are often common when using such an this purpose, the researcher adminis- burton, 2009). The visual and physical environment, and depth perception is tered a measure of general computer proximity of avatars to one another difficult for some users to learn when in self-efficacy and created and adminis- creates the physical presence. Spatially a three-dimensional environment (Cobb tered a specific self-efficacy instrument enhanced voice that allows one to sense & Fraser, 2005). related to using a MUVE. the direction from which another voice Positive aspects of using the Second originates, as well as communication via Life MUVE have also been noted. Users Cognitive Load Theory synchronous text chat and asynchronous enjoy creating and designing their avatar Because users interact with a MUVE’s text communications, such as e-mail and the feelings of creativity and accom- three-dimensional world and other users and group notices, create the commu- plishment they experience when build- by means of avatars, these environments nication presence. Finally, various tools ing in the environment (Sanchez, 2009). may present a higher level of cogni- that allow one to know when a friendly Additionally, users have reported having tive load than the more typical two- avatar is “in-world” or offline create a strong attachment to their avatar and dimensional, non-avatar, computer-user a status presence (Warburton, 2009). enjoyed communicating with others via interfaces with which most are familiar. Presence helps to establish a first-person their avatar. Some have noted that the Cognitive load refers to the level of diffi- experience when one is in a MUVE. This sense of enjoyment and creativity they culty a learner perceives when perform- experience has been said to help develop experience in this MUVE outweighs the ing or learning a given task. Cognitive direct, subjective, and personal knowl- sense of frustration they feel from the load theory is based on information edge (Sanchez, Lumbrera, & Silva, 2001). complexity of the user interface and the processing theory, which states that an technical issues associated with using individual must process information Affordances and Barriers to MUVE Use Second Life (Sanchez, 2009). using short-term memory in order to MUVES may be said to have structural The immersive nature of a MUVE place it into long-term memory for later and functional affordances. Structural combines these physical, social, and cul- use. If the cognitive load is too great, affordances consist of (a) a collection of tural dimensions to provide a space in the information will not be processed objects that model the mathematical/ which compelling simulations and role- into long-term memory efficiently, and physical properties of the domain, (b) playing activities may take place (War- learning will be inhibited. Cognitive links to numerous representations of the burton, 2009). However, research on load theory is a set of principles used underlying model, (c) opportunities to the use of modern MUVEs as distance by researchers to study the three types use the objects in complex ways, and (d) education platforms is in its infancy, of load: (a) intrinsic load, which is the challenges or activities for the student to and little is known about the “learning mental work imposed by the complexity solve or explore (Edwards, 1995). Func- curve” educators may experience when of a task and is largely determined by tional affordances include the interaction trying to create simple simulations in one’s goals; (b) germane (relevant) load, between the user, the software, and the such an environment or whether an which is the mental work imposed by setting in which the environment is used. educator’s computer self-efficacy has any an instructional activity that benefits the 66 | Journal of Digital Learning in Teacher Education | Volume 27 Number 2 Copyright © 2010, ISTE (International Society for Technology in Education), 800.336.5191 (U.S. & Canada) or 541.302.3777 (Int’l), iste@iste.org, www.iste.org. All rights reserved. Learning Efficiency and Efficacy in a MUVE Learning efficiency is a function of Performance achievement and mental effort, but it cannot tell us, by itself, how much more E = 0 quickly one can perform a task after 1.0 initial familiarity and over time. This may be accomplished by calculating 0.8 the learning curve. Learning curve is the measurement of time to complete High Efficiency 0.6 a specific task correctly over time, after practice. An illustration of the learning 0.4 curve, taken from The Learning Curve Deskbook (Teplitz, 1991), assumes a 0.2 piano student was learning to play the Mental Effort “Minute Waltz” by Chopin. The first time she played the piece, it took 3 minutes -1.0 -0.8 -0.6 -0.4 -0.2 0.2 0.4 0.6 0.8 1.0 to play. Her second attempt took 2.6 -0.2 minutes. Attempt number three took 2.37 minutes. The fourth attempt took -0.4 under 2 minutes. The rate of the student’s improvement, calculated every other attempt (attempt 1, 2, 4, 8, etc.), is 21%. -0.6 Low Efficiency This means that each doubling results -0.8 in an improvement of 79%. Although time it takes the student to succeed gets smaller and smaller, rates of improve- -1.0 ment tend to remain the same. This has been shown to remain constant in a variety of learning, manufacturing, and business situations (Teplitz, 1991). The Figure 1. The Learning efficiency grid. learning curve has been used in manu- learning of the task; and (c) extraneous for PME that are above the mean are facturing to help calculate production (irrelevant) load, which is mental work plotted to the right of the origin of the time and cost, to forecast labor require- that is irrelevant to the learning goal and grid. Z-scores for achievement (comple- ments, and as a metric by which manag- wastes limited mental resources (Clark, tion time) that are above the mean are ers monitor production (Yelle, 1979). Nguyen, & Sweller, 2006). Cognitive reversed, because negative scores repre- The same concept may also be applied load is operationalized in this study as sent higher achievement. The hypotheti- to calculate the “forgetting curve” of a perceived mental effort (Clark, Nguyen, cal line of zero efficiency runs diagonally task (Bailey, 1989). The learning curve & Sweller, 2006). The level of perceived from a point at the lower left of the grid has been used to study the improvement mental effort can enhance or inhibit (quadrant III), through the origin, to the in learning of computer-aided design one’s learning efficiency. upper right of the grid (quadrant I) along (CAD) students over time (Hamade, Any task that can be accomplished, a line that would extend through points Artail, & Jabar, 2005) and the improve- such as learning to perform a mathemati- (-1, -1) and (1, 1). See Figure 1 for an ment in disease pattern recognition and cal calculation or learning to build a chair example of the learning efficiency grid. diagnosis by medical students over time in a virtual world, can be measured in Low learning efficiency occurs when (Williams, Klamen, & Hoffman, 2008). terms of learning efficiency. Learning one learns something slowly with great The learning curve is useful for describ- efficiency is the relationship between a mental effort. High learning efficiency ing and studying tasks that require both measurement of achievement, such as a occurs when one learns something procedural and declarative knowledge test score or the amount of time it takes quickly and with low mental effort (Hamade, Artail, & Jabar, 2005). There- a learner to correctly perform a task, and (Clark, Nguyen, & Sweller, 2006). Thus, fore, it makes sense to apply the learning the perceived mental effort (PME) of learning efficiency can potentially be curve to the learning of the mostly pro- the learner. To measure learning effi- used to calculate the learning “curve” cedural knowledge required to perform ciency, achievement and PME scores are of a given task over time, to compare tasks in a virtual world. Learning curve converted to z-scores. Both are plotted on different programs to one another, or to may be calculated using various slope a Cartesian grid with PME on the X-axis compare learning in a variety of com- formulae (Yelle, 1979; Teplitz, 1991) or as and achievement on the Y-axis. Z-scores puter or non-computer conditions. a percentage of improvement. Volume 27 Number 2 | Journal of Digital Learning in Teacher Education | 67 Copyright © 2010, ISTE (International Society for Technology in Education), 800.336.5191 (U.S. & Canada) or 541.302.3777 (Int’l), iste@iste.org, www.iste.org. All rights reserved. Hearrington Based on the increased interest Table 1: Participant Demographics in MUVEs in higher education as a Standard Mean Deviation Range distance-learning platform, the potential of the environment to be used to meet Age in Years 32.6 6.3 25 - 47 NETS-T standards, and the increasing Years of Teaching Experience 9.0 5.7 1 - 21 use of MUVEs at all levels of education for building and using simulations, the The Computer User Self-Efficacy 10-point scale, with a 1 representing low researcher designed this study to answer Scale (Cassidy & Eachus, 2002) mea- confidence and a 10 representing high these questions: sures general computer self-efficacy confidence (Bandura, 1997). (GCSE). This scale consists of two parts: The researcher devised the “Maze 1. How efficient are new participants at (a) individual characteristics and (b) Task” as a means of measuring the basic creating and working in the MUVE, computer self-efficacy items. The indi- skills commonly required to utilize a and how does their efficiency change vidual characteristics section contains MUVE. The researcher designed a maze over time? seven items that ask the participant that required participants to navigate 2. What is the relationship between about whether they have attended a their avatars through a door, turn and participant general computer self-ef- computer course or own a computer, walk in various directions, and fly over ficacy and MUVE self-efficacy before basic demographics (age and gender), and land on the other side of a wall. Ad- and after using the environment for experience with computers, and types of ditionally, a sign in the maze required a period of time? Does using the software packages they have used. The them to take a picture of their avatar’s MUVE result in increased GCSE and computer self-efficacy section contains face and e-mail it to me to measure MUVE-SE? Does GCSE or MUVE- 30 items, each with a 5-point Likert mastery of changing the camera (user’s) SE predict learning efficiency? scale ranging from strongly disagree to point of view. Next, a sign in the maze 3. What were the participants’ impres- strongly agree. The researcher worded directed participants to answer a question sions of their user experience within approximately half of the items posi- by writing their answer on a note card the Second Life MUVE? tively and half negatively. The researcher of their own creation. They then had to reverse-coded the negatively worded deposit the note card into a drop box and METHOD items for analysis purposes. Example obtain a different note card from a dis- Participants items include: “Computers are far too penser. Finally, participants had to click The researcher collected data from 17 complicated for me,” and “Most difficul- on a teleporter to transport to another graduate students enrolled in a master’s ties I encounter when using computers, location within the virtual world. Users degree program in educational leader- I can usually deal with.” The internal commonly perform each of these tasks ship in a southeastern state university. consistency of the 30-item scale has when using the Second Life MUVE for The students were enrolled in the same been reportedly very high (α = 0.97, N = educational purposes. section of a class and therefore repre- 184). Test-retest reliability has been re- Finally, the researcher devised the sent a convenience sample. Two of the portedly high and statistically significant “Chair-Building Task” as a means of participants were male. Eight identi- (r = .086, N = 74, p < 0.0005) (Cassidy & measuring participants’ basic building fied themselves as African-American, Eachus, 2002). skills. The researcher gave participants and the remaining seven identified The researcher created the MUVE a model of a chair and asked them themselves as Caucasian. None of the Self-Efficacy (MUVE-SE) instrument to to replicate the design. Instructions participants had any experience using a measure the multi-user virtual environ- indicated that the chair did not have to MUVE or a similar environment before ment self-efficacy. This study represents have identical measurements or dimen- this study. Table 1 summarizes other an initial pilot test in the development sions, just an identical design. The chair demographic characteristics of the of the MUVE-SE. The MUVE-SE was to include four legs, a square seat, participants. contains 18 items designed to measure and a back. The back was to consist of a participant’s efficacy at performing two vertical braces connected by two Instruments tasks typically required when they use horizontal slats. Cognitive load was measured for two the Second Life MUVE as a learning Participants also synchronously at- defined tasks within the MUVE using environment. Example items include: “I tended two classes entirely in the virtual an established 9-point scale ranging believe I can teleport to other locations,” environment and met with their team- from 1 for very little mental effort to and “I believe I can move objects I create mates outside of regularly scheduled 9 for a great deal of mental effort to in Second Life.” Initial responses to the class time to discuss their assignments achieve a task (Clark, Nguyen, & Swell- 18 items were on a “Yes” or “No” binary and work on their building project. er, 2006). These two tasks—the Maze scale. Participants responding “Yes” to During class sessions, the instructor Task and the Chair-Building Task—are any item were then asked to rate their presented information verbally and described below. level of confidence in their ability on a using a PowerPoint slideshow. The 68 | Journal of Digital Learning in Teacher Education | Volume 27 Number 2 Copyright © 2010, ISTE (International Society for Technology in Education), 800.336.5191 (U.S. & Canada) or 541.302.3777 (Int’l), iste@iste.org, www.iste.org. All rights reserved. Learning Efficiency and Efficacy in a MUVE Table 2: Completion Times (Performance) and Perceived Mental Effort (PME) scores During the 6 weeks in which par- Maze Task Chair-Building Task ticipants used the MUVE, they worked Measurement M SD M SD in teams of three or four to build a Pretest simulated digital age classroom that Performance 12.88 6.92 16.55 5.66 would promote high levels of technology PME 5.41 2.06 7.41 1.94 integration based on assigned readings Posttest and each group’s literature review of Performance 6.65 2.77 11.79 4.79 technology integration and 21st century PME 4.35 1.91 4.94 2.01 or digital age classroom design topics. Note. n = 17 for the pretest and posttest. The instructional strategy used was a Table 3: Pre- and Posttest Learning Efficiency Scores combination of problem-based learn- Maze Task Chair-Building Task ing and cooperative learning groups, in which each team had the same digital M SD M SD age classroom building assignment. In Pretest -0.5419 1.4222 -0.6702 1.0468 addition to building a simulated learning Posttest 0.5415 0.9379 0.6690 1.1728 environment (SLE), teams were required Note. n = 17 for the pretest and posttest. to itemize the elements they believed Table 4: Mean Learning Curve Improvement and Weekly Improvement Percentages were important to the design of such Maze Task Chair-Building Task a learning environment and to explain how they addressed each element in their Total Improvement Weekly Improvement Total Improvement Weekly Improvement design as part of a presentation to their Achievement 48 % 8 % 28 % 4.6 % classmates. Participants uploaded Pow- PME 19 % 3 % 33 % 5.5 % erPoint presentations into Second Life Learning Efficiency 1.6 % .2 % 1.9 % .3 % and delivered their presentation inside of the MUVE. Each member of the team instructor also showed students a video GCSE, MUVE-SE, the Maze Task, and designed and helped construct the SLE, during class in the MUVE. Participants the Chair-Building Task. The researcher contributing approximately 100 objects to conducted small-group discussions in recorded the start and end times of the its construction. These objects are known the environment and submitted a simple tasks as well as the participants’ PME in Second Life as “prims” which is short assignment on a note card into a drop associated with both of these tasks. for “primitive” objects. A “prim” starts out box. To evaluate SLOODLE, participants The researcher measured learning as a basic three-dimensional shape, such used the quiz tool and the choice tool efficiency using the following formula as a cube, and is then transformed by (which allows users to vote and see the (Clark, Nguyen, & Sweller, 2006): participants into a variety of shapes that, three-dimensional results of their vote). when combined together, form complete Average Performance in Z-Scores – Finally, participants uploaded a Pow- objects. For example, a bookshelf with Average Difficulty Rating in Z-Scores erPoint presentation into the MUVE three shelves, two sides, and a base may and loaded it into a perpetual slideshow consist of six “prims.” Each SLE contained √2 viewer outside of their building. They 300–500 prims. did this to explain their design and their To conduct the learning efficiency At the end of the study, participants rationale for their decisions to their calculation, the researcher measured again completed the GCSE, MUVE-SE, classmates and their instructor. performance on the maze and chair Chair-Building Task, and Maze Task. tasks based on the amount of time it Additionally, the researcher adminis- Design and Procedure took participants to complete each task. tered a questionnaire designed to elicit This pilot study used a pre-experimental, The researcher measured the difficulty their impressions of the MUVE as a one-group pretest-posttest design (Camp- of each task using the PME scale (Clark, learning environment and as a place in bell & Stanley, 1963). Participants were Nguyen, & Sweller, 2006). The research- which to build virtual simulations. introduced to the MUVE in class and er calculated learning efficiency for each Finally, the researcher calculated a received a guided 3-hour practice session, of two performance tasks: (a) naviga- learning curve for the percentage of im- during which they were introduced to all tion through a maze in the MUVE provement in achievement (completion of the basic skills measured in the maze and (b) building a simple chair within time), PME, and learning efficiency over and chair tasks. During this time, partici- the MUVE. The researcher calculated the 6-week period of this investigation. pants built a chair identical to the one in learning efficiency after participants had the Chair-Building Task for practice. spent 3 hours in the MUVE (pretest) Analysis After this introductory session, and then after spending 6 weeks using The researcher analyzed the quantitative the researcher took measurements for the MUVE (posttest). data using SPSS and Microsoft Excel. Volume 27 Number 2 | Journal of Digital Learning in Teacher Education | 69 Copyright © 2010, ISTE (International Society for Technology in Education), 800.336.5191 (U.S. & Canada) or 541.302.3777 (Int’l), iste@iste.org, www.iste.org. All rights reserved. Hearrington The researcher transcribed the open- ended questionnaire responses, coded them, entered them into a qualitative Maze Task Posttest Performance research program (HyperResearch), (0.51, -0.26) and analyzed them using the constant E = 0.5415 0.6 comparative method to shed light on the quantitative findings. The researcher used a combination of 0.4 procedures to answer research question Chair Task Posttest 1, “How efficient are new participants at (0.42, -0.53) 0.2 creating and working in the MUVE, and E = -0.6690 Mental Effort how does efficiency change over time?” First, the researcher calculated pretest -0.6 -0.4 -0.2 0.2 0.4 0.6 Chair Task Pretest and posttest learning efficiency for both 0.2 (-0.42, 0.53) the Chair-Building and Maze Tasks. The E = -0.6702 researcher conducted paired sample t-tests to determine whether pre- and 0.4 posttest scores on both tasks (completion time, PME, and learning efficiency) 0.6 Maze Task Pretest were statistically different from one (-0.51, 0.26) another. The researcher calculated effect E = -0.5415 sizes using Cohen’s d. Additionally, the researcher calculated the learning curve Figure 2. Learning efficiency plot. based on the percentage of improvement participants achieved over the 6-week instructions was easy, and in about ments for the participants after 3 hours period of the study. 2.5 hours, the three programs were of experience (pretest) in the MUVE and Research question 2 contained three “talking” together. Once set up was again after 6 weeks of experience (post- parts. The researcher performed Pearson complete, participants were able to test). Table 2 (p. 69) provides means and correlation analyses and paired sample use SLOODLE’s quiz tool, voting tool, standard deviations. t-tests to answer part 1 of the question, and toolbar when attending a virtual The results of two-tailed, paired- “What is the relationship between par- class in Second Life to facilitate asking sample t-tests comparing the Maze Task ticipant general computer self-efficacy and answering questions by raising pretest performance and PME scores and MUVE self-efficacy before and after their avatars’ virtual hands to get the with the Maze Task posttest perfor- using the environment?” The researcher instructor’s attention. The instructor mance and PME scores showed that used paired sample t-tests were used to gave one quiz and took one vote dur- participants’ performance times were answer part 2 of the question, “Does using the project, and the technologies significantly better on the posttest (t[16] ing the MUVE result in increased GCSE worked together perfectly to transmit = 4.33, p < 0.01) and had a large effect and MUVE-SE?” Finally, the researcher each participant’s score and vote from size (d = 1.14). However, participants’ used linear regression analyses to answer Second Life, through SLOODLE, and PME was not significantly lower on the the third part of question 2, “Does GCSE into Moodle for final recording. The posttest (t[16] = 1.62, p = 0.12). or MUVE-SE predict learning effi- combined functionalities of SLOODLE Two-tailed, paired-sample t-tests com- ciency?” The researcher calculated effect and Moodle seem to offer educators paring the Chair Task pretest performance sizes using Cohen’s d. a viable, although not yet fully devel- and PME scores with Chair Task posttest The researcher conducted an analy- oped, means of using Second Life as a performance and PME scores indicated sis of open-ended survey questions for distance education platform in combi- that participants’ performance times were trends and commonalities to answer nation with the learning management significantly better on the posttest (t[16] research question 3, “What were the system capabilities of Moodle. = 3.01, p < 0.01) with a large effect size (d participants’ impressions of their user ex- = .88). Results also indicated that partici- perience within the Second Life MUVE?” Learning Efficiency pants’ PME was significantly lower on the The researcher measured learning effi- posttest (t[16] = 4.02, p < 0.01) with a large Results ciency for two tasks—the Maze Task and effect size (d = 1.21). Moodle and SLOODLE the Chair-Building Task—using comple- Table 3 (p. 69) presents means and The integration of Moodle, SLOODLE, tion time as a measure of performance standard deviations for learning ef- and Second Life worked quite well. and PME as a measure of cognitive load. ficiency. A two-tailed, paired-sample Following the printed and video set-up The researcher took these measure- t-test indicated that participants were 70 | Journal of Digital Learning in Teacher Education | Volume 27 Number 2 Copyright © 2010, ISTE (International Society for Technology in Education), 800.336.5191 (U.S. & Canada) or 541.302.3777 (Int’l), iste@iste.org, www.iste.org. All rights reserved. Learning Efficiency and Efficacy in a MUVE Table 5: GCSE and MUVE-SE Means and Standard Deviations a moderate amount. The same analyses GCSE MUVE-SE for the effects of posttest MUVE-SE on M SD M SD posttest Maze Task learning efficiency Pretest 156.18 12.52 101.29 38.95 were statistically significant (F[1, 16] = 19.802, p < .001) with a strong R2 of Posttest 162.47 11.52 157.76 19.03 .569. The posttest MUVE-SE predicted Note: n = 17 more than 56% of the variance in posttest Maze Task learning efficiency. The significantly more efficient in the Maze Table 5 provides means and stan- linear regression analysis for posttest Task (t[16] = 3.14, p < 0.01) after 6 weeks dard deviations for the GCSE and MUVE-SE and Chair-Building Task in the MUVE, with a large effect size MUVE-SE measures. learning efficiency was also statistically (d = -.90). Similarly, participants were The researcher performed Pearson significant (F[1, 16] = 6.663, p = .021) significantly more efficient in the Chair- correlations to determine the relation- and had an R2 of .308. This indicates that Building Task (t[16] = 4.08, p < 0.001) ship between GCSE and MUVE-SE the posttest MUVE explained 30% of after 6 weeks, with a large effect size (d pre- and posttest scores. The researcher the variance in posttest Chair-Building = -1.20). Contrary to typical effect size calculated two-tailed tests for signifi- Task learning efficiency. The MUVE-SE, interpretation, due to the use of standard- cance within SPSS for each correlation. when administered after participants ized scores and the way learning effi- Results indicate that pretest GCSE scores spent 6 weeks in the multi-user virtual ciency is calculated, the negative signs on are not significantly correlated with pre- environment, appeared to explain a these two effect sizes indicate that a large test MUVE-SE scores (r = .097, p = .712, moderate amount of the variance in positive improvement was realized. n = 17). Additionally, posttest GCSE participants’ learning efficiency on both Mean learning efficiency pretest and scores are not significantly correlated the Maze and Chair-Building Tasks. posttest scores are plotted in Figure 2 for with posttest MUVE-SE scores (r = .301, Paired-sample t-tests show that the Maze Task and Chair-Building Task. p = .240, r = 17). participants’ improvement in GCSE was Both learning efficiency scores indicated Linear regression analyses indicate that significant (t[16] = 2.25, p < 0.05) and low efficiency on the pretest and higher pretest GCSE scores do not predict pretest had a moderate effect size (p = -0.52). achievement based on the posttest scores. learning efficiency on either the Maze Participants’ improvement in self-effica- Table 4 (p. 69) presents learning Task (F[1, 16] = 0.159, p = .696) or the cy for using a multi-user virtual envi- curve values in percentages. These val- Chair-Building Task (F(1, 16) = 1.749, p = ronment was also statistically significant ues show the percentage of improvement .206). Similarly, pretest MUVE-SE scores (t[16] = 7.16, p < 0.001) and had a large for the Maze and Chair-Building tasks, do not predict pretest learning efficiency effect size (d = -1.84). It appears that as measured by achievement (comple- on the Maze Task (F[1, 16] = 0.811, p = using the Second Life MUVE for a pe- tion time), PME, and learning efficiency .382) or the Chair-Building Task (F[1, 16] riod of 6 weeks does result in increased over the 6-week duration of the investi- = 0.020, p = .890). Neither the GCSE nor GCSE as well as increased self-efficacy gation and as weekly averages. the MUVE-SE demonstrated the ability to related to using the MUVE. predict learning efficiency when adminis- Self-Efficacy tered as a pretest. Participant Impressions The researcher tested the 30-item instru- The researcher performed another The researcher asked the 17 partici- ment that the researcher used to measure set of linear regression analyses to pants to fill out a survey containing 14 General Computer Self-Efficacy (GCSE) determine whether the GCSE instru- open-ended items designed to elicit and the 18-item Multi-User Virtual ment or the MUVE-SE instrument the details of their experiences in the Environment Self-Efficacy (MUVE-SE) could predict learning efficiency when MUVE. Any names mentioned here are instrument for reliability using Chron- administered after participants had pseudonyms. The first item asked about bach’s α. The GCSE instrument showed spent 6 weeks (posttest) using the the presence of any technical issues that a high degree of reliability when the MUVE. Results indicated that posttest may have hindered or prevented use of researcher administered it at the begin- GCSE scores did not predict posttest the environment. No one reported dif- ning of the investigation (α = .840, n = learning efficiency on the Maze Task ficulty downloading and installing the 17) and again when it was administered (F[1, 16] = 3.235, p = .092). However, software on their home computers. The at the end of the study (α = .860, n = 17). posttest GCSE scores did predict learn- vast majority of participants reported The MUVE-SE instrument showed a high ing efficiency for the posttest Chair- no difficulty creating their accounts to level of reliability when the researcher Building Task (F[1, 16] = 4.726, p = use the MUVE or with logging in the administered it at the start of the investi- .046) with an R2 of .24. This indicates first time. However, designing their gation (α = .936, n = 17) and then again that a posttest administration of the personal avatars did take some time, the researcher administered it at the end GCSE instrument predicted 24% of the as participants spent up to an hour of the investigation (α = .927, n = 17). variance in posttest learning efficiency, customizing their avatars’ appearances. Volume 27 Number 2 | Journal of Digital Learning in Teacher Education | 71 Copyright © 2010, ISTE (International Society for Technology in Education), 800.336.5191 (U.S. & Canada) or 541.302.3777 (Int’l), iste@iste.org, www.iste.org. All rights reserved. Hearrington Two commented that they enjoyed the she replied, “It was interesting. It was what our goals were in terms of key experience of customizing the appear- very frustrating when some people were aspects for our environment so my ance and dress of their avatars. Six able to talk and when we were unable to building had a purpose. reported instances when the Second Life hear those who were talking.” Apparently Kelly said she was “so tuned in that I client software froze, and eight reported side conversations and comments from barely noticed anything going on around the monitor’s display orientation spon- multiple participants frustrated her. me. I had to make a special effort to not taneously flipping sideways because of Finally, the negative comments get so in the zone that I lose [sic] track. video cards that were incompatible with related to the user experience were con- I loved this [participant underlined the the client software. Finally, although cerning such things as feeling ill when word loved] & very much enjoyed build- it was not a technical problem with using the environment, unspecified ing.” When in the environment itself, the MUVE’s software, several partici- frustration, and technical difficulties. especially when working with their pants (who are also inservice teachers) Two participants reported that using groups, these 13 participants clearly re- reported not being able to access the the MUVE caused them to experience ported a high level of engagement when environment from their school comput- actual motion sickness. One com- in the MUVE. ers due to Internet filters put in place by mented that she experienced so much The remaining four participants their school system. downtime due to technical issues on reported low engagement for various Participants reported a mixture of her home computer that she was very reasons. Two felt intimidated by the in- impressions regarding their enjoyment frustrated. However, she did not explain tricacy of the environment and the com- of the virtual world experience. The what issues caused the downtime. When plexity of the user interface. One said majority were positive (n = 6) or neutral asked a follow-up question about any ill she did not like it because she felt like (n = 7), whereas some were negative (n effects they may have experienced, an she was playing a video game and she = 4). Examples of positive comments additional four participants indicated dislikes video games. Finally, another about enjoying the MUVE included some they had spent so much time looking stated that she did not like Second Life, remarking on their high level of engage- into the monitor when using the MUVE although she gave no specific reason, ment when using the MUVE, enjoyment that they experienced eye strain, blurred and felt that it had no place in her life. of holding class meetings in the MUVE, vision, or headaches. Therefore, there appeared to be some re- and the creativity that the tools in the Regarding the impact of designing sistance to the idea of using this MUVE MUVE enabled. Mya said, “Very engag- and building a learning environment in to complete a class assignment. ing, the researcher would love to do it the MUVE, participants overwhelmingly more.” Kelly’s remark was one of the most reported that the three-dimensional en- Summary and Conclusions positive about the experience of interact- vironment required an enhanced level of ing and creating in the MUVE: planning and afforded them the oppor- Summary tunity to demonstrate that their design Participants improved over the 6-week I loved this [participant under- was representative of best-practices lit- investigation on the Maze Task in terms lined the word loved]. I loved the erature. Perhaps the most representative of reduced PME (mean improvement = fact that I could chat with my pro- participant statement on this topic was: 1.06) and performance (mean improve- fessor and classmates very easily. I ment = 6.23 minutes), and the results enjoyed being creative in building, Having to actually create the space were statistically significant for perfor- purchasing my materials, and then brought the thought process to a mance (t[16] = 4.33, p < 0.01), with a uploading pictures of my choice. deeper level than just merely stating large effect size (d = 1.14). However, Finally, Michelle said, “The interact- what would be in the environment. PME was not significantly lower on the ing and building were fun. I really enjoy posttest for the Maze Task, indicating class online in the virtual environments.” The last item on which the partici- that after 6 weeks participants still per- Some of the comments about lik- pants responded concerned their im- ceived the environment to be challeng- ing the user experience were more pressions of their own engagement when ing to interface with. This is consistent neutral. One participant reported that in the MUVE. Of the 17 participants, with participants’ responses to the open- she “enjoyed it immensely, however it 13 subjectively reported having a high ended questionnaire, indicating a level was an acquired enjoyment.” Angelina level of engagement while in the MUVE. of frustration with performing tasks explained, “It was very difficult at first. Beyonce reported: such as changing one’s camera angle However, after creating more and more (angle of view from an avatar’s perspec- it became easy. I liked being able to chat Highly engaged. My team members tive) and landing one’s avatar where one with my classmates.” Emma reported a were building at the same time, so intends it to land. It is also consistent bit of frustration that detracted from her we communicated on how to build with the literature (Sanchez, 2009). user experience when participating in a and collaborated on what to build In addition to improvements in synchronous class in the MUVE when where. Also, I had to keep in mind performance and PME, participants 72 | Journal of Digital Learning in Teacher Education | Volume 27 Number 2 Copyright © 2010, ISTE (International Society for Technology in Education), 800.336.5191 (U.S. & Canada) or 541.302.3777 (Int’l), iste@iste.org, www.iste.org. All rights reserved. Learning Efficiency and Efficacy in a MUVE improved their learning efficiency The high reliability of both the GCSE (α that participants felt, may result in ill ef- during the investigation. T-tests indicat- > .80) and the MUVE-SE (α > .90) seem fects for some users. It may be advisable to ed that participants exhibited statistically to indicate that both may be useful in- recommend that users be aware of these significantly improved learning efficiency struments for researchers. Although the potential effects and moderate the amount on the Maze Task (t[16] = 3.14, p < 0.01) MUVE-SE is still under development, its of time they spend in the environment and the Chair-Building Task (t[16] = performance shows promise. during an individual session. All of these 4.08, p < 0.001), with large effect sizes of Neither measure of self-efficacy findings are consistent with earlier find- .90 and -1.20, respectively. This finding predicted pretest learning efficiency. The ings in the literature (Sanchez, 2009). indicates that inservice teachers can Posttest GCSE only predicted posttest become efficient at using and creating in learning efficiency on the Chair-Build- Implications the environment over a 6-week time pe- ing Task (F[1, 16] = 4.726, p = .046), The Second Life MUVE appears to be a riod. Participants’ self-reported engage- and it did so with a moderate R2 of .24. promising environment that fosters high ment in the environment likely played a Interestingly, the posttest MUVE-SE levels of engagement in adult learners, role in increasing learning efficiency, but demonstrated predictive ability for the supports synchronous online class ac- because engagement was not empirically posttest Maze Task (F[1, 16] = 19.802, tivities as a distance-education delivery measured, this study cannot shed any p < .001, R2 = .569) and Chair-Building platform, and provides a virtual environ- light on this relationship. Task (F[1, 16] = 6.663, p = .021, R2 = ment in which educators and teacher The learning curve percentages are .308). General computer self-efficacy educators may build simulated learning useful to know. They seem to provide does not appear to be specific enough to environments. It works well with both useful data about how rapidly a user predict learning efficiency in a MUVE. Moodle and SLOODLE. will learn to build and interact in the The MUVE-SE instrument has some Using this MUVE is likely to improve MUVE. Participants experienced the predictive ability, but this is only a pilot the GCSE of teachers and possibly other greatest improvement in terms of the test of the instrument, which needs to adults. However, general computer self- time it took them to complete tasks be refined through item analysis and efficacy does not appear to be directly (48% for the Maze Task, 28% for the further validation measures before it is related to using a MUVE. That is why Chair-Building Task), but also experi- ready for dissemination. the MUVE-SE instrument is promising; enced double-digit reductions in PME Finally, participant impressions of its further development offers the po- (Maze Task = 19%, Chair-Building using the Second Life MUVE for the tential of measuring self-efficacy related Task = 33%). Learning efficiency im- class were generally positive. The major- to the use of a MUVE that may have proved at the lowest rate of all (Maze ity of participants reported being highly the ability to predict learning efficiency Task = 1.6%, Chair-Building Task = engaged when using the environment when using such an environment. 1.9%). Again, this finding is consistent and enjoying interacting in the environ- Users of Second Life and other with participant reports of frustra- ment. Responses also indicated that using MUVEs can build individual objects tion related to using and building the environment enhanced their level using the built-in tools of the MUVE within the MUVE. These reports and of planning and thinking related to the itself, prim by prim. They can also build findings are also consistent with the building of a simulated educational struc- objects outside of the MUVE in a variety literature (Sanchez, 2009). Perhaps ture. Building in the environment took of programs, such as Google Sketch-up, additional guided practice and tutorial some time for participants to get used to, that create three-dimensional objects videos would have helped improve the however. This was supported by the slow and import the objects into the virtual learning curve related to building and rate of improvement in learning efficiency environment. Additionally, it is possible operating within the MUVE. scores. They reported frustration with to build objects within the MUVE and Findings on self-efficacy indicate that learning to build due to the three-dimen- duplicate them so that the user can give using a MUVE will likely contribute to sional nature of the environment and or sell the copy to another user within the improvement of GCSE and MUVE the complexity of manipulating objects the MUVE. Having the skills to build self-efficacy. Participants improved their with the user interface. Some participants or create within the MUVE will enable GCSE from the pretest to the posttest by reported technical difficulties related to educators to create any simulated envi- an average of 6.29 points and improved not having a recommended video card, a ronment for any subject they may wish their MUVE-SE by an average of 56.47 robust enough microprocessor, enough to create for teaching purposes. Having points over the 6-week duration of RAM, and/or a fast enough Internet con- prebuilt objects (created by someone this investigation. The improvements nection. A few even reported feelings of else) would greatly facilitate the creation in GCSE (t[16] = 2.25, p < 0.05) and nausea and motion sickness, eye strain, of virtual simulations by reducing the MUVE-SE (t[16] = 7.16, p < 0.001) were blurred vision, and headaches due to need to build objects prim by prim, statistically significant and had moderate prolonged use of the MUVE. The three- which should reduce the learning curve to large effect sizes as measured by Co- dimensional nature of the environment, and may facilitate greater adoption of hen’s d of -0.52 and -1.84, respectively. coupled with the feeling of immersion this type of technology. Volume 27 Number 2 | Journal of Digital Learning in Teacher Education | 73 Copyright © 2010, ISTE (International Society for Technology in Education), 800.336.5191 (U.S. & Canada) or 541.302.3777 (Int’l), iste@iste.org, www.iste.org. All rights reserved. Hearrington For teacher educators, the main impli- from a large population of K–12 educa- from simple building to complex build- cation seems to be that, with scaffolding tors. Therefore, it is not representative, ing tasks, should increase germane and time, teachers can be taught to create and the results of this investigation cognitive load and facilitate learning simple simulated environments in this may not be generalized. The researcher efficiency (van Merrienboer, Kester, & MUVE. This capability will enable them intends to repeat this experiment with Paas, 2006). It would be useful to know to use this type of tool to meet NETS- larger numbers of participants in the which combinations of methods of T standards with their own students future. In addition, the study did not teaching simulation construction pro- should they have the opportunity to do utilize a comparison group. It is my duce the greatest learning efficiency. so. Results from this study support the intention to accomplish future itera- Finally, empirical measures of conclusion that creation in a MUVE can tions of this study using some form of engagement should be developed and support adult engagement and higher lev- control-group design. Finally, the validated for the purpose of measuring els of thinking. The creation of advanced MUVE self-efficacy instrument that the the engagement of adults and children simulations, requiring the scripting of researcher created will require more data when they are using a MUVE and other object actions, was beyond the scope of to be validated. programs. It would be useful to be able this study. However, this study suggests to compare engagement levels for users that the creation of advanced simulations Suggestions for Further Research of a variety of simulations in a MUVE. will likely require significant training Multi-user virtual environment self- before educators are ready. Second Life efficacy is a construct that appears to Author Note and other MUVEs may offer educational have some usefulness because of its Doug Hearrington is an assistant professor of programming, instructional design, potential predictive abilities. A com- instructional technology at Kennesaw State Uni- versity (KSU), near Atlanta, Georgia. KSU is the and multimedia classes an environment parison of MUVE programs should be third largest institution in the University System of where students can create meaningful conducted to identify common features Georgia. His research focuses on the design and im- and engaging simulations. and user interface mechanisms. Based plementation of virtual and augmented reality to Additionally, Second Life has the po- on these similarities, the MUVE-SE advance learning, metacognitive functioning, self- tential to function as an environment in should be revised and validated with a efficacy, epistemological beliefs, self-determination, as well as technology leadership and integration. which to hold synchronous classes. Par- much larger sample. He has presented at several state, regional, and ticipants have to be taught how to be vir- Learning efficiency appears to be a national conferences and is the project director of tual students with their avatars, because useful framework on which to design Project OWL (Online World of Learning), a multi- the environment presents difficulties comparisons of user interfaces and user virtual environment and curriculum for the related to managing who gets to talk and programs. This methodology should advancement of third through fifth grade STEM learning and teaching. Correspondence regarding ask questions. Additionally, as partici- be validated with a much larger sample this article should be addressed to Doug Hear- pants reported, there is sometimes a lag of participants. To be generalizeable rington, EdD, Assistant Professor of Instructional between when an event begins, such as beyond Second Life, the Maze Task Technology, Department of Instructional Technol- showing a PowerPoint slide, and when and Chair-Building Task should be ogy, Kennesaw State University, 1000 Chastain that event actually takes place in the examined to determine the MUVE Road NW, Box 0127, Kennesaw, Georgia 30144. E-mail: dhearrin@kennesaw.edu MUVE. This is called lag-time, or just programs to which these tasks ap- lag. Lag presents a challenge that Second ply. Learning efficiency could then be References Life has to overcome before it can be calculated for each MUVE, and those Bailey, C. D. (1989). Forgetting and the learning used to hold large numbers of avatars in results could be applied to choose curve: A laboratory study. Management Science, one place representing whole classes of the environment best suited to the 35, 346–352. students. Those wishing to teach using a intended user population. Bandura, A. (1997). Self-efficacy: The exercise of MUVE as a synchronous class space will Due to the complexity of build- control. New York: W.H. Freeman and Company. Campbell, D. T., & Stanley, J. C. (1963). have to learn to overcome these same ing in a MUVE, methods of increas- Experimental and quasi-experimental designs for challenges as well. ing germane cognitive load should be research. Boston: Houghton Mifflin Company. For researchers, this study offers a implemented and studied to deter- Clark, R., Nguyen, F., & Sweller, J. (2006). potential contribution in the form of a mine their ability to increase learn- Efficiency in learning: Evidence-based guidelines method with which to study and com- ing efficiency. Such methods include to manage cognitive load. San Francisco: John Wiley & Sons, Inc. pare MUVE user interfaces and learning structuring building lessons that start Cobb, S., & Fraser, D. S. (2005). Multimedia learning through tasks in a MUVE. Learning with a combination of simple partial in virtual reality. In R. E. Mayer (Ed.), The efficiency and learning curve seem well and whole tasks before progressing to Cambridge handbook of multimedia learning (pp. suited to this task. more complex partial- and whole-task 525–547). New York: Cambridge University Press. activities. Next, start lessons on build- Edwards, L. D. (1995). Microworlds as representations. In A. A. diSessa, C. Hoyles, R. Limitations ing with worked examples, continuing Noss, & L. D. Edwards (Eds.), Computers and The sample used for this study was not to completion tasks, and ending with exploratory learning (pp. 127–154). New York: very large, nor was it randomly drawn whole tasks. Combining this sequence, Springer. 74 | Journal of Digital Learning in Teacher Education | Volume 27 Number 2 Copyright © 2010, ISTE (International Society for Technology in Education), 800.336.5191 (U.S. & Canada) or 541.302.3777 (Int’l), iste@iste.org, www.iste.org. All rights reserved.

ERIC EJ907002: Evaluation of Learning Efficiency and Efficacy in a Multi-User Virtual Environment

Similar ERIC EJ907002: Evaluation of Learning Efficiency and Efficacy in a Multi-User Virtual Environment