ERIC EJ907002: Evaluation of Learning Efficiency and Efficacy in a Multi-User Virtual Environment

2011·0.57 MB·English

by  ERIC

Evaluation of Learning Efficiency and Efficacy in a Multi-User Virtual Environment
Doug Hearrington
Kennesaw State University
Abstract
This study evaluated the multi-user Moodle (Modular, Object-Oriented, (a) Promote, support, and model
virtual environment (MUVE) known Dynamic Learning Environment) is a creative, innovative thinking and
as Second Life, integrated with Moodle free and open source learning manage- inventiveness
and SLOODLE technologies, as an ment system similar to Blackboard or (b) Engage students in exploring real-
exploratory course delivery platform WebCT. The researcher tested Moodle world issues and solving authentic
and for its ability to enable teachers to in this study because it interfaces with problems using digital tools and
meet elements of NETS•T. Graduate Second Life via the plugin known resources
student participants (N = 17) inter- as SLOODLE (Simulation-Linked, (c) Promote student reflection using
acted, constructed simulated schools, Object-Oriented, Dynamic Learning collaboration tools to reveal and
and attended classes in the MUVE. Environment). Using these two tech- clarify students’ conceptual under-
The researcher used pre- and posttest nologies with Second Life allows users standing and thinking, planning,
measures of self-efficacy and learning to blog, take quizzes, submit three-di- and creative processes
efficiency to understand the effects of mensional objects to drop-boxes, and (d) Model collaborative knowledge con-
the MUVE on participants and their much more. The ability to integrate struction by engaging in learning
rate of learning to make educational the Second Life MUVE with a learn- with students, colleagues, and others
use of the environment. Findings imply ing management system (Moodle) in face-to-face and virtual environ-
that the technologies have potential as via SLOODLE provided an additional ments (ISTE, 2008)
a distance-learning platform and as a reason to test the functionality of these
tool to meet elements of NETS•T. Pre- technologies to serve as a distance The second NETS•T standard is: “De-
paring teachers to use the MUVE for education platform. sign and develop digital age learning ex-
these purposes is likely to require a sig- Another compelling reason to use a periences and assessments.” Two of this
nificant amount of scaffolding. (Key- MUVE in teacher education programs standard’s performance indicators clearly
words: MUVE, Second Life, distance is that the technology may facilitate lend themselves to using a MUVE:
education, self-efficacy) the implementation of specific pro-
gram standards. ISTE has, in conjunc- (a) Design or adapt relevant learning
I
n recent years, an increasing number tion with a wide variety of professional experiences that incorporate digital
of colleges, universities, and edu- education organizations, established tools and resources to promote
cational institutions have started to the National Educational Technology student learning and creativity
use multi-user virtual environments Standards for Teachers (NETS•T). (b) Develop technology-enriched
(MUVEs) to host online classes and Teacher-education institutions ac- learning environments that enable
provide online content and provide credited by the National Council for all students to pursue their indi-
virtual educational interactions. Second Accreditation of Teacher Education vidual curiosities and become active
Life, a popular free and commercially (NCATE) use the NETS•T as the participants in setting their own
available MUVE, has attracted the standards by which to measure teacher educational goals, managing their
attention of educational institutions readiness to use information and com- own learning, and assessing their
worldwide and is being used by at munication technologies (ICTs). A own progress (ISTE, 2008)
least 142 colleges and universities, 41 MUVE may help a teacher implement
for profit and nonprofit educational the first two NETS•T standards and A MUVE is a form of virtual reality
organizations, eight libraries, and four several of the performance indicators (VR), a computer-based technology that
museums (SimTeach, 2008). Second for those standards. The first NETS•T provides the visual, aural, and tactile
Life is also finding increasing favor standard is: “Facilitate and inspire stimuli of a virtual world generated in
among secondary schools, and schools student learning and creativity.” This real time (Sanchez, Lumbreras, & Silva,
across the globe are using it for educa- standard contains four performance 2001). A MUVE is an example of inter-
tional purposes. This study utilized the indicators, each of which may be ac- active multimedia in which experiences
MUVE provided by Second Life. complished using a MUVE: are made possible by dynamic elements
Volume 27 Number 2 | Journal of Digital Learning in Teacher Education | 65
Copyright © 2010, ISTE (International Society for Technology in Education), 800.336.5191
(U.S. & Canada) or 541.302.3777 (Int’l), [e-mail], www.iste.org. All rights reserved.
Hearrington
under the user’s control (Rieber, 2005). Barriers to learning in the Second relationship with her or his ability to
For example, rather than reading about Live MUVE include a “high learning learn to build a simulation in a MUVE.
the surface of the Moon, one can repli- curve,” which is reflected in its compli- This study advances the field in this area
cate the environment in a MUVE and cated user interface, high difficulty level by providing a first step in this direc-
students can control their own explora- of building new objects for novice users, tion. In addition, this study represents
tion of a virtual Moon, thus providing a perceptions that working in the Second an attempt to pilot-test an instrument to
form of experiential learning. Addition- Life MUVE was too time-consuming, measure self-efficacy for using a MUVE.
ally, users may interact with objects and and technical difficulties (Sanchez, 2009).
events in simulations in MUVEs and Some users have expressed disappoint- Self-Efficacy
similar three-dimensional environments ment with the environment because of Perceived self-efficacy is one’s belief in
(Cobb & Fraser, 2005). their expectations that it should be more one’s ability to complete actions required
Users of a MUVE experience three like a playing a game. When Second Life to produce a result or to accomplish a
“presence layers” in a 360-degree, three- turned out to not be a game, these users given task (Bandura, 1997). Self-efficacy
dimensional environment, with the became bored and frustrated (Sanchez, influences the careers people pursue,
effect of combining physical and virtual 2009). Due to the limitations of moni- the level of effort they invest in a given
realism in the virtual space to produce tors, three-dimensional MUVE envi- endeavor, their resilience to adversity,
an immersive experience that “conveys ronments are viewed on a flat screen, and the level of accomplishments they
a feeling of being there and a strong creating a lack of peripheral vision. This achieve (Bandura, 1997). Therefore, it is
sense of co-presence when other avatars lack of peripheral vision detracts from possible that self-efficacy will be related
are present” (Warburton, 2009, p. 6). the representation of the environment to learning efficiency or the perceived
The three layers of presence are physi- (Cobb & Fraser, 2005). Time delays difficulty of using the environment. For
cal, communication, and status (War- are often common when using such an this purpose, the researcher adminis-
burton, 2009). The visual and physical environment, and depth perception is tered a measure of general computer
proximity of avatars to one another difficult for some users to learn when in self-efficacy and created and adminis-
creates the physical presence. Spatially a three-dimensional environment (Cobb tered a specific self-efficacy instrument
enhanced voice that allows one to sense & Fraser, 2005). related to using a MUVE.
the direction from which another voice Positive aspects of using the Second
originates, as well as communication via Life MUVE have also been noted. Users Cognitive Load Theory
synchronous text chat and asynchronous enjoy creating and designing their avatar Because users interact with a MUVE’s
text communications, such as e-mail and the feelings of creativity and accom- three-dimensional world and other users
and group notices, create the commu- plishment they experience when build- by means of avatars, these environments
nication presence. Finally, various tools ing in the environment (Sanchez, 2009). may present a higher level of cogni-
that allow one to know when a friendly Additionally, users have reported having tive load than the more typical two-
avatar is “in-world” or offline create a strong attachment to their avatar and dimensional, non-avatar, computer-user
a status presence (Warburton, 2009). enjoyed communicating with others via interfaces with which most are familiar.
Presence helps to establish a first-person their avatar. Some have noted that the Cognitive load refers to the level of diffi-
experience when one is in a MUVE. This sense of enjoyment and creativity they culty a learner perceives when perform-
experience has been said to help develop experience in this MUVE outweighs the ing or learning a given task. Cognitive
direct, subjective, and personal knowl- sense of frustration they feel from the load theory is based on information
edge (Sanchez, Lumbrera, & Silva, 2001). complexity of the user interface and the processing theory, which states that an
technical issues associated with using individual must process information
Affordances and Barriers to MUVE Use Second Life (Sanchez, 2009). using short-term memory in order to
MUVES may be said to have structural The immersive nature of a MUVE place it into long-term memory for later
and functional affordances. Structural combines these physical, social, and cul- use. If the cognitive load is too great,
affordances consist of (a) a collection of tural dimensions to provide a space in the information will not be processed
objects that model the mathematical/ which compelling simulations and role- into long-term memory efficiently, and
physical properties of the domain, (b) playing activities may take place (War- learning will be inhibited. Cognitive
links to numerous representations of the burton, 2009). However, research on load theory is a set of principles used
underlying model, (c) opportunities to the use of modern MUVEs as distance by researchers to study the three types
use the objects in complex ways, and (d) education platforms is in its infancy, of load: (a) intrinsic load, which is the
challenges or activities for the student to and little is known about the “learning mental work imposed by the complexity
solve or explore (Edwards, 1995). Func- curve” educators may experience when of a task and is largely determined by
tional affordances include the interaction trying to create simple simulations in one’s goals; (b) germane (relevant) load,
between the user, the software, and the such an environment or whether an which is the mental work imposed by
setting in which the environment is used. educator’s computer self-efficacy has any an instructional activity that benefits the
66 | Journal of Digital Learning in Teacher Education | Volume 27 Number 2
Copyright © 2010, ISTE (International Society for Technology in Education), 800.336.5191
(U.S. & Canada) or 541.302.3777 (Int’l), [e-mail], www.iste.org. All rights reserved.
Learning Efficiency and Efficacy in a MUVE
Learning efficiency is a function of
Performance
achievement and mental effort, but it
cannot tell us, by itself, how much more
E = 0
quickly one can perform a task after
1.0
initial familiarity and over time. This
may be accomplished by calculating
0.8
the learning curve. Learning curve is
the measurement of time to complete
High Efficiency
0.6
a specific task correctly over time, after
practice. An illustration of the learning
0.4
curve, taken from The Learning Curve
Deskbook (Teplitz, 1991), assumes a
0.2 piano student was learning to play the
Mental Effort
“Minute Waltz” by Chopin. The first time
she played the piece, it took 3 minutes
-1.0 -0.8 -0.6 -0.4 -0.2 0.2 0.4 0.6 0.8 1.0 to play. Her second attempt took 2.6
-0.2 minutes. Attempt number three took
2.37 minutes. The fourth attempt took
-0.4 under 2 minutes. The rate of the student’s
improvement, calculated every other
attempt (attempt 1, 2, 4, 8, etc.), is 21%.
-0.6
Low Efficiency This means that each doubling results
-0.8 in an improvement of 79%. Although
time it takes the student to succeed gets
smaller and smaller, rates of improve-
-1.0
ment tend to remain the same. This has
been shown to remain constant in a
variety of learning, manufacturing, and
business situations (Teplitz, 1991). The
Figure 1. The Learning efficiency grid.
learning curve has been used in manu-
learning of the task; and (c) extraneous for PME that are above the mean are facturing to help calculate production
(irrelevant) load, which is mental work plotted to the right of the origin of the time and cost, to forecast labor require-
that is irrelevant to the learning goal and grid. Z-scores for achievement (comple- ments, and as a metric by which manag-
wastes limited mental resources (Clark, tion time) that are above the mean are ers monitor production (Yelle, 1979).
Nguyen, & Sweller, 2006). Cognitive reversed, because negative scores repre- The same concept may also be applied
load is operationalized in this study as sent higher achievement. The hypotheti- to calculate the “forgetting curve” of a
perceived mental effort (Clark, Nguyen, cal line of zero efficiency runs diagonally task (Bailey, 1989). The learning curve
& Sweller, 2006). The level of perceived from a point at the lower left of the grid has been used to study the improvement
mental effort can enhance or inhibit (quadrant III), through the origin, to the in learning of computer-aided design
one’s learning efficiency. upper right of the grid (quadrant I) along (CAD) students over time (Hamade,
Any task that can be accomplished, a line that would extend through points Artail, & Jabar, 2005) and the improve-
such as learning to perform a mathemati- (-1, -1) and (1, 1). See Figure 1 for an ment in disease pattern recognition and
cal calculation or learning to build a chair example of the learning efficiency grid. diagnosis by medical students over time
in a virtual world, can be measured in Low learning efficiency occurs when (Williams, Klamen, & Hoffman, 2008).
terms of learning efficiency. Learning one learns something slowly with great The learning curve is useful for describ-
efficiency is the relationship between a mental effort. High learning efficiency ing and studying tasks that require both
measurement of achievement, such as a occurs when one learns something procedural and declarative knowledge
test score or the amount of time it takes quickly and with low mental effort (Hamade, Artail, & Jabar, 2005). There-
a learner to correctly perform a task, and (Clark, Nguyen, & Sweller, 2006). Thus, fore, it makes sense to apply the learning
the perceived mental effort (PME) of learning efficiency can potentially be curve to the learning of the mostly pro-
the learner. To measure learning effi- used to calculate the learning “curve” cedural knowledge required to perform
ciency, achievement and PME scores are of a given task over time, to compare tasks in a virtual world. Learning curve
converted to z-scores. Both are plotted on different programs to one another, or to may be calculated using various slope
a Cartesian grid with PME on the X-axis compare learning in a variety of com- formulae (Yelle, 1979; Teplitz, 1991) or as
and achievement on the Y-axis. Z-scores puter or non-computer conditions. a percentage of improvement.
Volume 27 Number 2 | Journal of Digital Learning in Teacher Education | 67
Copyright © 2010, ISTE (International Society for Technology in Education), 800.336.5191
(U.S. & Canada) or 541.302.3777 (Int’l), [e-mail], www.iste.org. All rights reserved.
Hearrington
Based on the increased interest Table 1: Participant Demographics
in MUVEs in higher education as a Standard
Mean Deviation Range
distance-learning platform, the potential
of the environment to be used to meet Age in Years 32.6 6.3 25 - 47
NETS-T standards, and the increasing Years of Teaching Experience 9.0 5.7 1 - 21
use of MUVEs at all levels of education
for building and using simulations, the The Computer User Self-Efficacy 10-point scale, with a 1 representing low
researcher designed this study to answer Scale (Cassidy & Eachus, 2002) mea- confidence and a 10 representing high
these questions: sures general computer self-efficacy confidence (Bandura, 1997).
(GCSE). This scale consists of two parts: The researcher devised the “Maze
1. How efficient are new participants at
(a) individual characteristics and (b) Task” as a means of measuring the basic
creating and working in the MUVE,
computer self-efficacy items. The indi- skills commonly required to utilize a
and how does their efficiency change
vidual characteristics section contains MUVE. The researcher designed a maze
over time?
seven items that ask the participant that required participants to navigate
2. What is the relationship between
about whether they have attended a their avatars through a door, turn and
participant general computer self-ef-
computer course or own a computer, walk in various directions, and fly over
ficacy and MUVE self-efficacy before
basic demographics (age and gender), and land on the other side of a wall. Ad-
and after using the environment for
experience with computers, and types of ditionally, a sign in the maze required
a period of time? Does using the
software packages they have used. The them to take a picture of their avatar’s
MUVE result in increased GCSE and
computer self-efficacy section contains face and e-mail it to me to measure
MUVE-SE? Does GCSE or MUVE-
30 items, each with a 5-point Likert mastery of changing the camera (user’s)
SE predict learning efficiency?
scale ranging from strongly disagree to point of view. Next, a sign in the maze
3. What were the participants’ impres-
strongly agree. The researcher worded directed participants to answer a question
sions of their user experience within
approximately half of the items posi- by writing their answer on a note card
the Second Life MUVE?
tively and half negatively. The researcher of their own creation. They then had to
reverse-coded the negatively worded deposit the note card into a drop box and
METHOD
items for analysis purposes. Example obtain a different note card from a dis-
Participants items include: “Computers are far too penser. Finally, participants had to click
The researcher collected data from 17 complicated for me,” and “Most difficul- on a teleporter to transport to another
graduate students enrolled in a master’s ties I encounter when using computers, location within the virtual world. Users
degree program in educational leader- I can usually deal with.” The internal commonly perform each of these tasks
ship in a southeastern state university. consistency of the 30-item scale has when using the Second Life MUVE for
The students were enrolled in the same been reportedly very high (α = 0.97, N = educational purposes.
section of a class and therefore repre- 184). Test-retest reliability has been re- Finally, the researcher devised the
sent a convenience sample. Two of the portedly high and statistically significant “Chair-Building Task” as a means of
participants were male. Eight identi- (r = .086, N = 74, p < 0.0005) (Cassidy & measuring participants’ basic building
fied themselves as African-American, Eachus, 2002). skills. The researcher gave participants
and the remaining seven identified The researcher created the MUVE a model of a chair and asked them
themselves as Caucasian. None of the Self-Efficacy (MUVE-SE) instrument to to replicate the design. Instructions
participants had any experience using a measure the multi-user virtual environ- indicated that the chair did not have to
MUVE or a similar environment before ment self-efficacy. This study represents have identical measurements or dimen-
this study. Table 1 summarizes other an initial pilot test in the development sions, just an identical design. The chair
demographic characteristics of the of the MUVE-SE. The MUVE-SE was to include four legs, a square seat,
participants. contains 18 items designed to measure and a back. The back was to consist of
a participant’s efficacy at performing two vertical braces connected by two
Instruments tasks typically required when they use horizontal slats.
Cognitive load was measured for two the Second Life MUVE as a learning Participants also synchronously at-
defined tasks within the MUVE using environment. Example items include: “I tended two classes entirely in the virtual
an established 9-point scale ranging believe I can teleport to other locations,” environment and met with their team-
from 1 for very little mental effort to and “I believe I can move objects I create mates outside of regularly scheduled
9 for a great deal of mental effort to in Second Life.” Initial responses to the class time to discuss their assignments
achieve a task (Clark, Nguyen, & Swell- 18 items were on a “Yes” or “No” binary and work on their building project.
er, 2006). These two tasks—the Maze scale. Participants responding “Yes” to During class sessions, the instructor
Task and the Chair-Building Task—are any item were then asked to rate their presented information verbally and
described below. level of confidence in their ability on a using a PowerPoint slideshow. The
68 | Journal of Digital Learning in Teacher Education | Volume 27 Number 2
Copyright © 2010, ISTE (International Society for Technology in Education), 800.336.5191
(U.S. & Canada) or 541.302.3777 (Int’l), [e-mail], www.iste.org. All rights reserved.
Learning Efficiency and Efficacy in a MUVE
Table 2: Completion Times (Performance) and Perceived Mental Effort (PME) scores
During the 6 weeks in which par-
Maze Task Chair-Building Task ticipants used the MUVE, they worked
Measurement M SD M SD in teams of three or four to build a
Pretest simulated digital age classroom that
Performance 12.88 6.92 16.55 5.66 would promote high levels of technology
PME 5.41 2.06 7.41 1.94 integration based on assigned readings
Posttest and each group’s literature review of
Performance 6.65 2.77 11.79 4.79 technology integration and 21st century
PME 4.35 1.91 4.94 2.01
or digital age classroom design topics.
Note. n = 17 for the pretest and posttest.
The instructional strategy used was a
Table 3: Pre- and Posttest Learning Efficiency Scores combination of problem-based learn-
Maze Task Chair-Building Task ing and cooperative learning groups, in
which each team had the same digital
M SD M SD
age classroom building assignment. In
Pretest -0.5419 1.4222 -0.6702 1.0468
addition to building a simulated learning
Posttest 0.5415 0.9379 0.6690 1.1728
environment (SLE), teams were required
Note. n = 17 for the pretest and posttest.
to itemize the elements they believed
Table 4: Mean Learning Curve Improvement and Weekly Improvement Percentages were important to the design of such
Maze Task Chair-Building Task a learning environment and to explain
how they addressed each element in their
Total Improvement Weekly Improvement Total Improvement Weekly Improvement
design as part of a presentation to their
Achievement 48 % 8 % 28 % 4.6 %
classmates. Participants uploaded Pow-
PME 19 % 3 % 33 % 5.5 %
erPoint presentations into Second Life
Learning Efficiency 1.6 % .2 % 1.9 % .3 %
and delivered their presentation inside
of the MUVE. Each member of the team
instructor also showed students a video GCSE, MUVE-SE, the Maze Task, and designed and helped construct the SLE,
during class in the MUVE. Participants the Chair-Building Task. The researcher contributing approximately 100 objects to
conducted small-group discussions in recorded the start and end times of the its construction. These objects are known
the environment and submitted a simple tasks as well as the participants’ PME in Second Life as “prims” which is short
assignment on a note card into a drop associated with both of these tasks. for “primitive” objects. A “prim” starts out
box. To evaluate SLOODLE, participants The researcher measured learning as a basic three-dimensional shape, such
used the quiz tool and the choice tool efficiency using the following formula as a cube, and is then transformed by
(which allows users to vote and see the (Clark, Nguyen, & Sweller, 2006): participants into a variety of shapes that,
three-dimensional results of their vote). when combined together, form complete
Average Performance in Z-Scores –
Finally, participants uploaded a Pow- objects. For example, a bookshelf with
Average Difficulty Rating in Z-Scores
erPoint presentation into the MUVE three shelves, two sides, and a base may
and loaded it into a perpetual slideshow consist of six “prims.” Each SLE contained
√2
viewer outside of their building. They 300–500 prims.
did this to explain their design and their To conduct the learning efficiency At the end of the study, participants
rationale for their decisions to their calculation, the researcher measured again completed the GCSE, MUVE-SE,
classmates and their instructor. performance on the maze and chair Chair-Building Task, and Maze Task.
tasks based on the amount of time it Additionally, the researcher adminis-
Design and Procedure took participants to complete each task. tered a questionnaire designed to elicit
This pilot study used a pre-experimental, The researcher measured the difficulty their impressions of the MUVE as a
one-group pretest-posttest design (Camp- of each task using the PME scale (Clark, learning environment and as a place in
bell & Stanley, 1963). Participants were Nguyen, & Sweller, 2006). The research- which to build virtual simulations.
introduced to the MUVE in class and er calculated learning efficiency for each Finally, the researcher calculated a
received a guided 3-hour practice session, of two performance tasks: (a) naviga- learning curve for the percentage of im-
during which they were introduced to all tion through a maze in the MUVE provement in achievement (completion
of the basic skills measured in the maze and (b) building a simple chair within time), PME, and learning efficiency over
and chair tasks. During this time, partici- the MUVE. The researcher calculated the 6-week period of this investigation.
pants built a chair identical to the one in learning efficiency after participants had
the Chair-Building Task for practice. spent 3 hours in the MUVE (pretest) Analysis
After this introductory session, and then after spending 6 weeks using The researcher analyzed the quantitative
the researcher took measurements for the MUVE (posttest). data using SPSS and Microsoft Excel.
Volume 27 Number 2 | Journal of Digital Learning in Teacher Education | 69
Copyright © 2010, ISTE (International Society for Technology in Education), 800.336.5191
(U.S. & Canada) or 541.302.3777 (Int’l), [e-mail], www.iste.org. All rights reserved.
Hearrington
The researcher transcribed the open-
ended questionnaire responses, coded
them, entered them into a qualitative Maze Task Posttest Performance
research program (HyperResearch), (0.51, -0.26)
and analyzed them using the constant E = 0.5415
0.6
comparative method to shed light on the
quantitative findings.
The researcher used a combination of 0.4
procedures to answer research question Chair Task Posttest
1, “How efficient are new participants at (0.42, -0.53) 0.2
creating and working in the MUVE, and E = -0.6690 Mental Effort
how does efficiency change over time?”
First, the researcher calculated pretest -0.6 -0.4 -0.2 0.2 0.4 0.6
Chair Task Pretest
and posttest learning efficiency for both
0.2 (-0.42, 0.53)
the Chair-Building and Maze Tasks. The
E = -0.6702
researcher conducted paired sample
t-tests to determine whether pre- and 0.4
posttest scores on both tasks (comple-
tion time, PME, and learning efficiency) 0.6 Maze Task Pretest
were statistically different from one (-0.51, 0.26)
another. The researcher calculated effect E = -0.5415
sizes using Cohen’s d. Additionally, the
researcher calculated the learning curve
Figure 2. Learning efficiency plot.
based on the percentage of improvement
participants achieved over the 6-week
instructions was easy, and in about ments for the participants after 3 hours
period of the study.
2.5 hours, the three programs were of experience (pretest) in the MUVE and
Research question 2 contained three
“talking” together. Once set up was again after 6 weeks of experience (post-
parts. The researcher performed Pearson
complete, participants were able to test). Table 2 (p. 69) provides means and
correlation analyses and paired sample
use SLOODLE’s quiz tool, voting tool, standard deviations.
t-tests to answer part 1 of the question,
and toolbar when attending a virtual The results of two-tailed, paired-
“What is the relationship between par-
class in Second Life to facilitate asking sample t-tests comparing the Maze Task
ticipant general computer self-efficacy
and answering questions by raising pretest performance and PME scores
and MUVE self-efficacy before and after
their avatars’ virtual hands to get the with the Maze Task posttest perfor-
using the environment?” The researcher
instructor’s attention. The instructor mance and PME scores showed that
used paired sample t-tests were used to
gave one quiz and took one vote dur- participants’ performance times were
answer part 2 of the question, “Does us-
ing the project, and the technologies significantly better on the posttest (t[16]
ing the MUVE result in increased GCSE
worked together perfectly to transmit = 4.33, p < 0.01) and had a large effect
and MUVE-SE?” Finally, the researcher
each participant’s score and vote from size (d = 1.14). However, participants’
used linear regression analyses to answer
Second Life, through SLOODLE, and PME was not significantly lower on the
the third part of question 2, “Does GCSE
into Moodle for final recording. The posttest (t[16] = 1.62, p = 0.12).
or MUVE-SE predict learning effi-
combined functionalities of SLOODLE Two-tailed, paired-sample t-tests com-
ciency?” The researcher calculated effect
and Moodle seem to offer educators paring the Chair Task pretest performance
sizes using Cohen’s d.
a viable, although not yet fully devel- and PME scores with Chair Task posttest
The researcher conducted an analy-
oped, means of using Second Life as a performance and PME scores indicated
sis of open-ended survey questions for
distance education platform in combi- that participants’ performance times were
trends and commonalities to answer
nation with the learning management significantly better on the posttest (t[16]
research question 3, “What were the
system capabilities of Moodle. = 3.01, p < 0.01) with a large effect size (d
participants’ impressions of their user ex-
= .88). Results also indicated that partici-
perience within the Second Life MUVE?”
Learning Efficiency pants’ PME was significantly lower on the
The researcher measured learning effi- posttest (t[16] = 4.02, p < 0.01) with a large
Results
ciency for two tasks—the Maze Task and effect size (d = 1.21).
Moodle and SLOODLE the Chair-Building Task—using comple- Table 3 (p. 69) presents means and
The integration of Moodle, SLOODLE, tion time as a measure of performance standard deviations for learning ef-
and Second Life worked quite well. and PME as a measure of cognitive load. ficiency. A two-tailed, paired-sample
Following the printed and video set-up The researcher took these measure- t-test indicated that participants were
70 | Journal of Digital Learning in Teacher Education | Volume 27 Number 2
Copyright © 2010, ISTE (International Society for Technology in Education), 800.336.5191
(U.S. & Canada) or 541.302.3777 (Int’l), [e-mail], www.iste.org. All rights reserved.
Learning Efficiency and Efficacy in a MUVE
Table 5: GCSE and MUVE-SE Means and Standard Deviations a moderate amount. The same analyses
GCSE MUVE-SE for the effects of posttest MUVE-SE on
M SD M SD posttest Maze Task learning efficiency
Pretest 156.18 12.52 101.29 38.95 were statistically significant (F[1, 16]
= 19.802, p < .001) with a strong R2 of
Posttest 162.47 11.52 157.76 19.03
.569. The posttest MUVE-SE predicted
Note: n = 17
more than 56% of the variance in post-
test Maze Task learning efficiency. The
significantly more efficient in the Maze Table 5 provides means and stan- linear regression analysis for posttest
Task (t[16] = 3.14, p < 0.01) after 6 weeks dard deviations for the GCSE and MUVE-SE and Chair-Building Task
in the MUVE, with a large effect size MUVE-SE measures. learning efficiency was also statistically
(d = -.90). Similarly, participants were The researcher performed Pearson significant (F[1, 16] = 6.663, p = .021)
significantly more efficient in the Chair- correlations to determine the relation- and had an R2 of .308. This indicates that
Building Task (t[16] = 4.08, p < 0.001) ship between GCSE and MUVE-SE the posttest MUVE explained 30% of
after 6 weeks, with a large effect size (d pre- and posttest scores. The researcher the variance in posttest Chair-Building
= -1.20). Contrary to typical effect size calculated two-tailed tests for signifi- Task learning efficiency. The MUVE-SE,
interpretation, due to the use of standard- cance within SPSS for each correlation. when administered after participants
ized scores and the way learning effi- Results indicate that pretest GCSE scores spent 6 weeks in the multi-user virtual
ciency is calculated, the negative signs on are not significantly correlated with pre- environment, appeared to explain a
these two effect sizes indicate that a large test MUVE-SE scores (r = .097, p = .712, moderate amount of the variance in
positive improvement was realized. n = 17). Additionally, posttest GCSE participants’ learning efficiency on both
Mean learning efficiency pretest and scores are not significantly correlated the Maze and Chair-Building Tasks.
posttest scores are plotted in Figure 2 for with posttest MUVE-SE scores (r = .301, Paired-sample t-tests show that
the Maze Task and Chair-Building Task. p = .240, r = 17). participants’ improvement in GCSE was
Both learning efficiency scores indicated Linear regression analyses indicate that significant (t[16] = 2.25, p < 0.05) and
low efficiency on the pretest and higher pretest GCSE scores do not predict pretest had a moderate effect size (p = -0.52).
achievement based on the posttest scores. learning efficiency on either the Maze Participants’ improvement in self-effica-
Table 4 (p. 69) presents learning Task (F[1, 16] = 0.159, p = .696) or the cy for using a multi-user virtual envi-
curve values in percentages. These val- Chair-Building Task (F(1, 16) = 1.749, p = ronment was also statistically significant
ues show the percentage of improvement .206). Similarly, pretest MUVE-SE scores (t[16] = 7.16, p < 0.001) and had a large
for the Maze and Chair-Building tasks, do not predict pretest learning efficiency effect size (d = -1.84). It appears that
as measured by achievement (comple- on the Maze Task (F[1, 16] = 0.811, p = using the Second Life MUVE for a pe-
tion time), PME, and learning efficiency .382) or the Chair-Building Task (F[1, 16] riod of 6 weeks does result in increased
over the 6-week duration of the investi- = 0.020, p = .890). Neither the GCSE nor GCSE as well as increased self-efficacy
gation and as weekly averages. the MUVE-SE demonstrated the ability to related to using the MUVE.
predict learning efficiency when adminis-
Self-Efficacy tered as a pretest. Participant Impressions
The researcher tested the 30-item instru- The researcher performed another The researcher asked the 17 partici-
ment that the researcher used to measure set of linear regression analyses to pants to fill out a survey containing 14
General Computer Self-Efficacy (GCSE) determine whether the GCSE instru- open-ended items designed to elicit
and the 18-item Multi-User Virtual ment or the MUVE-SE instrument the details of their experiences in the
Environment Self-Efficacy (MUVE-SE) could predict learning efficiency when MUVE. Any names mentioned here are
instrument for reliability using Chron- administered after participants had pseudonyms. The first item asked about
bach’s α. The GCSE instrument showed spent 6 weeks (posttest) using the the presence of any technical issues that
a high degree of reliability when the MUVE. Results indicated that posttest may have hindered or prevented use of
researcher administered it at the begin- GCSE scores did not predict posttest the environment. No one reported dif-
ning of the investigation (α = .840, n = learning efficiency on the Maze Task ficulty downloading and installing the
17) and again when it was administered (F[1, 16] = 3.235, p = .092). However, software on their home computers. The
at the end of the study (α = .860, n = 17). posttest GCSE scores did predict learn- vast majority of participants reported
The MUVE-SE instrument showed a high ing efficiency for the posttest Chair- no difficulty creating their accounts to
level of reliability when the researcher Building Task (F[1, 16] = 4.726, p = use the MUVE or with logging in the
administered it at the start of the investi- .046) with an R2 of .24. This indicates first time. However, designing their
gation (α = .936, n = 17) and then again that a posttest administration of the personal avatars did take some time,
the researcher administered it at the end GCSE instrument predicted 24% of the as participants spent up to an hour
of the investigation (α = .927, n = 17). variance in posttest learning efficiency, customizing their avatars’ appearances.
Volume 27 Number 2 | Journal of Digital Learning in Teacher Education | 71
Copyright © 2010, ISTE (International Society for Technology in Education), 800.336.5191
(U.S. & Canada) or 541.302.3777 (Int’l), [e-mail], www.iste.org. All rights reserved.
Hearrington
Two commented that they enjoyed the she replied, “It was interesting. It was what our goals were in terms of key
experience of customizing the appear- very frustrating when some people were aspects for our environment so my
ance and dress of their avatars. Six able to talk and when we were unable to building had a purpose.
reported instances when the Second Life hear those who were talking.” Apparently
Kelly said she was “so tuned in that I
client software froze, and eight reported side conversations and comments from
barely noticed anything going on around
the monitor’s display orientation spon- multiple participants frustrated her.
me. I had to make a special effort to not
taneously flipping sideways because of Finally, the negative comments
get so in the zone that I lose [sic] track.
video cards that were incompatible with related to the user experience were con-
I loved this [participant underlined the
the client software. Finally, although cerning such things as feeling ill when
word loved] & very much enjoyed build-
it was not a technical problem with using the environment, unspecified
ing.” When in the environment itself,
the MUVE’s software, several partici- frustration, and technical difficulties.
especially when working with their
pants (who are also inservice teachers) Two participants reported that using
groups, these 13 participants clearly re-
reported not being able to access the the MUVE caused them to experience
ported a high level of engagement when
environment from their school comput- actual motion sickness. One com-
in the MUVE.
ers due to Internet filters put in place by mented that she experienced so much
The remaining four participants
their school system. downtime due to technical issues on
reported low engagement for various
Participants reported a mixture of her home computer that she was very
reasons. Two felt intimidated by the in-
impressions regarding their enjoyment frustrated. However, she did not explain
tricacy of the environment and the com-
of the virtual world experience. The what issues caused the downtime. When
plexity of the user interface. One said
majority were positive (n = 6) or neutral asked a follow-up question about any ill
she did not like it because she felt like
(n = 7), whereas some were negative (n effects they may have experienced, an
she was playing a video game and she
= 4). Examples of positive comments additional four participants indicated
dislikes video games. Finally, another
about enjoying the MUVE included some they had spent so much time looking
stated that she did not like Second Life,
remarking on their high level of engage- into the monitor when using the MUVE
although she gave no specific reason,
ment when using the MUVE, enjoyment that they experienced eye strain, blurred
and felt that it had no place in her life.
of holding class meetings in the MUVE, vision, or headaches.
Therefore, there appeared to be some re-
and the creativity that the tools in the Regarding the impact of designing
sistance to the idea of using this MUVE
MUVE enabled. Mya said, “Very engag- and building a learning environment in
to complete a class assignment.
ing, the researcher would love to do it the MUVE, participants overwhelmingly
more.” Kelly’s remark was one of the most reported that the three-dimensional en-
Summary and Conclusions
positive about the experience of interact- vironment required an enhanced level of
ing and creating in the MUVE: planning and afforded them the oppor- Summary
tunity to demonstrate that their design Participants improved over the 6-week
I loved this [participant under-
was representative of best-practices lit- investigation on the Maze Task in terms
lined the word loved]. I loved the
erature. Perhaps the most representative of reduced PME (mean improvement =
fact that I could chat with my pro-
participant statement on this topic was: 1.06) and performance (mean improve-
fessor and classmates very easily. I
ment = 6.23 minutes), and the results
enjoyed being creative in building,
Having to actually create the space were statistically significant for perfor-
purchasing my materials, and then
brought the thought process to a mance (t[16] = 4.33, p < 0.01), with a
uploading pictures of my choice.
deeper level than just merely stating large effect size (d = 1.14). However,
Finally, Michelle said, “The interact- what would be in the environment. PME was not significantly lower on the
ing and building were fun. I really enjoy posttest for the Maze Task, indicating
class online in the virtual environments.” The last item on which the partici- that after 6 weeks participants still per-
Some of the comments about lik- pants responded concerned their im- ceived the environment to be challeng-
ing the user experience were more pressions of their own engagement when ing to interface with. This is consistent
neutral. One participant reported that in the MUVE. Of the 17 participants, with participants’ responses to the open-
she “enjoyed it immensely, however it 13 subjectively reported having a high ended questionnaire, indicating a level
was an acquired enjoyment.” Angelina level of engagement while in the MUVE. of frustration with performing tasks
explained, “It was very difficult at first. Beyonce reported: such as changing one’s camera angle
However, after creating more and more (angle of view from an avatar’s perspec-
it became easy. I liked being able to chat Highly engaged. My team members tive) and landing one’s avatar where one
with my classmates.” Emma reported a were building at the same time, so intends it to land. It is also consistent
bit of frustration that detracted from her we communicated on how to build with the literature (Sanchez, 2009).
user experience when participating in a and collaborated on what to build In addition to improvements in
synchronous class in the MUVE when where. Also, I had to keep in mind performance and PME, participants
72 | Journal of Digital Learning in Teacher Education | Volume 27 Number 2
Copyright © 2010, ISTE (International Society for Technology in Education), 800.336.5191
(U.S. & Canada) or 541.302.3777 (Int’l), [e-mail], www.iste.org. All rights reserved.
Learning Efficiency and Efficacy in a MUVE
improved their learning efficiency The high reliability of both the GCSE (α that participants felt, may result in ill ef-
during the investigation. T-tests indicat- > .80) and the MUVE-SE (α > .90) seem fects for some users. It may be advisable to
ed that participants exhibited statistically to indicate that both may be useful in- recommend that users be aware of these
significantly improved learning efficiency struments for researchers. Although the potential effects and moderate the amount
on the Maze Task (t[16] = 3.14, p < 0.01) MUVE-SE is still under development, its of time they spend in the environment
and the Chair-Building Task (t[16] = performance shows promise. during an individual session. All of these
4.08, p < 0.001), with large effect sizes of Neither measure of self-efficacy findings are consistent with earlier find-
.90 and -1.20, respectively. This finding predicted pretest learning efficiency. The ings in the literature (Sanchez, 2009).
indicates that inservice teachers can Posttest GCSE only predicted posttest
become efficient at using and creating in learning efficiency on the Chair-Build- Implications
the environment over a 6-week time pe- ing Task (F[1, 16] = 4.726, p = .046), The Second Life MUVE appears to be a
riod. Participants’ self-reported engage- and it did so with a moderate R2 of .24. promising environment that fosters high
ment in the environment likely played a Interestingly, the posttest MUVE-SE levels of engagement in adult learners,
role in increasing learning efficiency, but demonstrated predictive ability for the supports synchronous online class ac-
because engagement was not empirically posttest Maze Task (F[1, 16] = 19.802, tivities as a distance-education delivery
measured, this study cannot shed any p < .001, R2 = .569) and Chair-Building platform, and provides a virtual environ-
light on this relationship. Task (F[1, 16] = 6.663, p = .021, R2 = ment in which educators and teacher
The learning curve percentages are .308). General computer self-efficacy educators may build simulated learning
useful to know. They seem to provide does not appear to be specific enough to environments. It works well with both
useful data about how rapidly a user predict learning efficiency in a MUVE. Moodle and SLOODLE.
will learn to build and interact in the The MUVE-SE instrument has some Using this MUVE is likely to improve
MUVE. Participants experienced the predictive ability, but this is only a pilot the GCSE of teachers and possibly other
greatest improvement in terms of the test of the instrument, which needs to adults. However, general computer self-
time it took them to complete tasks be refined through item analysis and efficacy does not appear to be directly
(48% for the Maze Task, 28% for the further validation measures before it is related to using a MUVE. That is why
Chair-Building Task), but also experi- ready for dissemination. the MUVE-SE instrument is promising;
enced double-digit reductions in PME Finally, participant impressions of its further development offers the po-
(Maze Task = 19%, Chair-Building using the Second Life MUVE for the tential of measuring self-efficacy related
Task = 33%). Learning efficiency im- class were generally positive. The major- to the use of a MUVE that may have
proved at the lowest rate of all (Maze ity of participants reported being highly the ability to predict learning efficiency
Task = 1.6%, Chair-Building Task = engaged when using the environment when using such an environment.
1.9%). Again, this finding is consistent and enjoying interacting in the environ- Users of Second Life and other
with participant reports of frustra- ment. Responses also indicated that using MUVEs can build individual objects
tion related to using and building the environment enhanced their level using the built-in tools of the MUVE
within the MUVE. These reports and of planning and thinking related to the itself, prim by prim. They can also build
findings are also consistent with the building of a simulated educational struc- objects outside of the MUVE in a variety
literature (Sanchez, 2009). Perhaps ture. Building in the environment took of programs, such as Google Sketch-up,
additional guided practice and tutorial some time for participants to get used to, that create three-dimensional objects
videos would have helped improve the however. This was supported by the slow and import the objects into the virtual
learning curve related to building and rate of improvement in learning efficiency environment. Additionally, it is possible
operating within the MUVE. scores. They reported frustration with to build objects within the MUVE and
Findings on self-efficacy indicate that learning to build due to the three-dimen- duplicate them so that the user can give
using a MUVE will likely contribute to sional nature of the environment and or sell the copy to another user within
the improvement of GCSE and MUVE the complexity of manipulating objects the MUVE. Having the skills to build
self-efficacy. Participants improved their with the user interface. Some participants or create within the MUVE will enable
GCSE from the pretest to the posttest by reported technical difficulties related to educators to create any simulated envi-
an average of 6.29 points and improved not having a recommended video card, a ronment for any subject they may wish
their MUVE-SE by an average of 56.47 robust enough microprocessor, enough to create for teaching purposes. Having
points over the 6-week duration of RAM, and/or a fast enough Internet con- prebuilt objects (created by someone
this investigation. The improvements nection. A few even reported feelings of else) would greatly facilitate the creation
in GCSE (t[16] = 2.25, p < 0.05) and nausea and motion sickness, eye strain, of virtual simulations by reducing the
MUVE-SE (t[16] = 7.16, p < 0.001) were blurred vision, and headaches due to need to build objects prim by prim,
statistically significant and had moderate prolonged use of the MUVE. The three- which should reduce the learning curve
to large effect sizes as measured by Co- dimensional nature of the environment, and may facilitate greater adoption of
hen’s d of -0.52 and -1.84, respectively. coupled with the feeling of immersion this type of technology.
Volume 27 Number 2 | Journal of Digital Learning in Teacher Education | 73
Copyright © 2010, ISTE (International Society for Technology in Education), 800.336.5191
(U.S. & Canada) or 541.302.3777 (Int’l), [e-mail], www.iste.org. All rights reserved.
Hearrington
For teacher educators, the main impli- from a large population of K–12 educa- from simple building to complex build-
cation seems to be that, with scaffolding tors. Therefore, it is not representative, ing tasks, should increase germane
and time, teachers can be taught to create and the results of this investigation cognitive load and facilitate learning
simple simulated environments in this may not be generalized. The researcher efficiency (van Merrienboer, Kester, &
MUVE. This capability will enable them intends to repeat this experiment with Paas, 2006). It would be useful to know
to use this type of tool to meet NETS- larger numbers of participants in the which combinations of methods of
T standards with their own students future. In addition, the study did not teaching simulation construction pro-
should they have the opportunity to do utilize a comparison group. It is my duce the greatest learning efficiency.
so. Results from this study support the intention to accomplish future itera- Finally, empirical measures of
conclusion that creation in a MUVE can tions of this study using some form of engagement should be developed and
support adult engagement and higher lev- control-group design. Finally, the validated for the purpose of measuring
els of thinking. The creation of advanced MUVE self-efficacy instrument that the the engagement of adults and children
simulations, requiring the scripting of researcher created will require more data when they are using a MUVE and other
object actions, was beyond the scope of to be validated. programs. It would be useful to be able
this study. However, this study suggests to compare engagement levels for users
that the creation of advanced simulations Suggestions for Further Research of a variety of simulations in a MUVE.
will likely require significant training Multi-user virtual environment self-
before educators are ready. Second Life efficacy is a construct that appears to Author Note
and other MUVEs may offer educational have some usefulness because of its Doug Hearrington is an assistant professor of
programming, instructional design, potential predictive abilities. A com- instructional technology at Kennesaw State Uni-
versity (KSU), near Atlanta, Georgia. KSU is the
and multimedia classes an environment parison of MUVE programs should be
third largest institution in the University System of
where students can create meaningful conducted to identify common features
Georgia. His research focuses on the design and im-
and engaging simulations. and user interface mechanisms. Based plementation of virtual and augmented reality to
Additionally, Second Life has the po- on these similarities, the MUVE-SE advance learning, metacognitive functioning, self-
tential to function as an environment in should be revised and validated with a efficacy, epistemological beliefs, self-determination,
as well as technology leadership and integration.
which to hold synchronous classes. Par- much larger sample.
He has presented at several state, regional, and
ticipants have to be taught how to be vir- Learning efficiency appears to be a
national conferences and is the project director of
tual students with their avatars, because useful framework on which to design Project OWL (Online World of Learning), a multi-
the environment presents difficulties comparisons of user interfaces and user virtual environment and curriculum for the
related to managing who gets to talk and programs. This methodology should advancement of third through fifth grade STEM
learning and teaching. Correspondence regarding
ask questions. Additionally, as partici- be validated with a much larger sample
this article should be addressed to Doug Hear-
pants reported, there is sometimes a lag of participants. To be generalizeable
rington, EdD, Assistant Professor of Instructional
between when an event begins, such as beyond Second Life, the Maze Task Technology, Department of Instructional Technol-
showing a PowerPoint slide, and when and Chair-Building Task should be ogy, Kennesaw State University, 1000 Chastain
that event actually takes place in the examined to determine the MUVE Road NW, Box 0127, Kennesaw, Georgia 30144.
E-mail: [e-mail]
MUVE. This is called lag-time, or just programs to which these tasks ap-
lag. Lag presents a challenge that Second ply. Learning efficiency could then be
References
Life has to overcome before it can be calculated for each MUVE, and those
Bailey, C. D. (1989). Forgetting and the learning
used to hold large numbers of avatars in results could be applied to choose
curve: A laboratory study. Management Science,
one place representing whole classes of the environment best suited to the 35, 346–352.
students. Those wishing to teach using a intended user population. Bandura, A. (1997). Self-efficacy: The exercise of
MUVE as a synchronous class space will Due to the complexity of build- control. New York: W.H. Freeman and Company.
Campbell, D. T., & Stanley, J. C. (1963).
have to learn to overcome these same ing in a MUVE, methods of increas-
Experimental and quasi-experimental designs for
challenges as well. ing germane cognitive load should be
research. Boston: Houghton Mifflin Company.
For researchers, this study offers a implemented and studied to deter- Clark, R., Nguyen, F., & Sweller, J. (2006).
potential contribution in the form of a mine their ability to increase learn- Efficiency in learning: Evidence-based guidelines
method with which to study and com- ing efficiency. Such methods include to manage cognitive load. San Francisco: John
Wiley & Sons, Inc.
pare MUVE user interfaces and learning structuring building lessons that start
Cobb, S., & Fraser, D. S. (2005). Multimedia learning
through tasks in a MUVE. Learning with a combination of simple partial
in virtual reality. In R. E. Mayer (Ed.), The
efficiency and learning curve seem well and whole tasks before progressing to Cambridge handbook of multimedia learning (pp.
suited to this task. more complex partial- and whole-task 525–547). New York: Cambridge University Press.
activities. Next, start lessons on build- Edwards, L. D. (1995). Microworlds as
representations. In A. A. diSessa, C. Hoyles, R.
Limitations ing with worked examples, continuing
Noss, & L. D. Edwards (Eds.), Computers and
The sample used for this study was not to completion tasks, and ending with
exploratory learning (pp. 127–154). New York:
very large, nor was it randomly drawn whole tasks. Combining this sequence, Springer.
74 | Journal of Digital Learning in Teacher Education | Volume 27 Number 2
Copyright © 2010, ISTE (International Society for Technology in Education), 800.336.5191
(U.S. & Canada) or 541.302.3777 (Int’l), [e-mail], www.iste.org. All rights reserved.