ERIC ED592733: Transactivity as a Predictor of Future Collaborative Knowledge Integration in Team-Based Learning in Online Courses

2016·0.47 MB·English

by  ERIC

Transactivity as a Predictor of Future
Collaborative Knowledge Integration in
Team-Based Learning in Online Courses
Miaomiao Wen, Keith Maki, Xu Wang, Steven P Dow, James Herbsleb and Carolyn Rose
CarnegieMellonUniversity
Pittsburgh,USA
{mwen,kmaki,xuwang,spdow,jdh,cprose}@cs.cmu.edu
ABSTRACT team-basedlearningfeatures(e.g.,inEdX2).Thereisaccu-
Tocreateasatisfyingsociallearningexperience,anemerging mulatingevidence that social interaction is associatedwith
challengeineducationaldataminingistoautomaticallyas- enhancedcommitmenttothecourse[11],whichhasthepo-
signstudentsintoeffectivelearningteams. Inthispaper,we tential to address one of MOOC critics’ biggest concerns,
utilizediscoursedataminingasthefoundationforanonline namelyhighattritionrates[18]. However,howtoautomat-
team-formation procedure. The procedure features a delib- icallyassignstudentstoeffectiveMOOClearninggroupsis
erationprocesspriortoteamassignment,whereparticipants still an open question [12, 25, 20]. Methods for mining
hold discussions both to prepare for the collaboration task educational data have been used to optimize instruction or
andprovideindicatorsthatarethenusedduringautomated feedback for individuals [21]. In this paper we explore how
team assignment. We automatically assign teams in a way a form of educational data mining (namely, mining of dis-
that maximizes average observed pairwise transactivity ex- cussionbehavior)canbeusedtooptimizetheexperienceof
changewithinteams,whereasinacontrolcondition,teams collaborativelearnersthroughthesupportofeffectiveteam
areassignedrandomly. Wevalidateourteam-formationpro- formation.
cedure in a crowdsourced online environment that enables
effective isolation of variables, namely Amazon’s Mechan- Algorithms for group assignment typically bring together
ical Turk. We compare group knowledge integration out- studentsbasedonlearningstyle,personalityordemographic
comesbetweenthetwoteamassignmentconditions. Ourre- information. For team assignments based on such algo-
sultsdemonstratethattransactivity-basedteamassignment rithms,studentinformationmustbecollectedandthenpro-
isassociatedwithsignificantlygreaterknowledgeintegration vided to the algorithm [9]. Because of the paucity of avail-
(p<.05,effectsize3standarddeviations). able student personal information in MOOCs, designing a
team-formation process that relies on mining of discussion
datatofillinmissinginformationwouldbeavaluablecontri-
1. INTRODUCTION bution. Moreover,researchidentifyingvaluableevidencefor
AlthoughtherearetypicallythousandsofstudentsinaMas- effective team formation is needed since recent work shows
sive Open Online Course (MOOC), social isolation is still that forming teams based on typical demographic features,
the norm in the current generation of MOOCs. However, e.g. gender and time zone, does not significantly improve
there is evidence that many students would prefer to have teams’ engagement and success in MOOCs [25]. In an on-
more social engagement in that context. Recent research line interaction, demographic information about learners is
showsthataquarteroflearnerswanttomeetnewpeoplein onlyrelevanttotheextentthatitinfluenceshowthosestu-
theircourses;andanother20%oflearnersintypicalMOOCs dentscomeacrossandinteractwithothers. Thus,observa-
want to take their courses with friends or colleagues [17]. tionofbehaviorandinteractionbetweenstudentsmaybea
Tosatisfylearners’socialneeds,thereisgrowinginterestin better source of insight for assigning students to groups in
enabling group learning in MOOC learning contexts. Re- which they will function well as a team. This provides an
centemergingplatformslikeNovoEd1 andcMOOCsarede- excellentopportunityfordataminingtechnologytomakea
signedwithteam-basedlearningorsocialinteractionatcen- contribution in support of valued learning processes. The
terstage. Additionally,manyrecentxMOOCsareadopting alternative to automated assignment is self-selected teams.
Whenastudentpopulationislarge,whichisusuallythecase
1https://novoed.com inMOOCs,itisdifficultforstudentstonavigatethrougha
list of students or teams to find a team that fits. Previous
workhasshownthatmanyself-selectedteamsfailinteam-
based MOOCs [23]. As an alternative to both of these ap-
proaches, we design a practical group-formation procedure
throughwhichparticipantsareorganizedintosmallgroups
2https://courses.edx.org/courses/course-v1:
McGillX+GROOCx+T3_2015,
https://www.edx.org/course/medicinal-chemistry-
molecular-basis-drug-davidsonx-d001x-1
Proceedings of the 9th International Conference on Educational Data Mining 533
based on the data mined from their participation processes ontheinteractionsdisplayedduringthiscommunitydiscus-
in the course. This procedure uses a deliberation process, sion, students are automatically assigned to teams. The
where participants hold discussions in preparation for the students then enter their teams for the bulk of their group
collaboration task; teams are then automatically assigned work. Wetestatransactivity-maximizationteam-formation
basedonfeaturesoftheirinteractionduringdeliberation. method. Instead of grouping students high in transactivity
into teams and students low in transactivity together, the
Inrecentyearstherehasbeenincreasinginterestinmining team assignment algorithm maximizes the average amount
discoursedataforinsightsintolearningprocesses[7],forun- of transactive communication within all the teams through
derstandingfactorsassociatedwithattritioninMOOCs[16], aconstraintsatisfactionalgorithm.
andforbuildingmodelstotriggerdynamicsupportforcol-
laborative learning [11]. In this paper, we mine students’
2.1 ExperimentalParadigm
collaborative process to collect information for automatic
team assignment. In particular, we automatically identify 2.1.1 CollaborationTaskDescription
animportantpropertyofdiscourse,transactivity,fromstu- Fortheteamtask,wedesignedahighly-interdependentcol-
dents’discussion. Transactivityisknowntobehigherwithin laborationtaskthatrequiresnegotiationinordertocreatea
groups where there is mutual respect [5] and a desire to contextinwhicheffectivegroupcollaborationwouldbenec-
build common ground [14]. Previous studies showed that essary for task success. The task is comparable to a course
high transactivity groups are associated with higher learn- project where a student team writes a proposal collabora-
ing[22],higherknowledgetransfer[13],andbetterproblem tively. WeusedaJigsawparadigm,whichhasbeendemon-
solving [5]. Prior work has demonstrated success at auto- stratedasaneffectivewaytoachieveapositivegroupcom-
maticdetectionoftransactivityandrelevantdiscussioncon- positionandisassociatedwithpositivegroupoutcomes[4].
structs[14]. Becauseofthesocialunderpinningsoftransac- InaJigsawtask,eachstudentisgivenaportionoftheknowl-
tivity,itisreasonabletohypothesizethatautomateddetec- edge or resources needed to solve the problem, but no one
tionof transactivity could formthe basis for anautomated hasenoughtocompletethetaskalone. FollowingtheJigsaw
group assignment procedure in online learning contexts. In paradigm,eachmemberoftheteamwasgivenspecialknowl-
this paper, we combine text-mining and algorithm-based edge of one of the four energy sources, and was instructed
teamformation;Westudywhetherbygroupingindividuals to represent the values associated with their energy source
with a history of engaging in more transactive communica- in contrast to the rest, e.g. coal energy was paired with
tion during a pre-collaboration deliberation can help them an economical energy perspective. The team collaborative
achieve more effective collaboration in their teams. Simply taskwastoselectasingleenergyplanandwriteaproposal
stated,ourresearchquestionis: arguing in favor of the group decision with respect to the
associatedtrade-offs,meaningteammembersneededtone-
Can evidence of transactive discussions during deliberation gotiate a prioritization among the city requirements with
inform the formation of more successful teams? respect to the advantages and disadvantages they were cu-
mulatively aware of. The set of potential energy plans was
Asasteptowardseffectiveteam-basedlearninginMOOCs, constructedtoreflectdifferenttrade-offsamongtherequire-
in this paper, we explore the team-formation process in an ments, with no plan satisfying all of them perfectly. This
experimental study conducted in an online setting that en- ambiguity created an opportunity for intensive exchange of
ables effective isolation of variables, namely Amazon’s Me- perspectives. ThecollaborationtaskisshowninFigure1.
chanical Turk (MTurk). While crowd workers likely have
differentmotivationsfromMOOCstudents,theirremotein-
2.1.2 ExperimentalProcedure
dividualworksettingwithoutpeercontactresemblestoday’s
Wedesignedafour-stepprocessforthetask:
MOOC setting where most students learn in isolation [6].
Step 1: Preparation. In this step, each student was asked
This allows us to test the causal connection between vari-
toprovideanickname,whichwouldbeusedinthedeliber-
ables in order to identify principles that later we will test
ation and collaboration phases. To prepare for the Jigsaw
inanactualMOOC.Asimilarapproachwastakeninprior
task, each student was randomly assigned to read an in-
work to inform design of MOOC interventions for online
structionalarticleabouttheprosandconsofasingleenergy
group learning [6]. We designed a collaborative knowledge
source. Eacharticlewasapproximately500words,andcov-
integration task where participants work together on writ-
eredoneoffourenergysources(coal,wind,nuclear,andhy-
inganenergyproposalforacity. Thisknowledgeintegration
dropower). Tostrengthentheirlearningandpreparethem
taskismodeledafteronesusedinearliercollaborativelearn-
fortheproposalwriting, weaskedthemtocompleteaquiz
ingstudies[4]. Theparticipantsinourstudywillbereferred
reinforcing the content of their assigned article. The quiz
toasstudentsthroughoutthepaper.
consisted of 8 single-choice questions, and feedback includ-
ing correct answers and explanations was provided along
2. METHODS withthequiz.
Ourexperimentalstudyisdesignedasavalidationofateam-
formation paradigm. In this paradigm, we attempt to offer Step 2: Pre-task. In this step, we asked each student to
teams a running start in their collaboration work by start- writeaproposaltorecommendoneofthefourenergysources
ing them with individual work, which they then discuss as (coal, wind, nuclear, and hydro power) for a city given five
acommunity. Inadditiontoprovidingthebasisforassign- requirements, e.g. “The city prefers a stable energy”. After
ment to teams, the community engagement prior to team eachstudentfinishedthisstep,theirproposalwasautomat-
formation provides students with a breadth of exposure to ically posted in a forum as the start of a thread with the
different perspectives relevant to the group work. Based title“[Nickname]’sProposal”.
Proceedings of the 9th International Conference on Educational Data Mining 534
In this final step, you will work together with other Turkers to recommend a way of distributing resources across energy
types for the administration of City B. City B requires 12,000,000 MWh electricity a year from four types of energy
sources: coal power, wind power, nuclear power and hydro power. We have provided 4 different plans to choose from,
each of which emphasizes one energy source as primary. Your team needs to negotiate which plan is the best way of
meetingyourassignedgoals,giventhecity’srequirementsandinformationbelow.
CityB’srequirementsandinformation:
1. City B has a tight yearly energy budget of $900,000K. Coal power costs $40/MWh. Nuclear power costs $100/MWh.
Windpowercosts$70/MWh. Hydropowercosts$100/MWh.
2. The city is concerned with chemical waste. If the main energy source releases toxic chemical waste, there is a waste
disposalcostof$2/MWh.
3. Thecityisafamoustouristcityforitsnaturalbirdandfishhabitats.
4. The city is trying to reduce greenhouse gas emissions. If the main energy source releases greenhouse gases, there will
bea“Carbontax”of$10/MWhofelectricity.
5. Thecityhasseverallargehospitalsthatneedastableandreliableenergysource.
6. The city prefers renewable energy. If renewable energies generate more than 30% of the electricity, there will be a
renewabletaxcreditof$1/MWhfortheelectricitythatisgeneratedbyrenewableenergies.
7. Thecityprefersenergysourceswhosecostisstable.
8. Thecityisconcernedwithwaterpollution.
Energy Plan Cost Waste disposal Carbon Renewable Total
Coal Wind Nuclear Hydro cost tax tax credit
Plan 1 40% 20% 20% 20% $840,000K $14,400K $48,000K $9,600K $892,800K
Plan 2 20% 40% 20% 20% $912,000K $0 $0 $11,000K $901,000K
Plan 3 20% 20% 40% 20% $984,000K $14,400K $0 $9,600K $988,800K
Plan 4 20% 20% 20% 40% $984,000K $0 $0 $11,000K $973,600K
Figure 1: This figure displays the collaborative task as it was presented to the students. In addition to the
task statement, they had a chat interface and a shared document space to work in.
Step3: Deliberation. Inthisstep,studentsjoinedathreaded making statements based on reasoning from simply repeat-
forumdiscussionakintothoseavailableinmanyonlineen- ing what they have read. In particular, the scoring rubric
vironments. Each proposal written by the students in the definedhowtoidentifythefollowingelementsforaproposal:
Pre-task (Step 2) was displayed for students to read and (1) Which requirements were considered; (2) Which com-
comment on. Each student was required to write at least parisonsortrade-offsweremade;(3)Whichadditionalvalid
five replies to the proposals posted by the other students. desideratawereconsideredbeyondstatedrequirements;(4)
Toencouragethestudentstodiscusstransactively,thetask Whichincorrectstatementsweremadeaboutrequirements.
instruction for this step included the request to, when re- Positive points were awarded to each proposal for correct
plying to a post,“elaborate, build upon, question or argue requirementsconsidered,comparisonsmade,andadditional
against the ideas presented in that post, drawing from the validdesiderata. Negativepointswereawardedforincorrect
argumentationinyourownproposalwhereappropriate.” statements. We measured Team Knowledge Integration by
the total points assigned to the team proposal, i.e. team
Step4: Collaboration. Inthecollaborationstep,teammem- proposal score. Two PhD students who were blind to the
bersinagroupwerefirstgatheredforsynchronousinterac- conditionsappliedtherubrictofiveproposals(atotalof78
tionandthendirectedtoashareddocumentspacetowrite sentences) and the inter-rater reliability was good (Kappa
aproposaltogethertorecommendoneoffoursuggesteden- =0.74). Thetworatersthencodedalltheproposals.
ergy plans based on a city’s eight requirements. Students
inthesameteamwereabletoseeeachother’seditsinreal We used the length of chat discussion during teamwork as
time,andwereabletocommunicatewitheachotherusinga a measure of team process in the Collaboration step. On
synchronous chat utility on the right sidebar. The collabo- average the longer discussions referred to more substantive
rativetaskwasdesignedtocontainricherinformationthan issues.
theindividualproposalwritingtaskinStep2.
Group Experience Satisfaction was measured using a
2.1.3 OutcomeMeasures fouritemgroupexperiencesurveyadministeredtoeachstu-
dentaftertheCollaborationstep. Thesurveywasbasedon
We evaluated team success using two types of outcomes,
items used in prior work [19, 6]. In particular, the survey
namely objective success through quantitative task perfor-
instrumentincludeditemsrelatedto:
mance(i.e.,thequalityoftheintegratedproposal,whichin-
dicatescollaborativeknowledgeintegration[3])andprocess Satisfactionwithteamexperience.
measures,aswellassubjectivesuccessthroughagroupsatis- •
faction survey. The quantitative task performance measure Satisfactionwithproposalquality.
•
wasanevaluationofthequalityoftheproposalproducedby
Satisfactionwiththegroupcommunication.
the team. The goal of evaluating the team knowledge inte- •
grationprocessistodistinguishinstanceswhenstudentsare Perceivedlearningthroughthegroupexperience.
•
Proceedings of the 9th International Conference on Educational Data Mining 535
Eachoftheitemswasmeasuredona7-pointLikertscale. was maximized. A Minimal Cost Max Network Flow algo-
rithm was used to perform this constraint satisfaction pro-
cess [1]. This network flow algorithm tackles resource allo-
2.1.4 ControlVariables
cation problems with constraints. In our case, we need to
Intuitively,studentswhodisplaymoreeffortinthePre-task
satisfytheJigsawconstraint. Atthesametime,theminimal
mightperformbetterinthecollaborationtask,sothatlevel
cost part of the algorithm maximized the transactive com-
ofeffortisanimportantcontrolvariable. Weusedeachstu-
munication that was observed among the group members
dent’sindividualPre-taskproposallengthasacontrolvari-
during the Deliberation step. The algorithm finds an ap-
ableforIndividualPerformance. Analogously,weusedeach
proximately optimal grouping within O(N3) (N = number
group’s average group member Pre-task proposal length as
of students) time complexity. A brute force search algo-
acontrolvariableforthegroupknowledgeintegrationanal-
rithm,whichhasanO(N!)timecomplexity,wouldtaketoo
yses.
longtofinishinrealtime.
2.1.5 TransactivityAnnotation,Prediction,andMea- Our algorithm can achieve an approximately optimal so-
surement lution in an admissible time. Instead of maximizing the
pair-wise accumulated transitivity post count, we approx-
Toenableustousecountsoftransactivecontributionsasev-
imate the solution by maximizing the accumulated transi-
idencetoinformanautomatedgroupassignmentprocedure,
tivity post count between two adjacent pairs of users. The
we needed to automatically judge whether a reply post in
algorithm can be generalized to form teams of any size. In
the Deliberation step was transactive or not using machine
our experiment, the team size is 4. We build a directed
learning. A transactive contribution displays the author’s
weightedgraphbasedonstudents’discussionnetwork. Then
reasoning and connects that reasoning to material commu-
weusethesuccessiveshortestpathalgorithmtofindasub-
nicatedearlier. Twoexamplepostsillustratingthecontrast
optimal, butnevertheless substantially better than random
areshownbelow:
grouping[1]. Thealgorithmgreedilyfindsaflowwithmin-
Transactive imum cost until there is no remaining flow in the network,
• “Nuclearenergy,asitisefficient,itisnotsustainable. asoutlinedinAlgorithm1.
Also, think of the disaster probabilities”.
Algorithm 1SuccessiveShortestPathsforMinimumCost
Non-transactive MaxFlow
•
“Iagreethatnuclearpowerwouldbethebestsolution”. f(v ,v ) 0 (v ,v ) E
1 2 ← ∀ 1 2 ∈
Using a validated and reliable coding manual for transac- E0←a(v1,v2)∀(v1,v2)∈E
tivityfrompriorwork[14],anannotatorpreviouslytrained while∃Π∗∈G0=(V,E0)
toapplythatcodingmanualannotated426replypostscol- s.t. Π∗ a minimum cost path from S to D do
lected in pilot studies we conducted in preparation for the for each(v1,v2)∈Π∗
if f(v ,v )>0then
studiesreportedinthispaper. Eachofthosepostswasan- 1 2
f(v ,v ) 0
notatedaseither“transactive”or“non-transactive”. 70%of 1 2 ←
themweretransactive. remove−a(v2,v1)fromE0
adda(v1,v2)toE0
else
Automatic annotation of transactivity has been reported f(v ,v ) 1
in the Computer Supported Collaborative Learning litera- rem1ove2a(←v1,v2)fromE0
tinugre.usFinogr etxeaxmt,psleu,chresaesarcchhaetrsdhaatvae[a1p5p]laienddmtaracnhsincreipletasrno-f add−a(v2,v1)toE0
end
whole group discussions [2]. We trained a Logistic Regres-
end
sion model with L2 regularization using a set of features
consistingofsinglewordfeatures(i.e.,unigrams)aswellas
a feature indicating the post length [10]. We evaluated our
2.2.1 ExperimentalManipulation
classifierwitha10-foldcrossvalidationandachievedanac-
Inourstudy,studentsparticipatedinadeliberativediscus-
curacy of 0.843 and a 0.615 Kappa. Given the adequate
sion as a community in a threaded discussion forum prior
performance of the model, we used it to predict whether
to being assigned to teams automatically. We investigated
each reply post in the Deliberation step was transactive or
how the nature of the experience in that context may con-
not.
tributetothesuccessoftheteams. WemadeuseofaJigsaw
paradigmintheteamassignmentofteamsinboththeexper-
To measure the amount of transactive communication be-
imental and control conditions. In the experimental condi-
tweentwostudentsintheDeliberationstep,wecountedthe
tion,whichwetermedtheTransactivityMaximizationcon-
number of times a pair of their posts in a same discussion
dition, we additionally applied a constraint that preferred
threadweretransactive;oroneofthemwasathreadstarter
to maximize the extent to which students assigned to the
andtheotherstudent’sreplywastransactive.
sameteamhadparticipatedinautomaticallydetectedtrans-
active exchanges in the deliberation. In the control condi-
2.2 TransactivityMaximizationGrouping tion, which we termed the Random condition, apart from
TheTransactivityMaximizationteamswereformedsothat enforcingtheJigsawconstraint,teamswereformedbyran-
the average amount of transactive discussion observed in domassignment. Inthiswaywetestedthehypothesisthat
theDeliberationstepamongtheteammembersintheteam observed transactivity is an indicator of potential for effec-
Proceedings of the 9th International Conference on Educational Data Mining 536
tive team collaboration. We ran the study in 10 separate alsoindicatedteamsthatexperiencedmoretransactivecom-
batches,with5batchesineachcondition. Ineachbatch,all munication during deliberation demonstrated better Team
thestudentsinthatbatchwereassignedtoteamsusingthe KnowledgeIntegration.
Randomstrategyorallthestudentswereassignedtoteams
using the Transactivity Maximization strategy. The aver- Teams that experienced greater transactivity during deliber-
agelevelofamountoftransactivityduringthedeliberation ation demonstrate more intensive interaction within their
stage was not significantly different between batches. Thus teams.
we can test if the team-formation method can predict fu- Intheexperiment,studentswereassignedtoteamsbasedon
ture collaborative knowledge integration. All the steps and observedtransactivecommunicationduringthedeliberation
instructionsofthetaskwereidenticalforthetwoconditions, step. Assumingthatindividualsthatwereabletoengagein
asdescribedin2.1.2. positivecollaborativebehaviorstogetherduringthedeliber-
ationwouldcontinuetodosoonceintheirteams,wewould
2.3 Participants expecttoseeevidenceofthisreflectedintheirobservedteam
process. Group processes have been demonstrated to be
ParticipantswererecruitedonMTurkwiththequalifications
strongly related to group outcomes in face-to-face problem
of having a 95% acceptance rate on 1,000 tasks or more.
solvingsettings[24]. Thus,weshouldconsiderevidenceofa
Eachstudentwasonlyallowedtoparticipateonce. Atotal
positive effect on group processes as an additional positive
of246studentsparticipatedintheexperiment,thestudents
outcomeoftheexperimentalmanipulation.
who were not assigned into groups or did not complete the
group satisfaction survey were excluded from our analysis.
Inordertotestwhethersuchaneffectoccurred,webuiltan
Theexperimentlastedonaverage35.9minutes. Weincluded
ANOVA model with Grouping Criteria (Random, Transac-
onlyteamsof4studentsinouranalysis. Therewereintotal
tivityMaximization)astheindependentvariableandlength
27TransactiveMaximizationteamsand27Randomteams,
of chat discussion during teamwork as the dependent vari-
withnosignificantdifferenceinattritionbetweenconditions
(χ2(1) = 1.46, p = 0.23). The dropout rate of students in able. TherewasasignificanteffectofGroupingCriteriaon
length of discussion (F(1,45) = 9.26, p < 0.005). Random
Random groups was 27%. The dropout rate of students in
groups (M = 20.00, SD = 3.58) demonstrated significantly
TransactivityMaximizationgroupswas19%.
shorter discussions than Transactive Maximization groups
(M=34.52,SD=3.16),withaneffectsizeof4.06standard
3. RESULTS deviations.
Asamanipulationcheck,wecomparedtheaverageamount
of transactivity observed among teammates during the de- Survey results
liberation between the two conditions using a t-test. The For each of the four aspects of the group experience sur-
groupsintheTransactiveMaximizationcondition(M=12.85, vey, we built an ANOVA model with Grouping Criteria
SD = 1.34)3 were observed to have had significantly more (Random, Transactivity Maximization) as the independent
transactiveexchangesduringthedeliberationthanthosein variableandthesurveyoutcomeasthedependentvariable.
the Random condition (M = 7.00, SD = 1.52) (p < 0.01), TeamIDandassignedenergycondition(Coal,Wind,Hydro,
withaneffectsizeof3.85standarddeviations,demonstrat- Nuclear) were included as control variables nested within
ing that the maximization was successful in manipulating condition. There were no significant effects on Satisfaction
the average experienced transactive exchange within teams with team experience or with proposal quality. However,
betweenconditions. there was a significant effect of condition on Satisfaction
with communication within the group (F(1,112) = 4.83,
Teams that experienced greater transactivity during deliber- p<0.05),suchthatstudentsintheRandomteams(M=5.12,
ation demonstrate better team knowledge integration. SD=1.7)ratedthecommunicationsignificantlylowerthan
ToassesswhethertheTransactivityMaximizationcondition those in the Transactivity Maximization teams (M = 5.69,
resulted in more effective teams, we tested for a difference SD = 1.51), with effect size 0.38 standard deviations. Ad-
betweengroup-formationconditionsonTeamKnowledgeIn- ditionally, there was a marginal effect of condition on Per-
tegration. WebuiltanANOVAmodelwithGroupingCrite- ceived learning (F(1,112) = 2.72, p = 0.1), such that stu-
ria (Random, Transactivity Maximization) as the indepen- dents in the Random teams (M = 5.25, SD = 1.42) rated
dent variable and Team Knowledge Integration as the de- the perceived benefit to their understanding they received
pendentvariable. Averageteammember Pre-taskproposal fromthegroupworklowerthanstudentsintheTransactivity
lengthwasagainthecovariate. Therewasasignificantmain Maximizationteams(M=5.55,SD=1.27),witheffectsize
effect of Grouping Criteria (F(1,52) = 6.13, p < 0.05) on 0.21 standard deviations. Thus, with respect to subjective
TeamKnowledgeIntegrationsuchthatTransactivityMaxi- experience, we see advantages for the Transactivity Maxi-
mizationteams(M=11.74,SD=0.67)demonstratedsignif- mization condition, but the results are weaker than those
icantlybetterperformancethantheRandomgroups(M=9.37, observedfortheobjectivemeasures. Nevertheless,thesere-
SD = 0.67) (p < 0.05), with an effect size of 3.54 standard sultsareconsistentwithpriorworkwhereobjectivelymea-
deviations,whichisalargeeffect. Effectsizeismeasuredin sured learning benefits are observed in high transactivity
termsofCohen’sd. teams[8].
Across the two conditions, observed transactive communi-
4. DISCUSSION
cation during deliberation was significantly correlated with
Inthispaperwepresentedanexperimenttoaddressourre-
Team Knowledge Integration (r = 0.26, p < 0.05). This
searchquestionregardingtheextenttowhichbenefitcould
3SDisshortforstandarddeviationinthispaper. be achieved by selecting teams based on evidence of trans-
Proceedings of the 9th International Conference on Educational Data Mining 537
active exchange observed during the deliberation. We de- C.Rose.Fosteringdiscussionacrosscommunication
signed an automatic team-formation process that combines mediainmassiveopenonlinecourses.InCSCL,2015.
discoursedataminingandalgorithm-basedteamformation. [12] K.Ghadiri,M.H.Qayoumi,E.Junn,P.Hsu,and
Herewefoundthatteamsformedsuchthatobservedtrans- S.Sujitparapitaya.Thetransformativepotentialof
active interactions between team members in the delibera- blendedlearningusingmitedx6.002xonlinemooc
tionwasmaximizeddisplayedobjectivelybetterknowledge contentcombinedwithstudentteam-basedlearningin
integration than teams assigned randomly. On subjective class.environment,8:14,2013.
measures we see a significant positive impact of transactiv- [13] G.Gweon.Assessment and support of the idea
ity maximization on perceived communication quality and co-construction process that influences collaboration.
a marginal impact on perceived enhanced understanding, PhDthesis,CarnegieMellonUniversity,2012.
both of which are consistent with what we would expect [14] G.Gweon,M.Jain,J.McDonough,B.Raj,andC.P.
from the literature on transactivity where high transactiv- Ros´e.Measuringprevalenceofother-oriented
ityteamshavebeendemonstratedtoproducehigherquality transactivecontributionsusinganautomatedmeasure
outcomes and greater learning [22]. These results provide ofspeechstyleaccommodation.International Journal
positive evidence in favor of a design for a team-formation of Computer-Supported Collaborative Learning,
strategyintwostages: Individualsfirstparticipateinapre- 8(2):245–265,2013.
teamworkdeliberationactivitywheretheyexplorethespace
[15] M.JoshiandC.P.Ros´e.Usingtransactivityin
ofissuesinacontextthatprovidesbeneficialexposuretoa
conversationforsummarizationofeducational
wide range of perspectives. Individuals are then grouped
dialogue.InSLaTE,pages53–56,2007.
automatically through a transactivity detection and maxi-
[16] R.F.Kizilcec,C.Piech,andE.Schneider.
mization procedure that uses communication patterns aris-
Deconstructingdisengagement: analyzinglearner
ingnaturallyfromcommunityprocessestoinformgroupfor-
subpopulationsinmassiveopenonlinecourses.In
mationwithanaimforsuccessfulcollaboration.
Proceedings of the Third International Conference on
Learning Analytics and Knowledge,pages170–179.
ThisresearchwassupportedinpartbyfundingfromGoogle
ACM,2013.
andtheGatesfoundation.
[17] R.F.KizilcecandE.Schneider.Motivationasalens
tounderstandonlinelearners: Towarddata-driven
5. REFERENCES designwiththeoleiscale.ACM Transactions on
[1] R.K.AhujaandJ.B.Orlin.Afastandsimple Computer-HumanInteraction(TOCHI),22(2):6,2015.
algorithmforthemaximumflowproblem.Operations [18] C.Kulkarni,J.Cambre,Y.Kotturi,M.S.Bernstein,
Research,37(5):748–759,1989. andS.Klemmer.Talkabout: Makingdistancematter
[2] H.Ai,R.Kumar,D.Nguyen,A.Nagasunder,and withsmallgroupsinmassiveclasses.InCSCW,2015.
C.P.Ros´e.Exploringtheeffectivenessofsocial [19] I.Lykourentzou,A.Antoniou,andY.Naudet.
capabilitiesandgoalalignmentincomputersupported Matchingorcrashing? personality-basedteam
collaborativelearning.InIntelligent Tutoring Systems, formationincrowdsourcingenvironments.arXiv
pages134–143.Springer,2010. preprint arXiv:1501.06313,2015.
[3] M.AlaviandA.Tiwana.Knowledgeintegrationin [20] S.MacNeil,C.Latulipe,B.Long,andA.Yadav.
virtualteams: Thepotentialroleofkms.Journal of Exploringlightweightteamsinadistributedlearning
the American Society for Information Science and environment.InProceedings of the 47th ACM
Technology,53(12):1029–1037,2002. Technical Symposium on Computing Science
[4] E.Aronson.The jigsaw classroom.Sage,1978. Education,pages193–198.ACM,2016.
[5] M.AzmitiaandR.Montgomery.Friendship, [21] C.Piech,J.Huang,A.Nguyen,M.Phulsuksombati,
transactivedialogues,andthedevelopmentofscientific M.Sahami,andL.Guibas.Learningprogram
reasoning.Social development,2(3):202–221,1993. embeddingstopropagatefeedbackonstudentcode.In
[6] D.Coetzee,S.Lim,A.Fox,B.Hartmann,andM.A. Proceedings of the 32nd International Conference on
Hearst.Structuringinteractionsforlarge-scale Machine Learning (ICML-15),pages1093–1102,2015.
synchronouspeerlearning.InCSCW,2015. [22] S.D.Teasley,F.Fischer,A.Weinberger,
[7] M.Dascalu,S.Trausan-Matu,D.S.McNamara,and K.Stegmann,P.Dillenbourg,M.Kapur,andM.Chi.
P.Dessus.Readerbench: Automatedevaluationof Cognitiveconvergenceincollaborativelearning.In
collaborationbasedoncohesionanddialogism. International conference for the learning sciences,
International Journal of Computer-Supported pages360–367,2008.
Collaborative Learning,10(4):395–423,2015. [23] M.Wen,D.Yang,andC.P.Ros´e.Virtualteamsin
[8] R.DeLisiandS.L.Golbeck.Implicationsof massiveopenonlinecourses.Proceedings of Artificial
piagetiantheoryforpeerlearning.1999. Intelligence in Education,2015.
[9] R.Decker.Managementteamformationforlargescale [24] A.W.Woolley,C.F.Chabris,A.Pentland,
simulations.Developments in Business Simulation and N.Hashmi,andT.W.Malone.Evidencefora
Experiential Learning,22,1995. collectiveintelligencefactorintheperformanceof
[10] R.-E.Fan,K.-W.Chang,C.-J.Hsieh,X.-R.Wang, humangroups.science,330(6004):686–688,2010.
andC.-J.Lin.Liblinear: Alibraryforlargelinear [25] Z.Zheng,T.Vogelsang,andN.Pinkwart.Theimpact
classification.The Journal of Machine Learning ofsmalllearninggroupcompositiononstudent
Research,9:1871–1874,2008. engagementandsuccessinamooc.Proceedings of
[11] O.Ferschke,I.Howley,G.Tomar,D.Yang,and Educational Data Mining,7,2015.
Proceedings of the 9th International Conference on Educational Data Mining 538