ERIC EJ1135024: Evidence of Sustainable Learning from the Mastery Rubric for Ethical Reasoning

2017·0.26 MB·English

by  ERIC

education
sciences
Article
Evidence of Sustainable Learning from the Mastery
Rubric for Ethical Reasoning
RochelleE.Tractenberg1,*,KevinT.FitzGerald2andJeffCollmann3
1 CollaborativeforResearchonOutcomesandMetrics;DepartmentsofNeurology;Biostatistics,
Bioinformatics&Biomathematics;RehabilitationMedicine,GeorgetownUniversityMedicalCenter,
Suite207BuildingD,4000ReservoirRoadNW,Washington,DC20057,USA
2 CatholicHealthCareEthics;PellegrinoCenterforClinicalBioethics;DepartmentofOncology,
GeorgetownUniversityMedicalCenter,Suite236,BuildingD,4000ReservoirRoadNW,Washington,
DC20057,USA;[email protected]
3 ProfessorEmeritus,GeorgetownUniversity,37th&OStreet,N.W.,Washington,DC20057,USA;
[email protected]
* Correspondence:[email protected]
AcademicEditor:JamesAlbright
Received:8July2016;Accepted:8December2016;Published:23December2016
Abstract: Interestinsustainablelearninghasbeengrowingoverthepast20yearsbutithasnever
beendeterminedwhetherstudents—whoselearningwearetryingtosustain—canperceiveeither
thesustainabilityoftheirlearningoranyofthefeaturesofthisconstruct. Afour-itemsurveywas
developedbasedonapublisheddefinitionof“sustainablelearning”,andwassenttothe12graduate
studentswhohavecompletedanewseminarinethicalreasoning.Athematicanalysisofthenarrative
responseswassubmittedtoadegrees-of-freedomanalysistodeterminethelevelandtypeofevidence
forstudentperceptionofsustainability. Respondents(n=9)endorsedeachofthefourdimensionsof
sustainablelearning—andeachgaveexamplesforeachdimensionoutsideof,andaftertheendof,
the course. One respondent endorsed all dimensions of sustainable learning, but was uncertain
whetherthecourseitselfledtooneparticularsustainabilitydimension. Whiletheseresultsmustbe
consideredpreliminarybecauseoursampleissmallandthesurveyisthefirstofitskind,theysuggest
thatgraduatestudentscananddoperceiveeachofthefourfeaturesofsustainability. Thesurvey
needs refinement for future/wider use; but this four-dimensional definition could be useful to
developandpromote(andassess)sustainablelearninginhighereducation.
Keywords: Mastery Rubric; sustainable learning; ethics education; metacognition; actionable
evidenceoflearning
1. Introduction
“Sustainablelearning”isdefinedaslearningthatcontinuesbeyondtheendofformalinstruction[1].
InterestinsustainablelearninghasbeenbuildingineducationalcommunitiesoutsidetheUnitedStates
(e.g.,[2–5]),althoughsomeoftheconversationfocusesontheroleofassessmentinthesustainabilityof
learning[2,3,6];seealso[7]. WithintheUnitedStates(althoughnotexclusively),sustainablelearning
has been studied as “transfer”, the application of learned skills or knowledge from the learned-in
context to other contexts (excellently reviewed by Barnett and Ceci [8] and extensively discussed
by Ambrose et al [9]; particularly in Chapter 4). These latter references are generally (but neither
explicitlynorexclusively)focusedonundergraduateeducationandworkplacetraining. Inmedical
education, asimilarconstructis“lifelonglearning”(e.g.,[10]); however, “lifelonglearning”isnot
closelyalignedwithtransferorsustainability. Infact,withinmedicalandnursingeducationatleast,
“lifelongeducation”isdefinedinamannerthatisinconsistentwitheither“transfer”or“sustainability”,
Educ.Sci.2017,7,2;doi:10.3390/educsci7010002 www.mdpi.com/journal/education
Educ.Sci.2017,7,2 2of23
and instead focuses on maintaining competency with respect to the “state of the science” and
“keeping up to date” [11] (p. 15)—i.e., continuing to learn new things relevant to the profession
(continued professional development). Continuing professional development is an expectation in
manyfields,includingmedicine(e.g.,[10]),nursing(e.g.,[11])andstatistics[12]. Theassessmentof
ongoinggrowthanddevelopmentoftheknowledge,skills,andabilitiesthattheeducationalexperience
wereintendedtoinitiatebeyondtheendofformalinstruction(i.e., thesustainabilityofthatlearning)
ischallengingineveryeducationalcontext(undergraduate,graduate,post-graduate/professional).
Inarecentpolicystatement,theNationalInstituteforLearningOutcomesAssessment(NILOA)[13]
articulatedfiveprinciplesfordocumentinglearningoutcomesinhighereducation:
• Develop/articulatespecificactionablelearningoutcomes;
• Connectlearninggoalswithstudentwork;
• Articulatelearningoutcomescollaboratively;
• Outcomessupportassessmentthatgeneratesactionableevidence;and
• Outcomesarefocusedonimprovement.
The sustainability of learning may be a plausible addition to existing objectives for the
documentationandassessmentoflearningoutcomes[13]. Itcouldconceivablybeintegratedintothe
articulationof“specific,actionablelearningoutcomes”,orbeconsidered“assessmentthatgenerates
actionableevidence”. However,foreitheroftheseoptionstobeplausible,sustainabilityhastobe
assessableitself. Althoughtheconversationhasbeencontinuingforover15years[7],noevidencethat
studentscanperceive“sustainability”—orthatitcanbeassessedreliably—hasbeenpublished.
Knapper(2006)[1]declared“(l)ifelonglearningmeanseffectiveandsustainablelearning”,buthe
didnotdefinesustainability,whereasSchwänke(2008)[14](pp. 1–2)describedfourdistinctfeaturesof
“sustainablelearning”thatinclude,andgobeyond,transfer(amorecognitivemodel)andcontinuing
professionaldevelopment(amoreclinical-educationmodel). Thesefourdimensionsare:
1. Lifelong learning: an additional level of depth, or dimension, that you bring to a course or
experienceunrelatedtothe(primary)topic;
2. Changing your learning behavior as a result of the specific learning: describe how your
learning(fact-finding,thinking,understandingofsomething,orapproachtolearningsomething
new)changed;
3. Aprocessofpersonaldevelopmentcontinuingbeyondthecourse:somethingyoudid,orinitiated,
foryourownsenseoflearning(i.e.,nottakingacourseaspartofyourprogram,butalearningor
trainingexperiencethatyousought,created,oridentified—notalreadyplanned);
4. Deconstruction/reconstruction: anideaorconceptthatyouthoughtyouunderstood,butthatyou
recognizedyoudidnottrulyunderstand(deconstruction)andsosoughttounderstandmore
deeply,anddiscoveredanerrorinyouroriginalunderstandingthatyouremediedorsoughtto
remedy(reconstruction).
Whiletheconstruct“lifelonglearning”istypicallyonlytrackableinsofarasindividualsattend
workshops, read materials and/or answer multiple-choice questions on this content, or complete
other similar unindividualized work, Schwänke’s definition represents an opportunity to explore
sustainability in the (relatively) short term (see Appendix A). Moreover, this definition implicitly
includestransfer,anditalsodefines“lifelonglearning”implicitlyintermsoftransfer. Italsoexplicitly
incorporates metacognition, the knowledge of, and ability to regulate, one’s thinking ([15,16]).
Ambroseetal. (2010)[9], among others, argue that metacognition is one of seven principles that
promoteeffectivelearning;theNationalResearchCouncil(2001)[15](p.78)statesthat“(m)etacognition
is crucial to effective thinking and competent performance”. As noted, continuing professional
developmentand“lifelonglearning”—evidenceofwhichareobtainedoutsideoftheclassroomand
oftenininformalways—typicallydonotincludeoremphasizemetacognitivedevelopmentortransfer.
Educ.Sci.2017,7,2 3of23
In the context of higher education (undergraduate, graduate and post graduate training),
whiletransfermightbeakeyobjective(e.g.,[8]),therearemanychallengestobeovercomebefore
realsustainabilityinlearningcanbesuccessfullyintegratedintoinstructionalandlearningobjectives
(see, e.g., [2,6,7,17,18]). A significant challenge is that many instructors are experts in their own
discipline, but not in education or pedagogy. This makes them potentially excellent instructors,
butatthesametime, canmakeitdifficult, ifnotimpossible, forindividualinstructorstopromote
sustainabilityinthelearningoftheirstudents(see,e.g.,[9](Chapter4)). Arelatedchallengeisthat,
particularlyifyouteach“inthemajor”,thecoursesarestructuredandalignedwithadeepeninginterest,
experience,andknowledgebasethatisreiterated,andreinforced,throughoutaprogram. Because
studentsmaybetrainingforafutureinthedisciplineinwhichtheyarestudying,theirengagementwith
highereducationtoachievetheirdesiredfuture,andnottheactualinstructionitself,maybringabout
thesustainedlearning. Ifitseemslikestudentlearningissustainable,thereislittlereasonto“improve”
oreven“standardize”thisattribute. Itisalsodifficulttoidentifyapointinadegreeprogramwhere
thesustainablelearningwouldbeginorevenbebestinitiated. Thesearealsoconsiderationsinthe
assessabilityofsustainability.
However,thereareinstanceswheresustainabilitycannotbebroughttoacoursebythestudent’s
engagement with the discipline, nor can it come from the organization of a program of study:
for undergraduates these are the single courses outside the major that satisfy breadth or “general
education”requirements. Forgraduatestudentsandsomepostgraduatestudents,thesemightbea
singlerequired(“singleton”)courseinstatisticsorinethics/responsibleconductofresearch. Although
twooftheauthorshavetaughtbothofthesestand-alone“required”courses(RochelleE.Tractenberg,
statistics/KevinT.FitzGerald,ethics),inthismanuscriptwefocusonthepotentialtoidentifyand
assesssustainabilityfromwithinthecontextoftrainingintheresponsibleconductofresearch(RCR),
orresearchethics.
Likeother“required”courses,acourseonRCRisalsooftentheonlyformalexposurestudents
willhavetothetopic.However,unlikemostrequiredcourses,akeyassumptionfortraininginresearch
ethics,ortheresponsibleconductofresearch(RCR),isthatthesinglecourse(whetheritisthreehours
or a full semester) is sufficient to both promote and sustain research integrity for an entire career.
Most other singleton courses are intended to serve as a survey, overview, or general introduction
tothetopic;ifastudentcontemplatesutilizingthetopicofvirtuallyanyothersingletoncoursefor
futureworkorstudy,itislikelythatfurtherengagementwiththattopicwouldberecommendedor
recognizedasnecessarybythestudentortheinstitution. Ourjointexperienceinandwithgraduate
programs in the sciences is that this is rarely the case for courses in research ethics. The typical
paradigmisonecoursewiththesametrainingthatallresearchers,nomatterwhattheircareerstage,
role,orresponsibilities,mustcomplete,andcomplianceisthemaindocumentedlearningoutcome.
TheNationalAcademyofEngineering[19](p. 36)declaresthistobeaflawedmodelforethics
education;andNovossiolovaandSture(2012)[20]reviewedtheliteraturearoundethicseducation
anddescribehowthistypeofeducationalexperience—evenifsuccessfullydelivered—canfailtobe
broughttobearinpractice. Thisfailuremayarisefromthelearningitself(e.g.,[21]),orfromperceived
failures of the applicability of the ethics instruction to real world contexts and situations. From a
cognitivescientificperspective,thedominanttrainingparadigmfor“traininginethics”assumesthat
masteringtheinformationassociatedwithRCRtopics(functioningatthe“cognitivestage”)willlead
tothehabitsofmindthatcharacterizerealmasteryofthekeyconstructsinRCR(functioningatthe
“autonomousstage”[22](pp. 281–282)). Thisparadigmreplacesthecommunityvalueforintegrity
(e.g.,[23])withcommunityvaluefor“completingrequiredtraining”. Whilethismayalsobetruefor
othersinglecourses(outsideofamajorprogramofstudy),sustainabilityofthelearningthatisinitiated
intrainingfortheresponsibleconductofresearchisahigh—ifunacknowledged—priority. Acritical
feature ofthescholarlydiscussion ofsustainablelearningingeneral orwithrespect totrainingin
responsibleconductinresearch,isthatitisentirelyfocusedonwhatteacherscan/doimplementin
Educ.Sci.2017,7,2 4of23
theircoursestopromotesustainablelearning. Therehasneverbeenastudyofwhetherornotthe
construct“sustainablelearning”ismeaningfulorevenperceptibletostudents.
Oneoftheauthors(RochelleE.Tractenberg)createdtheMasteryRubric[24],atoolthatformally
and explicitly combines the knowledge, skills, and abilities (KSAs) that a given curriculum seeks
todelivertogetherwithadescriptionofhowperformanceofeachoftheseKSAsshouldchangeas
the learner moves from a more novice to a more expert level of achievement on each. A Mastery
Rubricisacurriculumbuildingandevaluationtool,similartoatraditionalrubric(e.g.,[25])inthat
thedesiredknowledge,skillsandabilitiesforacurriculum—ratherthananassignmentortask—are
outlinedtogetherwithperformancelevelsthatcharacterizetherespondentchangingfromnoviceto
proficiency[24], ratherthanfromtheworsttothebestgradeorscore. In2012, twooftheauthors
(RochelleE.Tractenberg,KevinT.FitzGerald)publishedtheMasteryRubricforEthicalReasoning
(MR-ER)[26]. TheMR-ERfocusesontheknowledge,skills,andabilities(KSAs)thatcompriseethical
reasoning, and performance levels follow a guild structure to support development from novice,
throughapprentice(beginner)tojourneyman(independence). Weincludedafourthlevel,“master”,
todifferentiatetheevidencerequiredtosupportaclaimthatanindividualhasachievedthejourneyman
(independentfunctioning)levelofperformanceofallreasoningKSAsfromtheevidencethatisrequired
tosupportaclaimthatanindividualisnotonlycapableofperformingethicalreasoningindependently
(journeyman), butalsohasempiricallydemonstratedtheirabilitiestodiagnoseandremediatethe
reasoningofless-advancedreasoners,i.e.,thattheyarecapableoftakingandtraininganapprentice,
i.e.,themaster[26,27].
A Mastery Rubric-based curriculum would specifically encourage individuals to reflectively
monitortheirowndevelopmentoftheidentifiedcurricularobjectives(knowledge,skills,andabilities)
sothatindividualshave—orknowtoseek—multipleopportunitiestolearn,practice,anddemonstrate
their mastery of specific knowledge, skills, and abilities that are consistent with the instructional
objectives [24]. Because it is intended to be public, i.e., accessible to both the students and the
instructors,aMasteryRubriccanpromotestudentmetacognitionbyencouragingthemtoevaluatethe
levelatwhichtheyperformspecificKSAs(see,e.g.,[4,7,15,24,27]).
Wecreatedasemester-longcoursebasedontheMR-ERthathasnowbeencompletedbythree
cohortsofgraduate(MastersandPhD)studentsinthebiomedicalsciences. Wedesignedthecourse
(syllabus in Appendix B) so that, over a semester, each of the ethical reasoning KSAs are taught,
and students are given practice employing, and metacognitively reflecting on, each KSA, using a
federally-recommendedtopiclist[28]toguideourselectionofrelevantcasestudies. Whiletrainingin
responsibleresearchisnotafocusofthismanuscript,itoutlinesthefirstevidencethattheMR-ERcan
supportacoursethatmightachievetheintentiontoencouragetransfer,reflection,andself-monitoring
inresponsibleconductinresearchbeyondthecourseitself(e.g.,see[29]). TheMR-ER-derivedcourse
happenstofocusonresearchethics,butthepointofthisstudyderivesfromtheorientationofthe
MasteryRubrictowardssupportingmetacognition. Becauseofthisorientation,andthedependenceof
sustainablelearningonmetacognition,wesoughttoprovidethefirstempiricaltest,toourknowledge,
thatsustainablelearning(asdefinedbySchwänke2008[14];seealso[2,19],andAppendixC)canbe
perceivedbystudents.
2. MaterialsandMethods
Thedevelopmentofthesurvey,structureofthecoursewhosesustainabilitywesoughtevidence
forfromourstudents,andouranalyticmethodsaredescribedbelowandsummarizedinAppendixD
outlininghowwemetcriteriaforreportingqualitativeresearch(followingtheconsolidatedcriteriafor
reportingqualitativeresearch,COREQ,[30]).
2.1. DevelopmentoftheSurvey
We created a four-item survey based on the features of sustainable learning as defined by
Schwänke(2008)[14]andenumeratedearlier. AscanbeseeninAppendixA,ouronlymodifications
Educ.Sci.2017,7,2 5of23
tothedefinitionofthesefeatureswastodirectrespondentattentiontoourcourse,andalsotodomains
outsidethecourseandtothetimeafterthecoursewasended. Thisfocuspreservedtheoriginaldefinition
features,whichwasapriorityforus,butitalsomadethesurveysospecifictoourcoursethatwewere
notabletoadministerittoa‘control’group. Thesurveywastheonlydatacollectioninstrumentand
thisstudywasitspilottest. Noonehaseversurveyedstudentsaboutwhether“sustainability”isa
characteristicoftheireducationalexperiencethattheycandetect. Inthis“proofofconcept”study,
wesoughtonlytodeterminewhetherstudentscanperceivesustainability;iftheevidencesupported
itsperceptionbythestudents,thenweplannedtorevisethesurveysoitcouldbemoregeneraland
usedmorewidely.
Prior to distributing the survey, a foundation grant was obtained by two of the authors
(Rochelle E. Tractenberg & Kevin T. FitzGerald) to engage doctoral students to participate in the
courseandtocontinueinarelatedprojectwhereinportfolioswerecreatedbasedontheMR-ER.Atthis
point, we obtained an exemption from our institutional review board, citing our intention to use
any results and research publications to improve the course and our teaching. All data presented
here were collected under this IRB exemption. All participants knew the two faculty who taught
theirclass(co-authorsRochelleE.Tractenberg&KevinT.FitzGerald)andknewoftheirinterestin
thecourse’sfunctioning;nostudentswerefromeitherinstructors’academicdepartments. Noneof
thestudentsknewthethirdco-author(JeffCollmann)whoisaresearchcollaboratorwiththeother
twoco-authorsandwhocompletedallthethematic(content)analysesindependently. Therewasno
interactionbetweenthecoderandthesurveyrespondents.
2.2. TheMR-ERDerivedCourse
TheethicalreasoningKSAswerederivedfromcompendiaofscholarlyworkreflectingethical
decision-making[31]describedindetailin[26],andalsodescribedasthecomponentskillsrequired
for(any)caseanalysisbyAmbroseetal.[9](p. 99). Thecourseanditsassessmentsweredesigned
aroundtheelementsofassessmentvalidityoutlinedbyMessick(1994)[32]:
1. Whatis/aretheknowledge,skills,andabilities(KSAs)thatstudentsshouldpossess(attheend
ofthecurriculum)?
2. Whatactions/behaviorsbythestudentswillrevealtheseKSAs?
3. Whattaskswillelicitthesespecificactionsorbehaviors?
Thesyllabus(seeAppendixB)reflectsourpurposesofprovidinginstruction,andpracticewith
formative feedback, around each KSA, with a final opportunity (highly scaffolded but, ultimately,
summative)toargue,usingworkproductsfromthesemesterasevidence,thattheindividualhadmoved
fromtheBeginnertotheNovicelevel(achangeofonelevelonourfour-levelrubric).Moreover,thecourse
was designed based on the 2012 manuscript the co-instructors (co-authors RochelleE.Tractenberg
&KevinT.FitzGerald)publishedin2012,andthisisrequiredreadingforthecourse.
We specified to the students that the course focus is reasoning, not “ethics” or mastery of the
factualmaterialcomprisingthetopicslistfromtheNationalInstitutesofHealth(NIH,[28]). Wenever
mentioned“sustainability”,“lifelonglearning”,or“transfer”inanysemesterweofferedthecourse,
butwerepeatedlydiscussedandmodeledmetacognition. Oneexampleofthisemphasisisthatwe
discussedeachstudent’scaseanalysisinclassmeetings,focusingontheirdescriptionsoftheirthought
processesandhowtheirwritingdidanddidnotreflecttheirownthinkinginawaythatwouldbe
accessible to any reader of their essays. For the two semester courses that enrolled only doctoral
students, we repeatedly pointed out that, just as important as their developing ethical reasoning
abilities,students’awarenessoftheirownthinkingwouldbeanimportantskillsetforthemtodeploy,
teach,andmodelwhentheyhadstudentsoftheirowninthefuture.
As outlined in the syllabus (Appendix B), the first meetings of the course were devoted to
orientingstudentstothecomponentknowledge,skills,andabilitiesintheMasteryRubricforEthical
Reasoningwhichweretobeutilizedintheircaseanalyseseachweek,utilizingtheNIHtopicslist
Educ.Sci.2017,7,2 6of23
toguideourselectionofcasesfordiscussionandreflection. Duringthefirstsevenweeks,oneweek
had been spent on each of the MR-ER KSAs in turn. We then asked the students to become more
activeintheiridentificationofweaknessesthattheyperceivedintheirKSAperformance. Forthefinal
thirdofthesemester,caseanalyseswerefocusedbyeachstudentonwhicheveroftheMR-ERKSAs
the student felt was most salient for considering, or resolving, the case, or the one for which they
felttheyneededadditionalpracticeorevidenceofgrowth. Eachmeetingandweeklyassignment
focused on one KSA (with all of them being included in each meeting), to be demonstrated and
emphasized, with a brief (20 min at most, in a 3 hour seminar) lecture on that KSA by faculty
(co-authorsRochelleE.Tractenbergand/orKevinT.FitzGerald). Followingthisoverview,students
demonstrated,anddiscussed,themetacognitivefeaturesoftheircaseanalysis,thetargetKSA,andits
reflectionintheirownessays.
The first semester course was offered using the university online system, where all students
(plus one auditor and two instructors, co-authors Rochelle E. Tractenberg & Kevin T. FitzGerald)
calledintoasingleline,onceperweekforathree-hourmeeting. Inthesecondandthirdsemesters,
allmeetings(weeklythree-hourdiscussions)wereheldoncampus,ledbythetwoinstructors(thetwo
auditorseachco-ledonesession).ThesyllabuswassimilartotheoneshowninAppendixB;werevised
the first and second semester syllabi for clarity (not content or structure), based on input that the
previoussemesterstudentsprovided. Thefinal(current)syllabusappearsinAppendixB.Asoutlined
inthesyllabus(seeAppendixB),participantswroteten500-wordessaysforeachmeetingduringthe
semester,andafinal1000-wordessay.
2.3. Subjects
Twelveindividualshavecompletedourcoursetodate,allofwhomhadcompletedeitheratleast
oneonlinemoduleontheresponsibleconductofresearchorasemestercourseinwhichtheseNIH
topicalareaswerediscussed. Atotaloftengraduatestudentsenrolledinthreesemesterofferingsof
thiscourse(threestudentsfromaMaster’sprograminSpring2012,threestudentsfromaPhDprogram
inFall2012;fourPhDstudentsfromthreeprogramsfromour(n=2),andanearby(n=2),university
completedthecourseinSpring2014). Inadditiontothesetenstudents,wehadtwoauditors. Oneof
theMaster’sstudentsinthefirstsemester(whohadcompletedaPhDsome15yearspreviously)also
auditedthesecondsemester,andtwoadditionalparticipantswithPhDs(fromthescientificcommunity)
contactedusrequestingpermissiontoauditthecourse. Oneoftheseindividualsalsoauditedthefirst
twosemesters;theotherauditorparticipatedinourSpring2014semester. Thetwoindividualswho
participatedinbothofthetwo2012semestersspecificallyrequestedpermissiontocontinue,soasto
furtherdeveloptheirreasoningskillsforasecondsemester. Ofthe12participantsinthecoursetodate,
allcompletedeverywrittenassignmentandalthoughtherewasasinglemissedmeetingfor60%of
attendeesineachsemester,nostudentmissedmorethantwomeetings,andsomemissednomeetings.
Thecoursewasrequiredforthestudentsinthefirstsemester(enrolledinamaster’sprogram). Noneof
thestudentswereinthedisciplinesofanyoftheco-authors(respondentswerenot“our”students).
After obtaining an IRB exemption for the project, we emailed former students and requested
representativeexperiences,iftheyhadhadanyoutsideofandafterourcourse,ofeachoftheexamples
ofsustainablelearningdescribedbySchwänke(2008)[14],outlinedintheintroductionandelaborated
inAppendixC(lifelonglearning;changingyourbehaviorasaresultofacquiringnewknowledge;
aprocessofpersonaldevelopmentcontinuingbeyondthecourse;anddeconstruction/reconstruction).
Allstudentswereemailedwitharequesttoparticipateinthesurvey;allstudentswhoresponded
wereemailedthedocumentwiththefourquestions;allrespondentscompletedthesurveyontheirown
andemailedthembacktooneofus(co-authorRochelleE.Tractenberg). De-identifiedsurveyswere
thensentasazippedfileofuniquely-numberedsurveystothequalitatively-trainedanthropologist
on our team (co-author Jeff Collmann), who completed the thematic analysis as described below
(seeAppendixD).Thisanalysiswasindependentlyreviewedbyoneotherco-authorwithexperience
inqualitativeanalysis(co-authorRochelleE.Tractenberg).
Educ.Sci.2017,7,2 7of23
2.4. Analysis
Thenarrativeresponsestotheseopen-endedquestionswerecontent-analyzedbyanindependent
coder(co-authorJeffCollmann)whohadnotbeeninvolvedwiththecoursesandsodidnotknow
therespondents. Thesedatawerethereforeanonymoustothecoder. Eachofthefoursurveyitems
has a yes/no part and a narrative part, so each item can be summarized as the proportion of the
sample that endorsed (said YES to) it. No software was used to analyze these brief responses;
as themes emerged from the narrative responses, they were collated into a table which was then
senttotheindependentreviewerofthethematicanalysis(co-authorRochelleE.Tractenberg)asa
summary—sothisrepresentationofthedatawasanonymoustothatanalystaswell. Thethematic
tableisdescribedinthenextsection. Thepurposeofthisanalysiswastodetermineifsustainability
wasperceptibletothisgroupofstudents.
We then created a Degrees of Freedom Analysis [33–35] matrix with the four features of
sustainability[14]representingour“predictions”,andthecollationofwhetherstudentsperceived
these features representing our “degrees”. The purpose of this analysis was to explore whether
differentdimensionsofsustainabilityweremoreorlessperceptibletostudents.
3. Results
ThesurveyispresentedinAppendixA.Itwasderivedexplicitlyfromthedefinitionofsustainable
learninggivenbySchwänke(2008)[14]. Responsestothesurveywerereceivedwithin6monthsof
ourinitialinquiry(allafterthecourseended)fromnineofthetwelvecoursecompleters,requested
betweentwomonthsandtwoyearsafterthecoursewascompleted. Threestudentsfromourfirst
semesterwerelosttofollowupduetojobchanges(n=2)andmaternityleave(n=1).
Table1showsthecontentanalysisresults.
Table1representstwokeyfeaturesofsurveyresponses: firstly,100%ofrespondentsanswered
YES to three of the four questions; a single “NO” response on one item was negative because the
respondent was unsure whether the attribute was the direct result of our course or not. That is,
allninerespondentsrecognizedeachofthedimensionsofsustainabilitywithrespecttoourcourse;
the majority (8/9) attributed the changes in every dimension to the course and 1/9 endorsed the
dimension,butcouldnotattributeittoourcourse.
Thesecondfeaturearisesfromthethemesinthenarrativeresponsestoallitems: allrespondents
identified work and/or life applications of the course, which is representative of transfer
(andsustainabilityinageneralsense). Threerespondents(allstudents)focusedontheimpactofthe
courseontheircareerplanningwhenansweringthreeofthefourquestions. Mostexamplesreferto
improvementsinreasoningeitherindirectuseoftheKSAprinciplesorthroughroutineexamination
ofalternativesinworkordailylife. Theexamplesasawholesuggestbehavioralaswellascognitive
changes. Respondentsdescribethemselvesasactingdifferentlyintheworldasaresultofchangesin
theirreasoningandproblem-solvingapproaches. Thesurveypromptedthemtoidentifychangesin
theirbehaviorthattheyattributetothelearninginthecourse,aprocessthatre-enforcedthecourse’s
fundamentalemphasisonself-reflectionandmetacognition.
All of our respondents work in research contexts, so responses relating to “research” may be
attributabletoeitherworkortotheircareer. Forexample,fiveofeightrespondentsidentified“work”
(or“workplace”)applicationsoflifelonglearning(e.g.,“learnedhowtostructuremythoughtsand
reflectuponthem=>improvedproposalwriting”(Student#8)and“worktounderstandresearch
workatadeeperlevel”(Student#2)). Threeoftheeightspecificallyidentified“dailylife”applications
(“‘lifelonglearning’... almostoccursonadailybasis,andmostlikelywouldnothaveoccurredat
allifitwasnotformyinvolvementwiththeethicalreasoningcourse”(Student#4)“(L)isteningto
alternateperspectivesandthenmakingaconcretedecision”(Student#6)). Onerespondentendorsed
theitembutprovidednotextresponse. Allrespondentsendorsedthelifelonglearningdimension
ofsustainability.
Educ.Sci.2017,7,2 8of23
Table1.ResultsoftheContentAnalysisofSustainabilitySurveyresponses.
SustainabilityDimension
LifelongLearning ChangeLearning PersonalDevelopment De/Reconstruction
MajorThemes Improved Improved Enhanced Focusedon
Work Career DailyLife Work Career Reasoning
fromResponses Reasoning Expression Reflection Career
Survey
1 1 1 1 1
2 1 1 1 1
3 1 1 1 1
4 1 1 1 1
5 1 1 1 1 1
6 1 1 1 1
7 * 1 1 1
8 1 1 1 1
9 1 1 1 **
Total 5 1 2 7 2 7 3 3 2 3
Notes:*Notextresponse,buttheitemwasendorsed(part1ofeachitem);**Respondentunsurethatthiscourseledtode/reconstruction,sogavenotextresponsebutdidendorseitem.
Educ.Sci.2017,7,2 9of23
All nine respondents endorsed the learningbehaviorchange dimension of sustainability,
and responses again represented work (e.g., “be sure I understand more than “going through the
motions” when acquiring these techniques and am seeking the details/theories/physics behind
the techniques” (Student #9); “generate alternative actions in solving problems, including science
problems”(Student#7))anddailylife(e.g.,“moreproductive,duringconflict,toseekunderstandingof
thevariousconflictsandfindcommonground”(Student#8);“improvedorganizationofmythoughts”
(Student#2))—includingotherlearningexperiences(“increasedinterestinlearningasa“messy”or
socialconstructivistexperience”(Student#1);“Mylearningbehaviorhasbeencompletelyrenovated
... WhenIhearsomethingnewIrunitthroughtheKSAs”(Student#4)).
Whendescribinganongoingprocessofpersonaldevelopment,notonlydidallninerespondents
endorsethisdimension,buttheyalsoprovidedexamplesthatdifferedfromtheirearlierresponses.
Theexamplesagainrelatedtocareerandwork(e.g.,“(t)orealizethatcareerdevelopmentiscrucial”
(Student#2); “endedupattendingthe(leadershipworkshop)andfindingittobebothreinforcing
and broadening in my understanding of leadership qualities I possess” (Student #9)), or to daily
lifeincludingnewlearningexperiences(e.g.,“Iseehowimportantalternativeperspectivesare,and
justhowdifferentlyeveryonehastheabilitytoseeonesingleperson,place,orthing”(Student#4);
“inanewlearningenvironmentwhereIhavetothinkcriticallyaboutsubjectareasthatarenewtome”
(Student#6);“Enhancedperspectiveandcognitiveawareness”(Student#7)).
Finally, while all nine respondents endorsed having had deconstruction/reconstruction
experiences outside of, and after, our course, one respondent was uncertain whether or not these
wereduetothecourseitself(“Idonotknowifmyrealizationoftheseerrorsinreasoningwasdueto
thiscourse”(Student#9))—whichisitselfahighlyreflectiveandmetacognitiveresponse—andthese
weretwokeyelementsofthecoursewhilethedeconstructionandreconstructionaspectswerenot.
Theotherresponseswereagainreflectiveofdailylifeinthreerespondents(“myunderstandingof
other’sperspectives... Bythoroughlydissectinganother’sperspectiveandtrulytryingtowalkinits
shoes,Iamlearningmuchmoreaboutmyself,inadditiontoothers,thanIcouldhaveneverimagined”
(Student #4); “I thought I understood the process of expressing myself and of reflection. I now
write daily and reflect on it” (Student #8); and, “I now know how to recognize moral dilemmas,
understand the internal conflicts that created them and construct an approach that is ethically
congruent” (Student #7)). The other four responses were more focused on work (e.g., “Securing
fundingrequiresalotofdifferentmovingpartsandmybasicunderstandingofthisentireprocess
hasgivenagreaterappreciationformydiscipline”(Student#6);“Nowseethebenefitofcontinuing
scientificeducation,especiallyfromexpertsatconferences. Increasedunderstandingofhowscience
affectspolicyandtheimportanceoffundingtoscience”(Student#2)).
Table2presentsthesummaryofourdataalignedwiththepredictionsderivedfromthedefinition
ofsustainabilitythatweused[14]inaDegreesofFreedomAnalysismatrix[33–35].Giventheinclusion
ofoneendorsementpartofthesurveyitemsforeachelementofsustainability,thethreeco-authors’
consensusontheevidence(whetherrespondentsdidperceivethatelement)wasunambiguousand
consensuswasreachedimmediately.
As was noted in the presentation of the content analysis of the narrative responses,
everyrespondentindicatedthatyes,theyperceivedeachofthefourelementsofsustainablelearning.
One respondent did perceive the first element (lifelong learning) but did not give an example;
onerespondentdidperceivethelastelement(deconstruction/reconstruction),butinthenarrative
responsesaidtheywerenotcertainthatthiswasduetoourcourse. Becauseweweremostinterested
injustwhetherstudentscouldperceivethesustainabilitydimensions,weconsideredtheendorsement
oftheelementsofsustainabilitytobesupportiveoftheconclusionthatthesestudentsfromthiscourse
didinfactperceiveallfouraspectsofsustainability.
Educ.Sci.2017,7,2 10of23
Table2.DegreesofFreedomAnalysis:dostudentsperceivethefoursustainabilityelements?
Respondent
Sustainability
1 2 3 4 5 6 7 8 9 Total
Element
Lifelonglearning 1* 1 1 1 1 1 1 1 1 9
Changelearning 1 1 1 1 1 1 1 1 1 9
Personaldevelopment 1 1 1 1 1** 1 1 1 1 9
De/reconstruction 1 1 1 1 1 1 1 1 1*** 9
Totals 4 4 4 4 4 4 4 4 4
Notes: *Theitemwasendorsed,butnonarrativeresponsewasincluded;**twodifferentthemesemerged
fromthissinglenarrativeresponse,theonlycasethishappenedacrossallnarrativeresponses;***Theitemwas
endorsed,buttherespondentwasnotsurethatthiscoursewasthecauseoftheperceivedsustainabilityelement.
4. Discussion
Wesurveyednineofour12completers(todate)onfourfeaturesofsustainablelearningandthese
preliminaryresultsindicatethatallcompletersbutoneidentifiedatleastoneexperiencemeetingall
fourcriteria;mostdescribedoneormoreexperiencesrepresentingeachcriterion. Theserespondentscome
fromdoctoralprogramsindifferentdisciplines,andincludetworespondentswhohadalreadycompleteda
PhDpriortoourcourse. Theonerespondentwhodidnotendorseoneofthesustainabilitydimensions
didrecognizeit,butwasunabletodetermineifitwasadirectresultofourcourse.Thefourdimensional
model of sustainability outlined in 2008 [14] did not create a “neat” survey—in fact, the items are
fairlycomplexascurrentlywritten(seeAppendixA),andareimpossibletoadministertoa“control”
groupbecausethelanguageissospecifictothecoursewedeveloped. However,allninerespondents
were able to recognize each of the four dimensions and provide examples from their lives of how
theirlearninginacoursewassustained. Thesedatasuggestthatsustainablelearningisobservableto
students;andtheirnarrativeresponsesrenderitobservabletoothers. Thus,sustainabilityoflearning
maybeaplausibleadditiontoexistingobjectivesforthedocumentationandassessmentoflearning
outcomes[13]. Moreover,ifthesepreliminaryresultsshowingtheperceptibilityofsustainabilitycan
bereplicatedwithawider,morerepresentativesampleofstudents,thensustainablelearningcouldbe
consideredtobe“assessable”,andstudentperceptionofitcouldbeconsidered“actionableevidenceof
studentlearning”[36].
Althoughsustainablelearningmightbeafocusintheeducationofeducators(see,e.g.,[2–7,37]),
enduringeffectsof“RCRtraining”haveneverbeenshown—nortoourknowledgehasanyempirical
evidenceofsustainabilityofhighereducationbeenpublishedtodate. Morethanshowingthatour
studentscontinuedtolearnethicscontentaftertheir“requirement”wassatisfied—whichtechnically
fitsthe“lifelonglearning”model(andmeetsboththespiritandtheletteroftheNationalInstitutesof
HealthrequirementsfortraininginRCR,[28])—theresponsesofourstudentsconcretelyrepresent
their application of ethics content knowledge, reasoning, and metacognition independent of and
beyondourcourse.
Continuing professional development is well-known to be difficult to foster, monitor,
anddocumentinmedicalandnursingeducation[10,11],andinthosecasesitismandated;inresearch
ethicsthecommunityvaluecontinuestoemphasizearequirement-satisfactionmodel,ratherthana
developmentalmodelorevenactionablelearningoutcomesassessment[13]. TheNationalInstitutes
ofHealth[28]requirethat“RCRtraining”spansthescientists’careerandNovossiolovaandSture
(2012)[20](pp. 80–81)recommended“continuingprofessionaldevelopment”inethicsforscientists,
and they call for research on how this might be achieved and assessed. Our work suggests that
incorporatingmetacognitionmightpromotetheresponsibleconductofresearchthroughoutacareer,
possiblyasamodelforactually-continuing“professionaldevelopment”. Moreover,thispreliminary
evidence suggests that it may be possible to obtain actionable evidence of learning outcomes that