A Systematic Literature Review on Intertemporal Choice in Software Engineering - Protocol and Results

2.5 MB·

by  Christoph Becker

1
A Systematic Literature Review on
Intertemporal Choice in Software Engineering –
Protocol and Results
Christoph Becker, Dawn Walker, and Curtis McCord
Abstract—Whenmakingchoicesinsoftwareprojects,engineers Research in psychology and behavioral economics calls
and other stakeholders engage in decision making that involves choices that involve trade-offs across time “intertemporal
7 uncertain future outcomes. Research in psychology, behavioral
choices,” defining them as “decisions involving tradeoffs
1 economicsandneurosciencehasquestionedmanyoftheclassical
0 assumptions of how such decisions are made. among costs and benefits occurring at different times” [5].
2 This literature review aims to characterize the assumptions Researchers have developed a number of theories of such
thatunderpin thestudyof thesedecisionsin SoftwareEngineer- choices [5], [6] and have demonstrated that straightforward
n
a ing. We identify empirical research on this subject and analyze assumptionsabouthowdecisionmakersevaluateanddiscount
J how the role of time has been characterized in the study of the future are often misguided and wrong [5], [6].
decision making in SE.
8 Herein,an important distinction is made between normative
The literature review aims to support the development of
2 descriptive frameworks for empirical studies of intertemporal and descriptive decision theories. Normative theories focus

decision making in practice. on the identification of the best decision, and model an
]
E ideal decision maker. Normative models of how decisions
Index Terms—Software Engineering, Behavioral Software En-
S gineering, Intertemporal Choice, Technical Debt, Sustainability are made in SE commonly assume a rational agent (with
. Debt, Trade-off decisions, Decision Theory, Sustainability reasonable cognitive boundaries) choosing between a set of
s
c options according to a value function.
[ From choosing a software development methodology to
I. INTRODUCTION
1 evaluating release planning, prioritizing requirements and
COMPLEX software-intensive systems play critical roles
v choosing between architectural design options, SE literature
0 in our societies: their ongoing development, innovation, commonly assumes that decision making operates in a pre-
1 and maintenance is intertwined with our everyday social and
dictable, rational way. For example, one author writes “In
3 economic activities. As recognition of the key role software
most problems, to make a decision, a situation is assessed
8
technologycanplayinsociety’ssustainabilitygrows,theneed
0 againstasetofcharacteristicsorattributes,alsocalledcriteria.
for a paradigm shift in the mindset of the software industry
. Decision making based on various criteria is supported by
1 has become clear. Sustainability is often defined within the
multi-criteria methodologies” [B1]. The assumption is that
0
domainof“sustainabledevelopment”,which“meetstheneeds
7 a team of competent engineers evaluates the options to the
of the present generation without compromising the ability of
1 best of their knowledge, and they choose the option with the
: future generations to meet their own needs” [1]. At its core, highest expected value. Much of their discussion in theory
v
sustainability is the capacity to endure, but sustainability of
i and practice focuses on how to best estimate that value. The
X social systems is different than technical or natural systems.
frameworksofValueBasedSoftwareEngineeringaimtobase
r Originally equated with environmental concerns, it is now SEdecisionsmoreexplicitlyonanunderstandingofvalue[7].
a
clear that sustainability requires equal consideration of five
Most commonly, this value is expressed in economic terms,
dimensions:environmental,societal,individual,economic,and
and the incommensurability of multiple aspects of value is
technical [2].
often addressed through application of utility functions [8].
Withinandacrosstheseconcerns,softwareengineering(SE)
The theory of expected utility stems from game theory [9]
decisions are made about system scope, goals and objectives,
and was developed from principles, not empirical study. By
features,functions,architecturaldesigns,andmanyotherareas
contrast, descriptive theories aim to characterize the behavior
throughout the development lifecycle. The effects of these
of actual decision making. As Tversky and Kahneman write,
choices are often delayed, and many critical decisions involve
“The modern theory of decision making under risk emerged
trade-offsbetweenoutcomesatdifferentpointsintime.Insuch
from a logical analysis of games of chance rather than from
cases, longer-term consequences are not always sufficiently
a psychological analysis of risk and value. The theory was
considered [3], [4].
conceived as a normative model of an idealized decision
maker, not as a description of the behavior of real people...
C. Becker is lead of the Digital Curation Institute and with the Faculty
of Information, University of Toronto, Toronto, ON, Canada. E-mail: (see the logic of choice does not provide an adequate foundation
https://ischool.utoronto.ca/faculty/christoph-becker). for a descriptive theory of decision making” [10].
D.WalkerandC.McCordaredoctoralstudentsattheFacultyofInforma-
Criticism of the prevailing normative decision theories has
tion,UniversityofToronto,Toronto,ON,Canada
SubmissiononJanuary27,2017. come from numerous perspectives, and various alternative
2
conceptions have been proposed. For example, well-known
Fig.1. OverlapbetweenSEandIntertemporalChoiceSCOPUSquery
experimentshaveshownthatpeopledonotdiscountthefuture
linearly [5] and that risk aversion is higher for gains than
for losses [10]. More substantively, Tversky and Kahneman
showed that some of the foundational axioms of normative
decision theory, and in particular expected utility theory, are
inconsistent with observable behavior. More radically, Klein’s
study of expert decision making showed convincingly that
experienced decision makers do not actually weigh a set of
alternatives against criteria to maximize expected utility when
making critical choices [11]. It is this divergence between
prevalent normative models and observed behavior that mo-
tivated this review. This corresponds to the recent emergence
of Behavioral Software Engineering, a field that aims to
draw in behavioural frameworks and concepts for a better TABLEII
understanding of software engineering [12], [13], [14]. OVERVIEWOFSCOPUSSEARCHRESULTSFORINTERTEMPORALCHOICE
PRELIMINARYSEARCH
InSE,choicesthatareexpedientintheshort-termbutcreate
unwantedlonger-termconsequenceshavebeenconceptualized Totalnumberofsearchresults 0
most prominently as ‘technical debt’, which focuses on en-
gineering choices that create hidden costs. The metaphor of
debt aims to make these hidden costs visible and manageable. RQ2 Which dimensions are considered in these studies?
Interpreted more broadly, the notion of ‘sustainability debt’ RQ3 How has the role of time been conceptualized in these
expands the metaphor to direct and indirect effects across all studies?
dimensions of sustainability [15]. RQ4 Which assumptions on decision making underpin these
studies?
While we are interested in the assumptions on decision
A. Objective
making that underpin the perspective of the non-empirical
This review is motivated by the need to better understand
studies,wefocusourin-depthanalysisonempiricalworkdue
how and why software practitioners incur sustainability debt
to time restrictions.
in practice. In order to develop a descriptive framework
for intertemporal choices in SE, we review the literature to
B. Roles and Responsibilities
identify whether the intersection of these concepts has been
acknowledgedandaddressed;describewhichperspectivesand The roles and responsibilities for this project are defined in
assumptionsaboutdecisionmakersunderpinexistingresearch; Table I. We have one principal researcher, Christoph Becker,
andanalyzehowtheroleoftimehasbeencharacterizedinthe and two supporting researchers, Curtis McCord and Dawn
study of decision making in SE. Walker.ExternalreviewswereconductedbyStefanieBetzand
Because of our interest in distinguishing normative models Ruzanna Chitchyan.
theorizing about decision-making in SE from descriptive, em-
piricalaccountsofhowtrade-offdecisionsrelatingtotimeare C. Search Strategy
made in software design projects, we will first map empirical
In order to produce a systematic overview of this area, the
and other types of research of decision making, and then
overall search process for this literature review is based on
analyze empirical research in detail in order to understand
guidelines established by Kitchenham [16].
the assumptions of decision making models that underpin this
1) InformationSources: Weperformedautomatedsearches
research.
on the following indexing systems and digital libraries: Sco-
pus, IEEE Xplore, and ACM Digital Library.
B. Contribution 2) Preliminarysearch: Theterm‘intertemporalchoice’has
Weaimtorevealhowtrade-offchoiceshavebeenconceptu- come to describe precisely our area of interest. At an early
alized within SE so far, identify gaps in how decision making stage, we conducted searches to identify whether there has
is reviewed and investigated, and map how SE literature been explicit attention to this concept in the literature.
approaches making trade-offs over time. “intertemporal choice”
AND “software engineering”
Fig. 1 and Table II show the resulting search numbers
II. LITERATUREREVIEWSTUDYDESIGN
for SCOPUS. While the exact numbers differ for the other
A. Research Questions
databases, the trend is mirrored and the intersection remained
WecharacterizeperspectivesondecisionmakingwithinSE empty for all searches.
research through the following questions: The search revealed that intertemporal choice is not ex-
RQ1 Which empirical research in SE has studied trade-off plicitly treated in the literature at all, and the phrasing was
decisions involving time? not present in any papers. This does not necessarily indicate
3
TABLEI
BREAKDOWNOFROLESANDRESPONSIBILITIES
ChristophBecker CurtisMcCord DawnWalker ExternalReviewers(Betz,Chitchyan)
DevelopProtocol X X X
PrototypeProtocol X
DefineSearchStrings X X
DefineClassificationScheme X X
ReviewofProtocol X X
FinalRevisionofProtocol X X
IdentifyPrimaryResearch X X
RetrievePrimaryResearch X X ‘
De-duplicate X
PrototypeRelevancyVoting X X X
RelevancyVoting X X
ReviewofRelevancyVote X
Dataextraction,Classification,andSynthesis X X
AnalysisValidation X
WriteTechnicalReport X X X
ReviewofTechnicalReport X X
that SE does not deal with intertemporal choices, but the frommultipledisciplineswouldincreasetheamountofpapers
absenceofexplicitmentionoftheterm“intertemporalchoice” captured, but not necessarily make the literature review more
suggests that the concepts arising from the field of behavioral effective or representative. This, and the possible bias intro-
economics have not been congruently linked to SE, i.e. that ducedthroughthesemorecomplexqueries,ledustochoosea
no direct conceptual mapping has been established between simplified more generic query string and move some of the
the two disciplines yet. detailed aspects of intertemporal choice to the coding and
3) Preliminary concept review: Before conducting further analysis stages.
searches, we aimed to establish a candidate set of concepts The same reasoning process governed our decisions on the
that would scaffold our understanding of decision making second clause of our query; we were interested in papers
vocabulary. To do so, we reviewed textbooks and standards in discussing trade-offs, but recognized that while the term is
Software Engineering [17], Value Based Software Engineer- widely used, it might not be used by all authors describing
ing [7], Decision Analysis, Behavioural Economics [18] and these types of decisions. What we really wanted to capture
Management Theory [19] to compose a working vocabulary through coding was choices that required parties to weigh
of terms related to intertemporal choice. From these texts decision dimensions against each other.
we developed a series of prototypical concept maps that Search queries were piloted twice (See Appendix A for
decomposedkeycomponentsofdecisionmakingintopotential pretest queries) prior to establishing the final search string:
search terms. Terms such as “cost”, “value”, “benefit”, “risk”, time
“decision-making”,for example,were widelyused acrossdis- AND “decision making”
ciplines,andhelpedtostructureourunderstandingofdecisions AND “software engineering”
and provide terminology for coding and analysis later on. 5) Ancillary Search: Using the same search strategy, one
4) Search String: The goal for the search string was to ancillarysearchwasperformedaspartoftheliteraturereview:
capture results that dealt with intertemporal decision-making “technical debt”
inSE,toexaminehowSEprojectssawtimeasafactorintheir The concept of Technical Debt (TD) is prominent in soft-
decision processes, how they make decisions about the future ware engineering and closely related to the dimensions of
oftheirprojects,andhowtheymightweighfutureandpresent our main query. Technical debt can be defined as: “a design
goods against each other. We included the clause “software or construction approach that is expedient in the short term
engineering” to limit the disciplinary scope of our research– but that creates a technical context in which the same work
other disciplinary scopes such as “requirements engineering” will cost more to do later than it would cost to do now
could conceivably lead to different perspectives. (including increased cost over time)” (Ernst [20], borrowing
To capture the temporal aspect of decision making, we from McConnell). In this framing, TD always includes an
settled on the general term of “time”, with the intention of explicitly temporal dimension, built into the concept of debt.
usingmorespecificcodingduringanalysis.Preliminarysearch Decisions that are made about TD would presumably include
queries (See Appendix A-B) were more complex and used a temporal dimension and the commensuration of future
more discipline-specific jargon (“life cycle”, “endurance”) and present goods. As such, the literature on TD could be
whose specificity would occlude relevant papers that could complementarytootherareasofintertemporalchoiceandshed
be captured by a more general query. light on specific assumptions.
While these searches included relevant results that con- Theresultsoftheancillaryqueryweredocumented,butthe
nected to the concepts that emerged from initial review, the only analysis performed within this review was an identifi-
results were mixed and widely spread across disciplines. It cation of the overlap with the primary search, as described
became clear that introducing divergent and specific terms further below (see Section III). The resulting corpus of publi-
4
cations will be used for further analysis in the future. F. Analysis
Papers from the relevancy review were analyzed in order to
D. Selection Criteria address the research questions established above. Application
ofachecklistaswellasreviewensuredthequalityofanalysis
1) InclusionCriteria: Weestablishedthefollowingcriteria
andcoding.Researchersextracteddatausingaformtocapture
to identify relevant publications that would answer research
fields relevant to our research questions.
questions:
1) Assessment: In order to ensure quality of analysis
• Publication Year: All years were included.
and findings, researchers conducted multiple internal reviews
• Publication Type: We included peer-reviewed papers
throughout many stages of the Literature Review: protocol,
publishedinjournals,conferenceproceedings,andwork-
relevancyvotingprocedure,relevancyvotingresults,technical
shop proceedings.
report. Additional external review of the technical report led
• Content: The paper had to contain a discussion of
to suggestions for improvement.
decision-making in software engineering projects.
2) Data Extraction: Researchers classified studies accord-
• Coverage: The paper had to cover development of a
ing to the type or domain of the decision-making studied,
software system rather than only hardware.
the methods of investigation and research, whether there
2) Exclusion Criteria:
was a trade-off decision, and if so, the dimensions of the
• PublicationLanguage:Weexcludedpapersinlanguages trade-off. Free annotation was also used to capture additional
other than English. information the coders deemed relevant.
• Publication Quality: We excluded papers retracted by A form was also designed to capture these fields as well
the publisher.
as metadata from the studies, including author, title, year of
• Publication Type: We excluded non-paper results in- publication, DOI, and unique document key (generated from
cluding:posters,abstract-onlysubmissions,bookreviews,
author, title, publication).
books, entire volumes of proceedings, panels, presenta- Coding Field 1: Scope of the Decision
tions, tutorials, opinion pieces.
(pm) Project Management [17]
• Technical: We excluded papers where the PDF was
(dev) SoftwareRequirements,Design,Architecture,Devel-
unavailable (behind a paywall or not locatable).
opment [17]
(mait) Software Maintenance
E. Selection Procedures (other)Including business and business strategy
After downloading, removing duplicates, and applying our WithinSE,requirementsdecisions,design,architecture,and
exclusion criteria, the remaining papers were screened for development cover a wide range of tasks, fields, etc. We
relevancy using the following procedure: intentionally grouped these together to cover all engineering
1) Thesecondaryresearchersvotedonrelevance:Theyread decisions as one, in part because these decisions often span
identifiedpapertitlesandabstractsinordertodecideon multiple areas.
inclusion using the criteria above. A yes or no decision Coding Field 2: Research Methodology
(“Y/N”) was assigned as well as a certainty value from (emp) Empirical methods were used and the object of
1-3 (where 3=certain). empirical study was a decision
2) Voters reviewed 10 of the 307 papers as a pilot, and (emp comp)
then conducted a larger pilot of 49 papers including the Empirical methods were used and the object of
original 10, and discussed the results together. empirical study was NOT a decision
3) Following this quality assurance step, the remaining (lit) The paper was (exclusively) a literature review or a
258 papers were split and reviewed by one voter each, systematic mapping study
following the voting process established above. (other)Theresearchwasnotempirical,i.e.itwastheoretical
4) In cases where papers were reviewed by more than one or attempted to develop a model
voter, disagreements were resolved through discussion
CodingField3:Doesthearticlediscusstrade-offdecisions?
and consensus.
(Y) Yes
5) All decisions on papers reviewed by only one voter
(N) No
were compiled. Those with a certainty value below 2
Coding Field 4: Dimensions of Trade-Off
(191) were reviewed and discussed by both voters, and
a randomly selected sample of 65 was evaluated for (cost) Cost (Often in monetary terms)
consistency.Incaseofremainingdoubt,thepaperswere (func) Functionality
included. (mait) Software Maintainability
6) 155 papers marked for further coding were looked (qual) Software Quality
over by the an internal reviewer to determine whether (risk) Risk
inclusion and exclusion criteria were appropriate before (time) Time (Includes scheduling, delivery and release)
analysis. Because of the rule to include papers in case (value)Value (As in terms of monetary value, “business
ofdoubts,thefocuswasonverifyingincludedpapersat value” or in some cases, in terms of benefit)
this stage. (other)See Appendix
5
Fig.2. OverlapbetweenmainqueryandTDpapers Fig.3. Segmentsdistinguishedaccordingtoresearchtypeandfocus
TABLEIII
OVERVIEWOFSEARCHRESULTSFORMAINSEARCH
TotalNumberofSearchResults 889
Total number of results after duplicate removal and 652
exclusioncriteriaapplied
Numberselectedafterpreliminaryrelevancyreview 307
Numberselectedaftervoting 155
the object of decision-making. For these reasons we were
3) Analysis of Extracted Data: The secondary researchers not surprised to find several recent papers that focus on
extracted data and analyzed the results included below. From decisions about monitoring, reporting and managing technical
this they synthesized findings on the current research. Feed- debt. Before relevancy voting, 7 papers were in both the
back was provided through an internal review by the prin- technical debt and main query corpus. After relevancy voting,
cipal researcher. In order to analyze the extracted data, the theintersectionofthetwocorpuseswas2papers,asillustrated
researchers: in Fig. 2:
• derived statistics of coded categories for mapping ex- • Martini and Bosch, 2016, An Empirically Developed
tracted data MethodtoAidDecisionsonArchitecturalTechnicalDebt
• mapped out areas of existing work Refactoring: AnaConDebt [A3]
• created visualizations with groups of dimensions • Oliveira, Goldman, and Santos, 2015, Managing Techni-
cal Debt in Software Projects Using Scrum: An Action
III. RESULTS Research [A11]
Search result statistics are provided in Table III. First the
search results from the indexing systems and digital libraries IV. FINDINGS
were compiled, then results were de-duplicated and exclusion
Toexploreassumptionsthatunderpintheexistingempirical
criteriaapplied.Fromthose652papers,andinitialassessment
work on trade-off decisions in SE, we first extracted statistics
to determine whether they were relevant led to 307 papers
from the coded categories of each paper in order to map
selected. Based on voting and discussion to reach consensus,
extracted data. Subsequently, areas of existing empirical work
that number was reduced to 155 for final coding.
were further identified, visualized, and data on the groups
Statistics of the ancillary “Technical Debt” query are sum-
of dimensions was collated. From this, final analysis was
marized in Table IV. As expected, there was some overlap
performed in depth on the subset of empirical papers that
between the papers returned in our Technical Debt query
discussed decision making in SE. We will discuss the main
and those returned in our main query. As technical debt
research questions in separate sections below.
has become a more prominent term in software engineering
discourse, it also becomes a phenomenon which can be ana-
lyzed and accounted for. In this way it becomes manageable: A. Which empirical research in SE has studied trade-off
decisions involving time?
Asdescribedabove,therelevantpublicationswerecodedfor
TABLEIV
OVERVIEWOFSEARCHRESULTSFORTECHNICALDEBTANCILLARY an empirical focus on studying decisions, and for including in
SEARCH particular decisions involving ‘trade-offs’.
We found 93/155 papers had some degree of empirical
Totalnumberofsearchresults 620
Total number of results after duplicate removal and 246 component,and88/155discussedtrade-offsinsuchcapacities.
exclusioncriteriaapplied The resulting Venn diagram shown in Fig. 3 shows a quite
6
TABLEV
ARRANGEMENTOFSELECTEDPAPERSBYTYPE(SEEAPPENDIXC)
Segment Count Description
[A1-13] 13 ResearchhasanEmpiricalComponentwhichStudiesTrade-OffsorLikeDecisions
[B1-46] 46 ResearchhasanEmpiricalComponentandDiscussesTrade-OffsorLikeDecisions
[C1-34] 34 ResearchhasanEmpiricalComponentandDoesNotDiscussTrade-OffsorLikeDecisions
[D1-29] 29 ResearchDoesNothaveanEmpiricalComponentandDiscussesTrade-OffsorLikeDecisions
[E1-33] 33 ResearchDoesNothaveEmpiricalComponentandDoesNotDiscussTrade-OffsorLikeDecisions
TABLEVII
Fig.4. Numberofstudieswithntrade-offdimensions
COUNTOFTRADE-OFFDIMENSIONSINALLSTUDIES
cost 39
time 38
other 38
quality 33
functionality 16
risk 15
value 13
maintenance 5
TABLEVIII
“OTHER”TRADE-OFFDIMENSIONS
benefit*
competition
complexity
labour*
methodology*
even distribution across the emerging subsegments, but indi-
opportunities
cates that only 13 studies were identified that explicitly used returnoninvestment
empiricalmethodstostudytrade-offdecisionswheretimewas goals*
technicaldebt
a relevant element. This set represents papers that attempt to
vendor
examine decision-making in software engineering in real or other
experimental situations. *thosewithanasteriskappearedmorethan5times
TableVIsummarizeskeycharacteristicsofthe13identified
papers, including the research method(s) and citation counts.
C. How has the role of time been conceptualized in these
The most prominent method is case study research.
studies?
We will focus later on this set of 13 papers in detail.
Time is the most popular dimension across all papers, 38
B. Which dimensions are considered in these studies? papers with time as a dimension; this is unsurprising consid-
ering the search term. Within this, time is addressed in the
Of the 155 papers coded, 88 identified at least one di-
various mapping groups: empirical (7), empirical component
mension of trade-off. The majority of those, 54, are choices
(19), non-empirical (12), and literature review (0).
within one or two dimensions, for example within differing
However, the role of time is of course not always intertem-
stakeholder goals or between costs and time.
poral. Time surfaces
The most discussed single trade-off dimension is cost (39),
thenexthighestmentionistime(38),thenquality(33),which • As the object of effort estimation: How much time will
includes nonfunctional requirements discussed as a group each of these options take?
or specifically as “usability,” “security,” etc... The “other” • As a factor in project management: How much time is
category(38)hadadiversityofdimensionsseeninTableVIII, available to the team?
those that occurred more than 5 times are indicated with an
asterisk.Inafewcaseswecategorizedtermsasthesamewhen
the language had some variation (e.g. value, business value, TABLEIX
andbusinessbenefitsinto“value”).Arecordofthesedecisions COUNTOFTRADE-OFFDIMENSIONSIN13EMPIRICALSTUDIES
was not recorded. cost 10
Of the 13 papers that empirically discussed a trade-off time 7
quality 6
decision, the emphasis within dimensions was on cost (10),
value 4
then time (7). Table IX shows the number of papers in which other(benefit) 3
each dimension occurs within this set. other(goals) 3
functionality 1
Further analysis of these dimensions could identify which
other 1
sets of dimensions frequently co-occur. The data set has been other(competition) 1
prepared to support this analysis. other(methodology) 1
7
TABLEVI
EMPIRICALSTUDIES(CHRONOLOGICAL)
Title Year Author(s) ResearchMethodandSummary Citation
Count(GS)
Acost-valueapproachforpriori- 1997 J. Karlsson and K. The authors developed a costvalue approach for prioritizing 715
tizingrequirements[A10] Ryan requirements and applied it to two commercial projects (case
studyevaluation).
Evaluating the cost of software 1998 S.A.Slaughter,D.E. Thepaperanalyzeslarge-scaledataaboutsoftwarequalityand 244
quality[A12] Harter,andM.S.Kr- costs collected empirically from software organizations across
ishnan time.Noindividualprojectsordecisionsarestudieddirectly.
Theimpactofgoalsonsoftware 1999 T. K. Abdel-Hamid, Anexperimentwasperformedwithaprojectsimulationgame 148
project management: An experi- K. Sengupta, and C. played in teams. Two control groups played the same game
mentalinvestigation[A6] Swett but with different goals– one focused on minimizing cost and
schedule, one delivering highest quality in minimal schedule.
Thefocuswasnotonthesedimensionhowever,butontherole
ofgoalsettinginperformance.
Measuring the ROI of software 2004 R.VanSolingen Twocasesof(real)projectsaredescribedaspartofanargument 120
processimprovement[A13] abouttheneedtoestimatevaluesothatcostandvaluecanbe
usedtoestimateROIofprocessimprovement.
A quality-driven systematic 2005 T. Al-Naeem, I. Gor- Acasestudyisconductedaspartoftheevaluation.Interviews 105
approach for architecting ton, M. A. Babar, F. with the architect of a real system are conducted to identify
distributed software applications Rabhi, and B. Bena- decisions. An approach is proposed and its applicability is
[A7] tallah discussed using the scenario of that real system. The architect
wasaskedforfeedback.
Whatisimportantwhendeciding 2005 C.WohlinandA.Au- Aquestionnairewasusedtoidentifytypesofcriteriathatare 57
toincludeasoftwarerequirement rum mostimportantforrequirementsprioritizationbasedonindustry
inaprojectorrelease?[A5] responses.
Aqualitativeempiricalevaluation 2005 C. Zannier and F. The paper proposes qualitative empirical research, but the re- 19
ofdesigndecisions[A1] Maurer search is not completed at that time. (A later paper reports on
this,withupdatedtheoreticalframeworks.)
Choosing the right prioritisation 2008 S.Hatton An experiment with students was conducted to study require- 31
method[A9] mentsprioritizationmethods.
Key aspects of software release 2008 M. Lindgren, R. Case study research is performed across multiple cases (orga- 19
planninginindustry[A2] Land, C. Norstrom, nizations)toidentifykeyaspectsofreleaseplanningincluding
andA.Wall severalaspectsinvolvingtime.
Howdorealoptionsconceptsfit 2010 Z. Racheva and M. Ahybrid research designcombinesascopingreview[6],the 3
in agile requirements engineer- Daneva CHAPLframework[7],andacasestudy[8].Inanotherplace,
ing?[A4] itisdescribedasacross-casestudyineightorganizations,and
itincludes11interviews.
Software for scientists facing 2015 J. B. Cushing, K. M. Casestudyresearchisconductedonacomplexsoftwareproject. 1
wicked problems lessons from Winters,andD.Lach The results include conclusions about the opportunities and
theVISTASProject[A8] challenges of embracing the complexities of wicked problems
insuchmulti-stakeholderenvironments.
Managing technical debt in soft- 2015 F. Oliveira, A. Gold- Action Research: A technical debt management framework is 2
ware projects using scrum: An man,andV.Santos evaluatedintherealcontextofsoftwareprojects.
actionresearch[A11]
An empirically developed 2016 A. Martini and J. DesignScienceResearchevaluatedinmultiplecases.Amethod 0
method to aid decisions on Bosch isdevelopediterativelywithindustrypartnersandthenevaluated
architectural Technical Debt inaseparatelycompletedcase.
Refactoring:AnaConDebt[A3]
• Asanattributeofdecisionmaking-howmuchtimedoes researcher)inaproject,andoftenameasureofthattime,
it take to make a choice? • an axis of discrete units ‘of time’ over which a sequence
• As a factor in scheduling, finally, the closest to intertem- of events take place,
poral choice: Should we release now or later? • the time to market or the time to delivery, and
In order to characterize the role of time in the 13 empirical • as an axis of change on which to pick suitable moments
studies, the primary researcher performed a detailed analysis for action.
ofthe13papersthatwerecodedasempiricalstudiesoftrade- Additionally, a number of unique attributes and aspects
off decisions involving time. Reading through the papers, the surfaced once that were interpreted as tangential, since the
researcherperformediterativequalitativecoding.Heidentified concept of time was not central to the focus or nature of
all mentioning of ‘time’, reviewed each of their contexts, decision making. For example, this included a case that dis-
and iteratively developed a set of codes that described each cussed technical consideration of real-time systems in project
new occurrence while continuing to apply to the existing decisions or a discussion of how fixed-time release cycles
occurrences. The resulting set of categories is summarized provided consistent structure and rhythm to an organization’s
in Table X. Time in this set of papers is discussed most processes.
importantly as: It is the last conception listed above, the axis of change,
• A constrained resource in software project, where intertemporal decisions arise explicitly. In this set, they
• Thetimeittakestoapplyamethod(e.g.designedbythe arose in particular in two specific forms, each represented by
8
TABLEX
USAGEOFTIMEASADIMENSION
Timeas... Constrained Time Sequenceof Time AxisofChange Other:Timeas...
Resource toApply ProjectEvents toMarket
...in
Abdel-Hamid[A6] X
Al-Naeem[A7] X X ...inreal-timesystems
Cushing[A8] X ...adomainconcern
Filho[B1] X ... time zone differences in distributed
teams
Hatton[A9] X X X
Karlsson[A10] X
Lindgren[A2] X X ReleasePlanning ...rhythmofanorganization
Martini[A3] X X X TechDebt
Oliveira[A11] X X X
Racheva[A4] X X TechDebt
Slaughter[A12] X X
VanSolingen[A13] X X
Wohlin[A5] X X X ReleasePlanning ...pathtoanuncertainfutureofchang-
ingpractice
Zannier[A1] X
two distinct papers: findingssuggestadistinctionbetweenproblemstructuringand
1) Technical Debt management raises questions around problemsolving,andtheauthorsconcludethatbothmodesare
whentorepay,andpapersdiscussedhowthesedecisions relevant and one of them is typically the dominant approach.
are being made; When the focus of decision making was on structuring, as
2) Release planning raises questions of timing and of was most commonly the case in the software design activities
which requirements to prioritize and include for a studied,naturalisticdecisionmakingmodesdominated.Where
given release. the focus was on problem solving, rational modes dominated.
In each modes, aspects of the other were present as well. The
However, in neither of these cases was explicit attention
authors add that “software designers often use satisficing and
giventobehavioralinsights,orintohowdecisionmakersarrive
singular evaluation in trying different approaches to design” –
at their choices.
where one option is checked for plausibility rather than being
evaluated against other options, as described by naturalistic
D. Which assumptions on decision making underpin these
decision making frameworks [11].
studies?
However, the normative, rational decision making model
The predominant model of decision making in the relevant also dominates the empirical papers that addressed the notion
papers, so dominant that it is normally not made explicit, of trade-offs across time more explicitly.
is a normative decision making model that builds on a In Lindgren’s study of release planning, intertemporal con-
Taylorist perspective on management, focused on efficiency siderationsareforegroundedexplicitlyindiscussionsofshort-
and effectiveness, measured in the most scientifically accurate andlong-termplanning,andanexplicitconnectionismadeto
manner possible. Decision making assumes the presence and the need to repay technical debt [A2]. However, while the
validity of normative theories of decision analysis in which paper highlights the need for longer-term perspectives and
clearly defined options are weighed against stated criteria to reports on empirical work, it focuses on larger questions and
determine the best choice. The actual choosing is then often leaves open how precisely the decision makers acted in these
presumedtobeunproblematic.Rationalchoiceisthestandard decisions.
model, sometimes with explicit awareness of its limitations, In Martini’s study of technical debt management, time is
often articulated in the frame of, or consistent with, bounded similarly prominent: “the TD theoretical framework instanti-
rationality. Yet, awareness is also present that “research has atesarelationshipbetweenthecostandtheimpactofasingle
proven that humans make trade-off analyses continuously– if sub-optimal solution over time. In particular, the metaphor
not on the basis of objective measurements then on intuition.” stresses the short-term gain given by a sub-optimal solution
(VanSolingen[A13]pointingtoBeach’sImageTheory).One againstthelong-termoneconsideredoptimal”[A3].Decisions
paperexplicitlyproposestocontrast‘rational’decisionmaking have to be taken at the right time and have to anticipate
with‘naturalistic’frameworksandcategorizesframeworksin- uncertain future outcomes. The assumptions that are surfaced
cludingSimon’sBoundedRationality,ProspectTheory,Image about the decision makers suggest that in the presence of
Theory, and other models [A1]. The paper itself does not perfect information, they would take the correct, optimal
conduct empirical work, but a subsequent paper of the same decision, surfacing assumptions of rational choice, contingent
authors does [21]. upon and bounded by the availability of information.
In this paper, Zannier and Maurer conduct multiple inter- Racheva’s study examines requirements prioritization in
views to develop an understanding of two modes of decision an agile environment at inter-iteration time– the moment
making characterized as rational and naturalistic [21]. The “when requirements are re-prioritized in the face of project
9
TABLEXI There is awareness in parts of the empirical literature that
NUMBEROFPAPERSWITHTIMEASATRADE-OFFDIMENSION normativedecisiontheoryhaslimitedrelevancefordescriptive
and explanatory purposes. Nonetheless, little empirical work
Empirical 7
EmpiricalComponent 19 surfacedthatexplicitlypursuesempiricallygrounded,descrip-
NonEmpirical 12 tive approaches, and none that studied trade-offs in time in
LiteratureReview 0
depth.
Total 38
However, the most explicit discussion of intertemporal
choice and trade-offs were found in decisions about technical
uncertainties”. At that stage, trade-offs consist in choosing debt management. This suggests that the body of work on
what to do now and what not to do at the next iteration, a technical debt should be analyzed in more depth to char-
decision taken between iterations when requirements are re- acterize the tension between normative decision theory and
prioritized[A4].Overtime,moreinformationwillbedelivered descriptiveapproachesandtoidentifyopportunitiestoimprove
and less of the limited resource of time will be available to our understanding of trade-off decision making in practice.
act on it: “The client can wait to the last responsible moment
B. Threats to Validity
...to make his decision...The term ‘responsible’ means that
the client needs to understand the last point of time to make 1) Internal Validity: Although we followed Kitchenham’s
a decision without affecting the delivery of the project” [A4]. procedure for systematic literature reviews, minor deviations
Theframeworkthatisintroducedaimstoprovideaconceptual from the protocol should be noted:
frame for uncertainty over time by introducing Real Options • The documented of detailed codes being merged is not
Analysis. Given the agile focus, it is unsurprising that the comprehensive. These codes were merged carefully and
researchfocusesongroundingproposalsonempiricalinsights only when the terms were close, as discussed; however,
and being responsive to the actual behavior of practitioners. this limits the traceability of analysis.
The underpinning assumptions are implicit, but build clearly • Within the TD set, a corpus was constructed to enable
on ideas of bounded rationality. future corpus-assisted discourse analysis. However, we
Finally,Wohlin’spaperonreleaseplanningaimstoprovide were unable to include 4 papers which could not be
guidance for which types of criteria practitioners should con- converted from .pdf to text (presumably due to their
sider when conducting requirements prioritization for release encoding).
planning.However,theactualdecisionmakingisnotdiscussed 2) External Validity: The searches were limited to 3
[A5]. databases, and no snowballing was conducted. This limits the
external validity of our findings. However, the databases we
V. CONCLUSIONSANDOUTLOOK used are commonly considered the main sources, and Google
Scholar is often seen as the ‘most comprehensive’ source.
A. Discussion
By including “time” in our search term, we wanted to
Trade-offdecisionsinsoftwareengineeringhavebeenstud- get a sense of how time was treated as a dimension in
ied and modeled for a long time. However, the role of time empirical discussions of trade-offs in SE. However, not all
in most studies focuses on time as a limited resource and a relevant papers discuss time in this manner: some are about
ticking clock. However, some work, such as technical debt SEdecisionmakingingeneral.Wewerenottryingtoexamine
research, has foregrounded the attention to trade-off decisions the assumptions underlying SE decision making in general,
across time. and our results cannot be generalized as such.
In general, it is difficult to find the most relevant work Some of the test searches also included the term “require-
on such a subtle topic as ‘decision making involving trade- ments engineering”, but this term was later dropped. While
off decisions across time’, because the terminology that can it is plausible that many of the results could be captured
be used to describe it is not stabilized and thus the terms by software engineering, the search cannot be said to be
are used in ambiguous and varied ways. This means that we generalizable to requirements engineering.
cannot assume we have covered the body of literature that 3) Construct Validity: Intertemporal choice is not a term
in fact discusses these questions comprehensively. However, that is used in SE literature. Indeed, there appears to be
because the main goal is to understand common assumptions no blanket term for describing the types of decisions and
andnorms,acomprehensiveidentificationofallworkshaving tradeoffs that we attempt to study in this literature review. We
studied these aspects in depth is not the primary criterion. thus refrained from the use of terms specific to the domain
A number of studies have suggested that normative models of intertemporal choice to ensure we identify how the SE
are inadequate in explaining how people actually take de- community talks about these concerns. We believe this is an
cisions [5]. Behavioral perspectives and empirical research adequate measure to tease out intertemporal choice from the
are needed to provide new and deeper understanding on larger body of SE literature, but this cannot be guaranteed,
the practice of software engineering [13]. This suggests that as authors may speak about intertemporal choice in different
more descriptive research is needed to provide a bottom-up terms that may have been missed by our search.
empirically grounded description of decision making. We are The search terms are known to be incomplete in the sense
aware of some studies within the domain of SE, but none that terms related to ‘time’ and ‘decision making’, and dis-
focused on time trade-offs. ciplinary terms such as “requirements engineering”, have not
10
been included in the search. This choice was taken since the ”end of life” OR end-of-life OR future OR ”life cycle”
aimwastoidentifycommonassumptionsanddecisionmaking OR ”life-cycle” OR ”lifecycle” OR enduring OR tem-
theories within an acceptable time frame. poral OR sustain*)
4) Reliability: Not all researchers were trained software AND (stakeholder OR values OR preferences OR goals
engineers,constitutingathreattothereliabilityoftheliterature OR benefits OR elicitation OR negotiation OR prioriti-
review because terms might not be properly understood. This zation OR incentive) AND (”software engineering”)
had to be addressed and considered in the setup of the
protocol, as discussed above: We conducted iterative coding,
APPENDIXB
reviewed with the internal expert, and erred on the side of
TRADE-OFFDIMENSIONSDESCRIPTIONS
caution. In order to ensure reliability in relevancy rating and
Herewecapturedself-reporteddimensionsthatweencoun-
coding, researchers conducted the work individually and then
tered while coding:
afterward compared. Where there was discontinuity between
voters or where they were not sure how to evaluate a paper, • (benefit) as a direct component of cost-benefit analysis
results were obtained by consensus. If errors or ambiguities • (competition)
in the review process were discovered, another iteration was • (complexity) Complexity of project architecture/code
done to correct for that. • (labour)
The internal and external review were conducted from a • (methodology) software development methodology or
perspective of software engineering expertise. system development life cycle
• (opportunities)
• (ROI) Return on Investment
C. Future work
• (goals) goals
Topreparetheempiricalstudyoftimetrade-offs,ananalysis • (techdebt) Technical Debt
ofthetechnicaldebtcorpusisthelogicalnextstep:Thiswork • (vendor) vendor viability
isveryclearlyboundedandfocusedonacloselyrelatedareaof
highrelevancefortheSEcommunity.Aspartofthisliterature
review,wehavepreparedatextcorpuswiththeidentified2311 APPENDIXC
SELECTEDPAPERS
papers that can be downloaded and analyzed quantitatively.
This will make it possible to use complementary techniques RESEARCHHASANEMPIRICALCOMPONENTWHICH
such as corpus-assisted discourse analysis to identify the as- STUDIESDECISIONS
sociations and meanings attributed to time trade-off decisions
[A1] C.ZannierandF.Maurer,“Aqualitativeempiricalevaluationofdesign
in the domain of technical debt management. decisions,”inACMSIGSOFTSoftwareEngineeringNotes,vol.30,no.4.
ACM,2005,pp.1–7.
[A2] M. Lindgren, R. Land, C. Norstro¨m, and A. Wall, “Key aspects of
APPENDIXA software release planning in industry,” in Software Engineering, 2008.
PRETESTSEARCHSTRINGS ASWEC 2008. 19th Australian Conference on. IEEE, 2008, pp. 320–
329.
A. Pretest 1: June 30, 2016
[A3] A. Martini and J. Bosch, “An empirically developed method to aid
decisionsonarchitecturaltechnicaldebtrefactoring:Anacondebt,”inPro-
• tradeoff AND software engineering OR requirement*
ceedings of the 38th International Conference on Software Engineering
engineering Companion. ACM,2016,pp.31–40.
• intertemporal choice AND software engineering OR re- [A4] Z. Racheva and M. Daneva, “How do real options concepts fit in
agile requirements engineering?” in Software Engineering Research,
quirement* engineering
Management and Applications (SERA), 2010 Eighth ACIS International
• behavioral economic* AND software engineering OR Conferenceon. IEEE,2010,pp.231–238.
requirement* engineering [A5] C.WohlinandA.Aurum,“Whatisimportantwhendecidingtoinclude
a software requirement in a project or release?” in Empirical Software
Engineering,2005.2005InternationalSymposiumon. IEEE,2005,pp.
B. Pretest 2: October 28, 2016 10–pp.
[A6] T.K.Abdel-Hamid,K.Sengupta,andC.Swett,“Theimpactofgoals
• trade-off OR tradeoff OR ”trade off” OR conflict on software project management: An experimental investigation,” MIS
AND long-living OR ”long living” OR ”long lasting” quarterly,pp.531–555,1999.
[A7] T.Al-Naeem,I.Gorton,M.A.Babar,F.Rabhi,andB.Benatallah,“A
OR ”long-lasting” OR longevity OR ”long term” OR
quality-driven systematic approach for architecting distributed software
”end of life” OR end-of-life OR future OR ”life cycle” applications,” in Proceedings of the 27th international conference on
OR ”life-cycle” OR ”lifecycle” OR enduring OR Softwareengineering. ACM,2005,pp.244–253.
[A8] J. B. Cushing, K. M. Winters, and D. Lach, “Software for scientists
temporal OR sustain*
facingwickedproblemslessonsfromthevistasproject,”inProceedings
AND ”requirement engineering” OR ”requirements of the 16th Annual International Conference on Digital Government
engineering” Research. ACM,2015,pp.61–70.
[A9] S. Hatton, “Choosing the right prioritisation method,” in Software
Engineering,2008.ASWEC2008.19thAustralianConferenceon. IEEE,
• (trade-off OR tradeoff OR ”trade off” OR conflict) 2008,pp.517–526.
AND (long-living OR ”long living” OR ”long lasting” [A10] J. Karlsson and K. Ryan, “A cost-value approach for prioritizing
requirements,”IEEEsoftware,vol.14,no.5,pp.67–74,1997.
OR ”long-lasting” OR longevity OR ”long term” OR
[A11] F.Oliveira,A.Goldman,andV.Santos,“Managingtechnicaldebtin
softwareprojectsusingscrum:Anactionresearch,”inAgileConference
1Downfrom246. (AGILE),2015. IEEE,2015,pp.50–59.