Empirical research in software engineering : concepts, analysis, and applications

486 Pages·2015·10.228 MB·English

by  Malhotra, Ruchika

INFORMATION TECHNOLOGY M
a
l
h
o
t
r
a
EMPIRICAL
EMPIRICAL RESEARCH IN SOFTWARE ENGINEERING
CONCEPTS, ANALYSIS, AND APPLICATIONS RESEARCH
E in
M
P
SOFTWARE
Empirical Research in Software Engineering: Concepts, Analysis, and Applications shows how to I
R
implement empirical research processes, procedures, and practices in software engineering. The book describes the I
C
necessary steps to perform replicated and empirical research. It explains how to plan and design experiments, conduct A
ENGINEERING
L
systematic reviews and case studies, and analyze the results produced by the empirical studies.

R
The book balances empirical research concepts with exercises, examples, and real-life case studies. The author discusses E Study
S
Definition
the process of developing predictive models, such as defect prediction and change prediction, on data collected from E
A CONCEPTS, ANALYSIS,
source code repositories. She also covers the application of machine learning techniques in empirical software engineering,
R Reporting Experimental
includes guidelines for publishing and reporting results, and presents popular software tools for carrying out empirical C Results Design
H AND APPLICATIONS
studies.

I
N
Mining Data
Validating
Ruchika Malhotra is an assistant professor in the Department of Software Engineering at Delhi Technological University S Threats from
O Repositories
(formerly Delhi College of Engineering). She was awarded the prestigious UGC Raman Fellowship for pursuing post-
F
T
doctoral research in the Department of Computer and Information Science at Indiana University–Purdue University. She W Model Data Analysis
Development & & Statistical
earned her master’s and doctorate degrees in software engineering from the University School of Information Technology A Interpretation Testing
of Guru Gobind Singh Indraprastha University. She received the IBM Best Faculty Award in 2013 and has published R
E Ruchika Malhotra
more than 100 research papers in international journals and conferences. Her research interests include software testing,

E
improving software quality, statistical and adaptive prediction models, software metrics, neural nets modeling, and the N
G
definition and validation of software metrics.
I
N
E
E
R
I
N
G
K25508
ISBN: 978-1-4987-1972-8
90000
6000 Broken Sound Parkway, NW
Suite 300, Boca Raton, FL 33487
711 Third Avenue
New York, NY 10017 9 781498 719728
an informa business 2 Park Square, Milton Park www.crcpress.com
Abingdon, Oxon OX14 4RN, UK A CHAPMAN & HALL BOOK
EMPIRICAL
RESEARCH
in
SOFTWARE
ENGINEERING
CONCEPTS, ANALYSIS,
AND APPLICATIONS

EMPIRICAL
RESEARCH
in
SOFTWARE
ENGINEERING
CONCEPTS, ANALYSIS,
AND APPLICATIONS
Ruchika Malhotra
Boca Raton London New York
CRC Press is an imprint of the
Taylor & Francis Group, an informa business
CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742
© 2016 by Taylor & Francis Group, LLC
CRC Press is an imprint of Taylor & Francis Group, an Informa business
No claim to original U.S. Government works
Version Date: 20150821
International Standard Book Number-13: 978-1-4987-1973-5 (eBook - PDF)
This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been
made to publish reliable data and information, but the author and publisher cannot assume responsibility for the valid-
ity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright
holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this
form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may
rectify in any future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or uti-
lized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopy-
ing, microfilming, and recording, or in any information storage or retrieval system, without written permission from the
publishers.
For permission to photocopy or use material electronically from this work, please access www.copyright.com (http://
www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923,
978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For
organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for
identification and explanation without intent to infringe.
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the CRC Press Web site at
http://www.crcpress.com
I dedicate this book to my grandmother, the late Shrimathi Shakuntala Rani Malhotra,
for her infinite love, understanding, and support. Without her none of my success would
have been possible and I would not be who I am today. I miss her very much.

Contents
Foreword ......................................................................................................................................xix
Preface ...........................................................................................................................................xxi
Acknowledgments ....................................................................................................................xxiii
Author ..........................................................................................................................................xxv
1. Introduction .............................................................................................................................1
1.1 What Is Empirical Software Engineering? ................................................................1
1.2 Overview of Empirical Studies ...................................................................................2
1.3 Types of Empirical Studies ..........................................................................................3
1.3.1 Experiment ........................................................................................................4
1.3.2 Case Study ........................................................................................................5
1.3.3 Survey Research ...............................................................................................6
1.3.4 Systematic Reviews ..........................................................................................7
1.3.5 Postmortem Analysis ......................................................................................8
1.4 Empirical Study Process ..............................................................................................8
1.4.1 Study Definition ...............................................................................................9
1.4.2 Experiment Design ........................................................................................10
1.4.3 Research Conduct and Analysis ..................................................................11
1.4.4 Results Interpretation ....................................................................................12
1.4.5 Reporting ........................................................................................................12
1.4.6 Characteristics of a Good Empirical Study ................................................12
1.5 Ethics of Empirical Research .....................................................................................13
1.5.1 Informed Content ..........................................................................................14
1.5.2 Scientific Value ...............................................................................................15
1.5.3 Confidentiality ...............................................................................................15
1.5.4 Beneficence ......................................................................................................15
1.5.5 Ethics and Open Source Software ...............................................................15
1.5.6 Concluding Remarks .....................................................................................15
1.6 Importance of Empirical Research ...........................................................................16
1.6.1 Software Industry ..........................................................................................16
1.6.2 Academicians .................................................................................................16
1.6.3 Researchers .....................................................................................................17
1.7 Basic Elements of Empirical Research .....................................................................17
1.8 Some Terminologies ...................................................................................................18
1.8.1 Software Quality and Software Evolution .................................................18
1.8.2 Software Quality Attributes .........................................................................20
1.8.3 Measures, Measurements, and Metrics ......................................................20
1.8.4 Descriptive, Correlational, and Cause–Effect Research ...........................22
1.8.5 Classification and Prediction .......................................................................22
1.8.6 Quantitative and Qualitative Data ..............................................................22
1.8.7 Independent, Dependent, and Confounding Variables ...........................23
1.8.8 Proprietary, Open Source, and University Software ................................24
1.8.9 Within-Company and Cross-Company Analysis .....................................25
vii
viii Contents
1.8.10 Parametric and Nonparametric Tests .......................................................26
1.8.11 Goal/Question/Metric Method .................................................................26
1.8.12 Software Archive or Repositories .............................................................28
1.9 Concluding Remarks ..................................................................................................28
Exercises ..................................................................................................................................28
Further Readings ....................................................................................................................29
2. Systematic Literature Reviews ...........................................................................................33
2.1 Basic Concepts .............................................................................................................33
2.1.1 Survey versus SRs .......................................................................................33
2.1.2 Characteristics of SRs ..................................................................................34
2.1.3 Importance of SRs .......................................................................................35
2.1.4 Stages of SRs .................................................................................................35
2.2 Case Study ....................................................................................................................36
2.3 Planning the Review ..................................................................................................37
2.3.1 Identify the Need for SR .............................................................................37
2.3.2 Formation of Research Questions .............................................................38
2.3.3 Develop Review Protocol ...........................................................................39
2.3.4 Evaluate Review Protocol ...........................................................................45
2.4 Methods for Presenting Results ................................................................................46
2.4.1 Tools and Techniques ..................................................................................46
2.4.2 Forest Plots ...................................................................................................49
2.4.3 Publication Bias ............................................................................................51
2.5 Conducting the Review ..............................................................................................51
2.5.1 Search Strategy Execution ..........................................................................51
2.5.2 Selection of Primary Studies ......................................................................53
2.5.3 Study Quality Assessment .........................................................................53
2.5.4 Data Extraction ............................................................................................53
2.5.5 Data Synthesis ..............................................................................................53
2.6 Reporting the Review .................................................................................................56
2.7 SRs in Software Engineering .....................................................................................58
Exercises ..................................................................................................................................60
Further Readings ....................................................................................................................61
3. Software Metrics...................................................................................................................65
3.1 Introduction .................................................................................................................65
3.1.1 What Are Software Metrics? ......................................................................66
3.1.2 Application Areas of Metrics .....................................................................66
3.1.3 Characteristics of Software Metrics ..........................................................67
3.2 Measurement Basics ...................................................................................................67
3.2.1 Product and Process Metrics .....................................................................68
3.2.2 Measurement Scale......................................................................................69
3.3 Measuring Size ............................................................................................................71
3.4 Measuring Software Quality .....................................................................................72
3.4.1 Software Quality Metrics Based on Defects ............................................72
3.4.1.1 Defect Density ..............................................................................72
3.4.1.2 Phase-Based Defect Density .......................................................73
3.4.1.3 Defect Removal Effectiveness ....................................................73
Contents ix
3.4.2 Usability Metrics ............................................................................................74
3.4.3 Testing Metrics ...............................................................................................75
3.5 OO Metrics ...................................................................................................................76
3.5.1 Popular OO Metric Suites .............................................................................77
3.5.2 Coupling Metrics ...........................................................................................79
3.5.3 Cohesion Metrics............................................................................................83
3.5.4 Inheritance Metrics ........................................................................................85
3.5.5 Reuse Metrics .................................................................................................86
3.5.6 Size Metrics .....................................................................................................88
3.6 Dynamic Software Metrics ........................................................................................89
3.6.1 Dynamic Coupling Metrics ..........................................................................89
3.6.2 Dynamic Cohesion Metrics ..........................................................................89
3.6.3 Dynamic Complexity Metrics ......................................................................90
3.7 System Evolution and Evolutionary Metrics ...........................................................90
3.7.1 Revisions, Refactorings, and Bug-Fixes ......................................................91
3.7.2 LOC Based ......................................................................................................91
3.7.3 Code Churn Based .........................................................................................91
3.7.4 Miscellaneous .................................................................................................92
3.8 Validation of Metrics ..................................................................................................92
3.9 Practical Relevance .....................................................................................................93
3.9.1 Designing a Good Quality System ..............................................................93
3.9.2 Which Software Metrics to Select? ..............................................................94
3.9.3 Computing Thresholds .................................................................................95
3.9.3.1 Statistical Model to Compute Threshold ....................................96
3.9.3.2 Usage of ROC Curve to Calculate the Threshold Values .........98
3.9.4 Practical Relevance and Use of Software Metrics in Research ...............99
3.9.5 Industrial Relevance of Software Metrics ................................................100
Exercises ................................................................................................................................100
Further Readings ..................................................................................................................101
4. Experimental Design .........................................................................................................103
4.1 Overview of Experimental Design .........................................................................103
4.2 Case Study: Fault Prediction Systems ....................................................................103
4.2.1 Objective of the Study .................................................................................104
4.2.2 Motivation .....................................................................................................104
4.2.3 Study Context ...............................................................................................105
4.2.4 Results ...........................................................................................................105
4.3 Research Questions...................................................................................................106
4.3.1 How to Form RQ? ........................................................................................106
4.3.2 Characteristics of an RQ .............................................................................107
4.3.3 Example: RQs Related to FPS .....................................................................108
4.4 Reviewing the Literature .........................................................................................109
4.4.1 What Is a Literature Review? .....................................................................109
4.4.2 Steps in a Literature Review.......................................................................110
4.4.3 Guidelines for Writing a Literature Review ............................................111
4.4.4 Example: Literature Review in FPS ...........................................................112
4.5 Research Variables ....................................................................................................117
4.5.1 Independent and Dependent Variables ....................................................117