A Friendly Introduction to Software Testing

229 Pages·2016·0.777 MB·English

by  Bill Laboon

A Friendly Introduction
to Software Testing
Bill Laboon
© 2017-02-24
Compiled in PDFLATEX
for AKS and CKN
2
Contents
1 Introduction 7
1.1 The Story So Far . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.2 Testing and Quality Assurance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3 What This Book Covers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.4 What This Book Does Not Cover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.5 A Note on Language Choice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2 What is Software Testing? 11
2.1 A Definition of Software Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2 Verification and Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3 A Preliminary Definition of Defect . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.4 A Real-Life Testing Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3 Why Test Software? 17
3.1 To Test Or Not To Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2 No Software Developer is Perfect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.3 Catching Defects Sooner Rather than Later . . . . . . . . . . . . . . . . . . . . . . . 18
3.4 Stability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.5 Customer Advocacy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.6 An Independent, Whole-System Perspective . . . . . . . . . . . . . . . . . . . . . . . 19
3.7 Ensuring Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.8 Risk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4 Testing Basics 21
4.1 Equivalence Classes and Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.2 Interior and Boundary Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.3 Base Cases, Edge Cases, Corner Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.4 Success Cases and Failure Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.5 Black-, White-, and Grey-Box Testing . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.6 Static and Dynamic Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
5 Requirements 29
5.1 Testability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5.2 Functional versus Non-Functional . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.3 A Note on Naming Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
6 Test Plans 37
6.1 The Basic Layout of a Test Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
6.2 Developing a Test Plan. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3
4 CONTENTS
6.3 Test Fixtures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
6.4 Executing a Test Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
6.5 Test Run Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
6.6 Traceability Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
7 Breaking Software 51
7.1 Errors To Look For . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
7.2 The List Goes On and On . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
8 A Test Plan Walkthrough 61
8.1 Examining The Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
8.2 Plotting Out the Test Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
8.3 Filling Out the Test Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
8.4 Determining Focus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
8.5 Test Cases for a Non-Functional Requirement . . . . . . . . . . . . . . . . . . . . . . 69
9 Defects 71
9.1 What is a Defect? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
9.2 The Defect Life Cycle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
9.3 A Standardized Defect Template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
9.4 Exceptions to the Template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
9.5 Defect Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
10 Smoke And Acceptance Testing 83
10.1 Smoke Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
10.2 Acceptance Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
11 Exploratory Testing 87
11.1 Benefits and Drawbacks of Exploratory Testing . . . . . . . . . . . . . . . . . . . . . 87
11.2 Guidelines for Exploratory Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
12 Manual versus Automated Testing 91
12.1 Benefits and Drawbacks of Manual Testing . . . . . . . . . . . . . . . . . . . . . . . 91
12.2 Benefits and Drawbacks of Automated Testing . . . . . . . . . . . . . . . . . . . . . 93
12.3 The Real World . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
13 An Introduction to Unit Testing 97
13.1 Unit Tests: The Very Idea . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
13.2 An Example in Natural Language. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
13.3 Turning Our Example Into a Unit Test . . . . . . . . . . . . . . . . . . . . . . . . . . 100
13.4 Problems With Unit Testing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
13.5 Creating a Test Runner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
14 Advanced Unit Testing 109
14.1 Test Doubles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
14.2 Stubs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
14.3 Mocks and Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
14.4 Fakes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
14.5 Setup and Teardown . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
14.6 Testing System Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
14.7 Testing Private Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
14.8 Unit Test Structure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
CONTENTS 5
14.9 Code Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
15 Test-Driven Development 129
15.1 What is TDD? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
15.2 The Red-Green-Refactor Loop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
15.3 Principles of Test-Driven Development . . . . . . . . . . . . . . . . . . . . . . . . . . 133
15.4 Example: Building FizzBuzz via TDD . . . . . . . . . . . . . . . . . . . . . . . . . . 133
15.5 Benefits of TDD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
15.6 Drawbacks of TDD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
16 Writing Testable Code 141
16.1 What Do We Mean By Testable Code? . . . . . . . . . . . . . . . . . . . . . . . . . . 141
16.2 The Basic Strategies for Testable Code . . . . . . . . . . . . . . . . . . . . . . . . . . 142
16.3 Provide a Scriptable Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
16.4 Write Tests Up-Front. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
16.5 DRYing Up Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
16.6 Dependency Injection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
16.7 TUFs and TUCs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
16.8 Dealing with Legacy Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
16.9 Final Thoughts On Writing Testable Code . . . . . . . . . . . . . . . . . . . . . . . . 150
17 Pairwise and Combinatorial Testing 151
17.1 Permutations and Combinations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
17.2 Pairwise Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
17.3 n-Way Interactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
17.4 Working with Larger Variable Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
18 Stochastic and Property-Based Testing 159
18.1 Infinite Monkeys and Infinite Typewriters . . . . . . . . . . . . . . . . . . . . . . . . 159
18.2 Property-Based Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
18.3 Smart, Dumb, Evil, and Chaos Monkeys . . . . . . . . . . . . . . . . . . . . . . . . . 161
18.4 Mutation Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
19 Performance Testing 169
19.1 Categories of Performance Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
19.2 Testing Performance: Thresholds and Targets . . . . . . . . . . . . . . . . . . . . . . 170
19.3 Key Performance Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
19.4 Testing Service-Oriented Indicators: Response Time . . . . . . . . . . . . . . . . . . 172
19.5 Testing Service-Oriented Indicators: Availability . . . . . . . . . . . . . . . . . . . . 175
19.6 Testing Efficiency-Oriented Indicators: Throughput . . . . . . . . . . . . . . . . . . . 178
19.7 Testing Efficiency-Oriented Indicators: Utilization . . . . . . . . . . . . . . . . . . . 179
19.8 General Tips and Advice When Performance Testing . . . . . . . . . . . . . . . . . . 180
20 Security Testing 183
20.1 Challenges in Security Testing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
20.2 Basic Concepts in Computer Security . . . . . . . . . . . . . . . . . . . . . . . . . . 185
20.3 Common Attacks and How to Test Against Them. . . . . . . . . . . . . . . . . . . . 187
20.4 Penetration Testing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
20.5 General Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
21 Interacting with Stakeholders 193
21.1 Who and What are Stakeholders? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
6 CONTENTS
21.2 Reporting and Communicating . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
21.3 The Red-Yellow-Green Template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
21.4 Status Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
21.5 A Note on Managing Expectations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
21.6 A Note on Meetings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
21.7 Clarifying Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
21.8 Ethical Obligations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
21.9 Respect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
22 Conclusion 203
23 Testing Templates 205
23.1 Test Case Template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
23.2 Defect Reporting Template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
23.3 Red-Yellow-Green Template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
23.4 Daily Status Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
24 Using Reflection to Test Private Methods in Java 207
25 Further Reading 213
26 Glossary of Terms 215
27 Acknowledgments 229
Chapter 1
Introduction
Let me say that I rarely ever read introductions to technical books; I won’t be offended if you don’t
read this one.
1.1 The Story So Far
A long time ago, in a company far, far away, I was a test lead. One of my responsibilities was
to interview potential test engineers for our team, and I found that many candidates did not have
a background in testing. Those who did well in their career often picked things up as they went.
Even those who had degrees in computer science (or a related field) had often not learned about
software testing. Developers had to learn how to properly test their own code, often following the
old “apprentice” model by asking more senior developers what needed to be tested.
I figured that I could complain about it, or do something about it. I was able to convince the
Computer Science department at the University of Pittsburgh to allow me to develop and teach a
class on software testing. This small class has grown to cover many different aspects of software
quality, and become a popular, ongoing course (CS 1632: Software Quality Assurance, if you’re
interested in taking it!). I developed my own curriculum, as I couldn’t find a good book or syllabus
on the subject that struck the right balance between theory and application. Finding a good book
to use was even more difficult than finding a good Quality Assurance (QA) Engineer to interview!
Once again, I figured I could either complain about it, or do something about it, and once again, I
chose the latter.
1.2 Testing and Quality Assurance
Testing software is a big part of the software development process, and useful not just for those
looking specifically for a career in QA. A developer who doesn’t care about software quality is not
a good developer. This book is targeted to those interested in software testing or writing tests as a
developer.
7
8 CHAPTER 1. INTRODUCTION
1.3 What This Book Covers
This book is intended to provide a relatively comprehensive overview of software testing. By the
end, my expectation is that the reader will have all the skills necessary to enter the workforce in
the field of quality assurance, although I feel that managers, developers, and others involved in the
software development process may also find it very useful.
To that end, the book starts with a generalized overview of the field—what is software testing,
anyway? It’s difficult to study something without understanding what it is! We’ll then move on to
some of the theory and terminology used by those in the software testing industry. I’ll let you in on
a little secret—this is probably the least interesting part of the book. However, we really should be
speaking the same language when we talk about concepts. It would be difficult to explain concepts
in plumbing if I were forced to avoid using the word “pipe” for fear of someone not understanding
it.
After that, we’ll move on to the basics of developing a manual test plan, and dealing with defects
that are found in the process of executing it. Manual tests are rarer now than they used to be;
automated tests have relieved much of the tedium of executing them. There are definite benefits to
developing tests qua tests without worrying about a specific language syntax or tool chain, though.
We’ll also go over how to properly record our testing and report defects that are found from our
testing.
Once writing manual tests is old hat, we can move on to automated tests—system-level and unit
tests. Automated tests allow you to very quickly execute tests, from either a very low level (like
making sure a sorting algorithm works correctly) to a very high level (ensuring that you can add
something to your cart on an e-commerce website). If you’ve actually executed some manual test
plans, you’ll see why letting the computer execute all of the tests for you is such a time-saver.
Perhaps more importantly, it’s an excellent aggravation-saver, as repeatedly running manual tests
is one of the fastest ways to drive a tester insane.
Finally, we get to the really interesting stuff! This section of the book is where we’ll get to read
about specialized kinds of testing, like combinatorial, performance, and security tests. The world of
software testing is quite a large one—testing embedded software is very different from testing web
applications,performancetestingisverydifferentfromfunctionaltesting,andtestingforastartup’s
firstprototypeproductisverydifferentfromtestingforalargecompanydevelopingamedicaldevice.
1.4 What This Book Does Not Cover
This book is an introductory text to the world of software testing; there’s definitely lots of room to
explore and go into more detail on all of the topics here. However, the purpose isn’t to try to rival
War and Peaceinlength,buttoprovideageneralintroductiontothefield. Spendinglotsoftimeon
the intricacies of each of them might be interesting to me or to a small subset of the people reading
thebook,butmygoalsaretoprovideagoodfoundationofpracticalsoftwaretestingknowledge,and
give a taste of some of the various specialties and subfields in the industry. Consider it a whirlwind
tour of Europe instead of studying abroad for a year in Paris.
1.5 A Note on Language Choice
The examples I give are in Java and use much of the Java tool chain (e.g., JUnit). This is not due
to some particular love of Java. There are certainly other languages which have a good claim on
1.5. A NOTE ON LANGUAGE CHOICE 9
being easier to use or more helpful in avoiding defects. However, Java does serve as a lingua franca
amongst developers nowadays. It is a standard, popular, ALGOL-derived programming language,
so even if you’re not familiar with it, you can probably reason out what’s happening.
10 CHAPTER 1. INTRODUCTION