Software testing : testing across the entire software development life cycle

273 Pages·2008·2.92 MB·English

by  Gerald D Everett

Software Testing
Testing Across the Entire
Software Development Life Cycle
Gerald D. Everett
Certifi ed Senior Testing Education Specialist
IBM
Raymond McLeod, Jr.
University of Texas at Austin
Austin, TX
∞
This book is printed on acid-free paper.
Copyright © 2007 by John Wiley & Sons, Inc.All rights reserved.
Published by John Wiley & Sons, Inc., Hoboken, New Jersey.
Published simultaneously in Canada.
No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form
or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as
permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without either the prior
written permission of the Publisher, or authorization through payment of the appropriate per-copy fee
to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400,
fax [phone], or on the web at www.copyright.com. Requests to the Publisher for permission
should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street,
Hoboken, NJ 07030, [phone], fax [phone].
Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts
in preparing this book, they make no representations or warranties with respect to the accuracy
or completeness of the contents of this book and specifi cally disclaim any implied warranties of
merchantability or fi tness for a particular purpose. No warranty may be created or extended by sales
representatives or written sales materials. The advice and strategies contained herein may not be
suitable for your situation. You should consult with a professional where appropriate. Neither the
publisher nor author shall be liable for any loss of profi t or any other commercial damages, including
but not limited to special, incidental, consequential, or other damages.
For general information on our other products and services please contact our Customer Care
Department within the U.S. at 877-762-2974, outside the U.S. at 317-572-3993 or fax [phone].
Wiley also publishes its books in a variety of electronic formats. Some content that appears in print,
however, may not be available in electronic format.
Wiley Bicentennial Logo: Richard J. Pacifi co
Library of Congress Cataloging-in-Publication Data:
Everett, Gerald D., 1943-
Software testing : testing across the entire software development life
cycle / by Gerald D. Everett, Raymond McLeod, Jr.
p. cm.
Includes index.
ISBN 978-0-471-79371-7 (cloth)
1. Computer software–Testing. 2. Computer software–Development. I.
McLeod, Raymond.
II. Title.
QA76.76.T48E94 2007
005.1’4–dc22 2007001282
Printed in the United States of America.
10 9 8 7 6 5 4 3 2 1
To my wife Nell and her steadfast encouragement during the
relentless weekends and vacations while I wrote this book.
Jerry
To my good friend Carolyn, whose reminders, suggestions, and
inspiration have made me a better person, father, and appreciator
of the beauty that the world has to offer.
Ray
Contents
Preface xi
Acknowledgments xv
1. Overview of Testing 1
1.1 Introduction 1
1.2 Objectives and Limits of Testing 2
1.3 The Value Versus Cost of Testing 11
1.4 Relationship of Testing to the Software Development Life Cycle 16
1.5 Tester Versus Developer Roles in Software Testing 22
1.6 Putting Software Testing in Perspective 25
1.7 Summary 25
2. The Software Development Life Cycle 29
2.1 Introduction 29
2.2 Methodologies and Tools 29
2.3 The Evolution of System Development Life Cycles 30
2.4 The Phased Development Methodology 33
2.5 The Preliminary Investigation Stage 37
2.6 The Analysis Stage 43
2.7 The Design Stage 46
2.8 The Preliminary Construction Stage 50
2.9 The Final Construction Stage 54
2.10 The Installation Stage 56
2.11 Putting Phased Development in Perspective 57
2.12 Summary 57
3. Overview of Structured Testing 59
3.1 Introduction 59
vii
viii Contents
3.2 Checklist Mentality for Software Testers 60
3.3 SPRAE—A Generic Structured Testing Approach 61
3.4 Putting the Overview of Structured Testing in Perspective 65
4. Testing Strategy 66
4.1 Introduction 66
4.2 The Chess Pieces for Testing Strategies 66
4.3 The Two-Dimensional Testing Strategy Chess Board 70
4.4 The Three-Dimensional Testing Strategy Chess Board 75
4.5 Putting the Testing Strategy into Perspective 77
5. Test Planning 79
5.1 Introduction 79
5.2 The Test Plan 79
5.3 Test Cases 83
5.4 Writing Your Test Plan and Test Cases in the Real World 88
5.5 Test Document Standards 90
5.6 Putting Test Planning in Perspective 91
6. Static Testing 93
6.1 Introduction 93
6.2 Goal of Static Testing 93
6.3 Candidate Documents for Static Testing 94
6.4 Static Testing Techniques 96
6.5 Tracking Defects Detected by Static Testing 98
6.6 Putting Static Testing in Perspective 98
7. Functional Testing 99
7.1 Introduction 99
7.2 Functional Test Cases from Use Cases 100
7.3 An Approach to Functional Testing 103
7.4 An Approach to Regression Testing 106
7.5 Detailed White Box Testing Techniques 107
7.6 Detailed Black Box Testing Techniques 112
7.7 Summary 119
7.8 Putting Functional Testing in Perspective 121
8. Structural (Non-functional) Testing 122
8.1 Introduction 122
8.2 Interface Testing 123
8.3 Security Testing 124
8.4 Installation Testing 125
Contents ix
8.5 The Smoke Test 125
8.6 Administration Testing 126
8.7 Backup and Recovery Testing 126
8.8 Putting Structural Testing in Perspective 127
8.9 Summary 127
9. Performance Testing 129
9.1 Introduction 129
9.2 Workload Planning Techniques 130
9.3 Workload Execution Techniques 134
9.4 Component Performance Testing 135
9.5 Round Trip Performance 136
9.6 Putting Performance Testing in Perspective 147
9.7 Summary 148
10. The Testing Environment 150
10.1 Introduction 150
10.2 Simulations 151
10.3 Benchmarking 151
10.4 Testing Environments 152
10.5 The Goal of a Testing Environment 152
10.6 Good Testing Environments and Why They Should Be Used 155
10.7 Bad Testing Environments and Why They Should Be Avoided 156
10.8 Putting the Testing Environment in Perspective 157
10.9 Summary 157
11. Automated Testing Tools 159
11.1 Introduction 159
11.2 Brief History of Automated Testing Tools for Software 160
11.3 Test Tool Record/Playback Paradigm 162
11.4 Test Tool Touchpoint Paradigms 164
11.5 Test Tool Execution Pardigm 168
11.6 The Benefi ts that Testing Tools Can Provide 169
11.7 The Liabilities that Testing Tools Can Impose 173
11.8 Putting Automated Testing Tools in Perspective 174
11.9 Summary 175
12. Analyzing and Interpreting Test Results 176
12.1 Introduction 176
12.2 Test Cases Attempted Versus Successful 176
12.3 Defect Discovery Focusing on Individual Defects 179
12.4 Defect Discovery Focusing on the Defect Backlog 181
12.5 Defect Discovery Focusing on Clusters of Defects 182
x Contents
12.6 Prior Defect Discovery Pattern Usefulness 187
12.7 The Rayleigh Curve—Gunsights for Defect Discovery Patterns 196
12.8 More Defect Tracking Metrics 200
12.9 Putting Test Results in Perspective 201
12.10 Summary 201
13. A Full Software Development Lifecycle Testing Project 203
13.1 Introduction 203
13.2 Preliminary Investigation Stage 204
13.3 Analysis Stage 206
13.4 Design Stage 213
13.5 Preliminary Construction Stage 219
13.6 Final Construction Stage 229
13.7 Implementation Stage 232
13.8 Postimplementation Stage 232
13.9 Case Study Closure 233
14. Testing Complex Applications 235
14.1 Introduction 235
14.2 1-Tier Applications 235
14.3 2-Tier Applications 237
14.4 3-Tier Applications 241
14.5 n-Tier Applications 246
14.6 Putting Testing Complex Applications in Perspective 249
14.7 Summary 249
15. Future Directions in Testing 250
15.1 Introduction 250
15.2 Future Directions in Software Development That Could Increase
the Need for Testing Professionals 250
15.3 Software Testing Challenges Already Upon Us 251
15.4 Software Testing Near Future Challenges 252
15.5 Software Testing Challenges To Come 252
15.6 Putting Future Testing Directions in Perspective 253
15.7 Summary 254
References 255
Index 259
Preface
A
n informal survey of twenty-one U.S. universities by the authors found that
nineteen were without any software testing courses. When talking with the faculty
responsible for the software testing courses in three of the universities, we learned
that the largest single impediment to creating a software testing course was the
absence of a good textbook. We were told that the current selection of textbooks
necessitated a combination of three or four to cover many of the topics, with some
topics not even covered at all. This situation leaves much of the material coverage to
the professor. If he or she does not have a background in software testing, the text-
books leave gaps that is hard to fi ll.
Whereas this situation is disconcerting, universities and businesses in Europe
and Asia seem to value testing expertise more than in the US. Instead of only three of
twenty-one universities delivering testing education as in the US, the ratio in Europe
is more like seven out of ten. The reason for this discrepancy is because academic
and business cultures that already value software testing do not need to be sold on
the value of a comprehensive, basic textbook on the subject.
THE IMPORTANCE OF SOFTWARE TESTING
Software Testing: Testing Across the Entire Lifecycle provides the fundamental
concepts and approaches to software testing. The topic is important for two reasons.
First, according to US Government surveys there has been an estimated $59.5B
in business losses since 2000 due to poor quality software. Second, based on the
authors’ inability to fi nd experienced software testers to address some of the esti-
mated $22.2B testing opportunity, the current pool of experienced software testers
is already gainfully employed.
The topic merits a book because in the authors’ opinion there is no single, com-
prehensive software testing textbook available that gives novice testers the whole
picture. There are a large number of narrowly scoped, deep textbooks that are
xi
xii Preface
excellent for experienced testers, but they tend to leave the novice tester confused
and discouraged. Our task is to provide the novice tester with a complete coverage
of software testing as it is practiced today, as it will be practiced in the future, and
as a viable career option.
THE APPROACH
Software Testing: Testing Across the Entire Lifecycle takes a four-fold approach.
First, it examines the general mind-set of a tester using non-technical examples
like buying a car. Second, it examines the structured approach that emphasizes test
planning. Third, it examines the choices of software testing approaches and when
during the software development cycle they are normally used. Finally, it walks
the reader through a software development project from end to end, demonstrat-
ing appropriate use of the software testing approaches previously discussed on an
individual basis.
DISTINCTIVE FEATURES
The most distinctive features of Software Testing: A Comprehensive Software Test-
ing Approach are:
• A comprehensive treatment of what a technology professional needs to know
to become a software tester. The presentation sequence builds from simple
examples to complex examples. The descriptions and examples are directed
toward practitioners rather than academicians.
• A chapter on analyzing test results effectively using simple math and complex
math models. We have seen no other software testing textbook that treats
test results analysis statistically. Quite to the contrary, other software testing
textbook authors have expressed the opinion that statistics do not belong in a
testing textbook.
• A choice of case studies.
Case Study A The fi rst case study uses a popular Internet application called
PetStore2 developed by Sun Microsystems to demonstrate best practices ap-
plication development using Java. The textbook demonstrates through reader
exercises how to plan and execute approaches described in Chapters 7 through
12 on this well-known application. The benefi t to the reader is two-fold. First,
the reader is given an application that is well suited for hands-on experience
that reinforces the testing approaches described in the textbook. Second, when
the reader successfully completes the case study exercises, she or he can claim
resumé testing experience with an industry recognized application.
Case Study B The second case study is a step-by-step description and ex-
ercises that follows a successful testing project presented in Chapter 13.
Preface xiii
Diffi culties are encountered along the way because this is a real testing
project. The case study unfolds in a manner that allows the authors to
incorporate most of the testing concepts and approaches discussed indi-
vidually in previous chapters and attempted hands-on in Case Study A.
ORGANIZATION
Chapter 1 provides an overview of testing, addressing such topics as the objectives
and limits of testing, and the value versus the cost of testing. Chapter 2 describes
the system development life cycle (SDLC) within which testing occurs. The major
SDLCs, such as the waterfall cycle, prototyping, rapid application development, and
the phased development methodology are described. This textbook uses the phased
development methodology as its basic software development framework.
Chapter 3 provides an overview of structured testing, explaining a generic struc-
tured testing approach called SPRAE, which consists of the components of SPECI-
FICATION, PREMEDITATION, REPEATABAILITY, ACCOUNTABILITY, AND
ECONOMY. Following, in Chapter 4, is an overview of four basic testing strategies—
Static, White Box, Black Box, and Performance (Load) Testing. Both two- and three-
dimensional “Chess Boards” are used to illustrate these basic strategies. Once the testing
strategy has been devised, test planning can proceed and that is the subject of Chapter 5.
Guidelines are offered for writing your Test Plan and Test Cases in the real world.
Chapters 6-9 explain the basic types of testing introduced in Chapter 4—
Chapter 6 explains Static Testing, Chapter 7 explains Functional Testing, Chapter 8
explains Structural (Non-functional) testing, and Chapter 9 explains Performance
Testing. As an example of the thoroughness of these explanations, the discussion of
Structural Testing includes coverage of Interface Testing, Security Testing, Installa-
tion Testing, and the appropriately named Smoke Test.
With an understanding of the mechanics of testing, attention is directed in
Chapter 10 to the testing environment, identifying both good and bad environments.
Then, Chapter 11 describes the important topic of automated test tools, and Chapter 12
explains how to analyze and interpret test results.
With this foundation laid, Chapter 13 goes through a Full Software Develop-
ment Lifecycle based on a project performed by the lead author for the State of
Colorado.
The textbook concludes with coverage of Testing Complex Applications in
Chapter 14, and identifi cation of Future Directions of Testing in Chapter 15 that
should prove helpful in considering a software testing career.
LEARNING AIDS
After the introductory chapter, Chapter 2 lays a conceptual foundation of meth-
odologies and tools. This chapter relies heavily on diagrams that serve as
frameworks, helping the reader successfully understand the concepts. Chapters