Software Testing PDF
Preview Software Testing
Software Testing Fourth Edition Software Testing A Craftsman’s Approach Paul C. Jorgensen www.auerbach-publications.com K15906 cvr mech.indd 1 9/19/13 10:04 AM Software Testing A Craftsman’s Approach Fourth Edition Software Testing A Craftsman’s Approach Fourth Edition Paul C. Jorgensen CRC Press Taylor & Francis Group 6000 Broken Sound Parkway NW, Suite 300 Boca Raton, FL 33487-2742 © 2014 by Taylor & Francis Group, LLC CRC Press is an imprint of Taylor & Francis Group, an Informa business No claim to original U.S. Government works Version Date: 20130815 International Standard Book Number-13: 978-1-4665-6069-7 (eBook - PDF) This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot assume responsibility for the valid- ity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may rectify in any future reprint. Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or uti- lized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopy- ing, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers. For permission to photocopy or use material electronically from this work, please access www.copyright.com (http:// www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged. Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. Visit the Taylor & Francis Web site at http://www.taylorandfrancis.com and the CRC Press Web site at http://www.crcpress.com To Carol, Kirsten, and Katia vii Contents Preface to the Fourth Edition.............................................................................................xix Preface to the Third Edition ...............................................................................................xxi Preface to the Second Edition ......................................................................................... xxiii Preface to the First Edition ...............................................................................................xxv Author .............................................................................................................................xxvii Abstract ............................................................................................................................xxix part I a MatHEMatICaL CONtEXt 1 A Perspective on Testing ...............................................................................................3 1.1 Basic Definitions ..................................................................................................... 3 1.2 Test Cases ................................................................................................................ 4 1.3 Insights from a Venn Diagram ................................................................................ 5 1.4 Identifying Test Cases ............................................................................................. 6 1.4.1 Specification-Based Testing ........................................................................ 7 1.4.2 Code-Based Testing .................................................................................... 8 1.4.3 Specification-Based versus Code-Based Debate .......................................... 8 1.5 Fault Taxonomies .................................................................................................... 9 1.6 Levels of Testing .................................................................................................... 12 References .........................................................................................................................13 2 Examples ....................................................................................................................15 2.1 Generalized Pseudocode .........................................................................................15 2.2 The Triangle Problem .............................................................................................17 2.2.1 Problem Statement.....................................................................................17 2.2.2 Discussion .................................................................................................18 2.2.3 Traditional Implementation .......................................................................18 2.2.4 Structured Implementations ......................................................................21 2.3 The NextDate Function ......................................................................................... 23 2.3.1 Problem Statement.................................................................................... 23 2.3.2 Discussion ................................................................................................ 23 2.3.3 Implementations ....................................................................................... 24 viii ◾ Contents 2.4 The Commission Problem ..................................................................................... 26 2.4.1 Problem Statement.................................................................................... 26 2.4.2 Discussion ................................................................................................ 27 2.4.3 Implementation ........................................................................................ 27 2.5 The SATM System ................................................................................................. 28 2.5.1 Problem Statement.................................................................................... 29 2.5.2 Discussion ................................................................................................ 30 2.6 The Currency Converter ........................................................................................ 30 2.7 Saturn Windshield Wiper Controller ......................................................................31 2.8 Garage Door Opener ..............................................................................................31 References .........................................................................................................................33 3 Discrete Math for Testers ...........................................................................................35 3.1 Set Theory ..............................................................................................................35 3.1.1 Set Membership ........................................................................................ 36 3.1.2 Set Definition ........................................................................................... 36 3.1.3 The Empty Set .......................................................................................... 37 3.1.4 Venn Diagrams ......................................................................................... 37 3.1.5 Set Operations .......................................................................................... 38 3.1.6 Set Relations ............................................................................................. 40 3.1.7 Set Partitions ............................................................................................ 40 3.1.8 Set Identities ..............................................................................................41 3.2 Functions ............................................................................................................... 42 3.2.1 Domain and Range .................................................................................. 42 3.2.2 Function Types ......................................................................................... 43 3.2.3 Function Composition.............................................................................. 44 3.3 Relations .................................................................................................................45 3.3.1 Relations among Sets .................................................................................45 3.3.2 Relations on a Single Set ........................................................................... 46 3.4 Propositional Logic .................................................................................................47 3.4.1 Logical Operators ..................................................................................... 48 3.4.2 Logical Expressions................................................................................... 49 3.4.3 Logical Equivalence .................................................................................. 49 3.5 Probability Theory ................................................................................................. 50 Reference ..........................................................................................................................52 4 Graph Theory for Testers ............................................................................................53 4.1 Graphs ....................................................................................................................53 4.1.1 Degree of a Node ...................................................................................... 54 4.1.2 Incidence Matrices .....................................................................................55 4.1.3 Adjacency Matrices ................................................................................... 56 4.1.4 Paths ......................................................................................................... 56 4.1.5 Connectedness ...........................................................................................57 4.1.6 Condensation Graphs ............................................................................... 58 4.1.7 Cyclomatic Number ................................................................................. 58 4.2 Directed Graphs .....................................................................................................59 Contents ◾ ix 4.2.1 Indegrees and Outdegrees ......................................................................... 60 4.2.2 Types of Nodes ......................................................................................... 60 4.2.3 Adjacency Matrix of a Directed Graph ......................................................61 4.2.4 Paths and Semipaths ................................................................................. 62 4.2.5 Reachability Matrix .................................................................................. 62 4.2.6 n-Connectedness ...................................................................................... 63 4.2.7 Strong Components .................................................................................. 64 4.3 Graphs for Testing ..................................................................................................65 4.3.1 Program Graphs ........................................................................................65 4.3.2 Finite State Machines ............................................................................... 66 4.3.3 Petri Nets ................................................................................................. 68 4.3.4 Event-Driven Petri Nets ............................................................................ 70 4.3.5 StateCharts ............................................................................................... 73 References ........................................................................................................................ 75 part II UNIt tEStING 5 Boundary Value Testing .............................................................................................79 5.1 Normal Boundary Value Testing ........................................................................... 80 5.1.1 Generalizing Boundary Value Analysis ......................................................81 5.1.2 Limitations of Boundary Value Analysis ................................................... 82 5.2 Robust Boundary Value Testing ............................................................................ 82 5.3 Worst-Case Boundary Value Testing ..................................................................... 83 5.4 Special Value Testing ............................................................................................. 84 5.5 Examples ................................................................................................................85 5.5.1 Test Cases for the Triangle Problem ..........................................................85 5.5.2 Test Cases for the NextDate Function ...................................................... 86 5.5.3 Test Cases for the Commission Problem ....................................................91 5.6 Random Testing .................................................................................................... 93 5.7 Guidelines for Boundary Value Testing ................................................................. 94 6 Equivalence Class Testing ..........................................................................................99 6.1 Equivalence Classes ............................................................................................... 99 6.2 Traditional Equivalence Class Testing ................................................................. 100 6.3 Improved Equivalence Class Testing ....................................................................101 6.3.1 Weak Normal Equivalence Class Testing ................................................102 6.3.2 Strong Normal Equivalence Class Testing ...............................................102 6.3.3 Weak Robust Equivalence Class Testing..................................................103 6.3.4 Strong Robust Equivalence Class Testing ................................................104 6.4 Equivalence Class Test Cases for the Triangle Problem ........................................105 6.5 Equivalence Class Test Cases for the NextDate Function .....................................107 6.5.1 Equivalence Class Test Cases ...................................................................109 6.6 Equivalence Class Test Cases for the Commission Problem..................................111 6.7 Edge Testing .........................................................................................................113 6.8 Guidelines and Observations ................................................................................113 References .......................................................................................................................115 x ◾ Contents 7 Decision Table–Based Testing ..................................................................................117 7.1 Decision Tables .....................................................................................................117 7.2 Decision Table Techniques ...................................................................................118 7.3 Test Cases for the Triangle Problem .................................................................... 122 7.4 Test Cases for the NextDate Function ................................................................. 123 7.4.1 First Try ................................................................................................. 123 7.4.2 Second Try ............................................................................................. 124 7.4.3 Third Try ................................................................................................ 126 7.5 Test Cases for the Commission Problem .............................................................. 127 7.6 Cause-and-Effect Graphing ................................................................................. 128 7.7 Guidelines and Observations ............................................................................... 130 References .......................................................................................................................131 8 Path Testing ..............................................................................................................133 8.1 Program Graphs ...................................................................................................133 8.1.1 Style Choices for Program Graphs ...........................................................133 8.2 DD-Paths ............................................................................................................ 136 8.3 Test Coverage Metrics ..........................................................................................138 8.3.1 Program Graph–Based Coverage Metrics ................................................138 8.3.2 E.F. Miller’s Coverage Metrics .................................................................139 8.3.2.1 Statement Testing ....................................................................139 8.3.2.2 DD-Path Testing .....................................................................140 8.3.2.3 Simple Loop Coverage .............................................................140 8.3.2.4 Predicate Outcome Testing ......................................................140 8.3.2.5 Dependent Pairs of DD-Paths .................................................141 8.3.2.6 Complex Loop Coverage .........................................................141 8.3.2.7 Multiple Condition Coverage ..................................................142 8.3.2.8 “Statistically Significant” Coverage ..........................................142 8.3.2.9 All Possible Paths Coverage .....................................................142 8.3.3 A Closer Look at Compound Conditions ................................................142 8.3.3.1 Boolean Expression (per Chilenski) .........................................142 8.3.3.2 Condition (per Chilenski) .......................................................143 8.3.3.3 Coupled Conditions (per Chilenski) ........................................143 8.3.3.4 Masking Conditions (per Chilenski) .......................................144 8.3.3.5 Modified Condition Decision Coverage ..................................144 8.3.4 Examples .................................................................................................145 8.3.4.1 Condition with Two Simple Conditions ..................................145 8.3.4.2 Compound Condition from NextDate ....................................146 8.3.4.3 Compound Condition from the Triangle Program ..................147 8.3.5 Test Coverage Analyzers ..........................................................................149 8.4 Basis Path Testing .................................................................................................149 8.4.1 McCabe’s Basis Path Method ..................................................................150 8.4.2 Observations on McCabe’s Basis Path Method ........................................152 8.4.3 Essential Complexity ...............................................................................154 8.5 Guidelines and Observations ................................................................................156 References .......................................................................................................................158 Contents ◾ xi 9 Data Flow Testing ....................................................................................................159 9.1 Define/Use Testing ...............................................................................................160 9.1.1 Example ...................................................................................................161 9.1.2 Du-paths for Stocks .................................................................................164 9.1.3 Du-paths for Locks ..................................................................................164 9.1.4 Du-paths for totalLocks ..........................................................................168 9.1.5 Du-paths for Sales ...................................................................................169 9.1.6 Du-paths for Commission .......................................................................170 9.1.7 Define/Use Test Coverage Metrics ..........................................................170 9.1.8 Define/Use Testing for Object-Oriented Code ........................................172 9.2 Slice-Based Testing ...............................................................................................172 9.2.1 Example ...................................................................................................175 9.2.2 Style and Technique ................................................................................179 9.2.3 Slice Splicing ...........................................................................................181 9.3 Program Slicing Tools ...........................................................................................182 References .......................................................................................................................183 10 Retrospective on Unit Testing ..................................................................................185 10.1 The Test Method Pendulum .................................................................................186 10.2 Traversing the Pendulum ......................................................................................188 10.3 Evaluating Test Methods ......................................................................................193 10.4 Insurance Premium Case Study ............................................................................195 10.4.1 Specification-Based Testing .....................................................................195 10.4.2 Code-Based Testing .................................................................................199 10.4.2.1 Path-Based Testing ..................................................................199 10.4.2.2 Data Flow Testing .................................................................. 200 10.4.2.3 Slice Testing ............................................................................201 10.5 Guidelines ........................................................................................................... 202 References ...................................................................................................................... 203 part III BEYOND UNIt tEStING 11 Life Cycle–Based Testing .........................................................................................207 11.1 Traditional Waterfall Testing ............................................................................... 207 11.1.1 Waterfall Testing .................................................................................... 209 11.1.2 Pros and Cons of the Waterfall Model .................................................... 209 11.2 Testing in Iterative Life Cycles ..............................................................................210 11.2.1 Waterfall Spin-Offs .................................................................................210 11.2.2 Specification-Based Life Cycle Models .....................................................212 11.3 Agile Testing ........................................................................................................214 11.3.1 Extreme Programming ............................................................................215 11.3.2 Test-Driven Development ........................................................................215 11.3.3 Scrum ......................................................................................................216 11.4 Agile Model–Driven Development .......................................................................218 11.4.1 Agile Model–Driven Development ..........................................................218 11.4.2 Model–Driven Agile Development ..........................................................218 References .......................................................................................................................219 xii ◾ Contents 12 Model-Based Testing ................................................................................................221 12.1 Testing Based on Models ......................................................................................221 12.2 Appropriate Models ............................................................................................. 222 12.2.1 Peterson’s Lattice .................................................................................... 222 12.2.2 Expressive Capabilities of Mainline Models ............................................ 224 12.2.3 Modeling Issues ...................................................................................... 224 12.2.4 Making Appropriate Choices .................................................................. 225 12.3 Commercial Tool Support for Model-Based Testing ............................................ 226 References ...................................................................................................................... 227 13 Integration Testing ...................................................................................................229 13.1 Decomposition-Based Integration ....................................................................... 229 13.1.1 Top–Down Integration .......................................................................... 232 13.1.2 Bottom–Up Integration .......................................................................... 234 13.1.3 Sandwich Integration ...............................................................................235 13.1.4 Pros and Cons .........................................................................................235 13.2 Call Graph–Based Integration ............................................................................. 236 13.2.1 Pairwise Integration ................................................................................ 237 13.2.2 Neighborhood Integration ...................................................................... 237 13.2.3 Pros and Cons ........................................................................................ 240 13.3 Path-Based Integration .........................................................................................241 13.3.1 New and Extended Concepts...................................................................241 13.3.2 MM-Path Complexity ............................................................................ 243 13.3.3 Pros and Cons ........................................................................................ 244 13.4 Example: integrationNextDate ............................................................................ 244 13.4.1 Decomposition-Based Integration ...........................................................245 13.4.2 Call Graph–Based Integration .................................................................245 13.4.3 MM-Path-Based Integration ....................................................................250 13.5 Conclusions and Recommendations .....................................................................250 References .......................................................................................................................251 14 System Testing ..........................................................................................................253 14.1 Threads .................................................................................................................253 14.1.1 Thread Possibilities ................................................................................. 254 14.1.2 Thread Definitions...................................................................................255 14.2 Basis Concepts for Requirements Specification .....................................................256 14.2.1 Data.........................................................................................................256 14.2.2 Actions ....................................................................................................257 14.2.3 Devices ....................................................................................................257 14.2.4 Events ......................................................................................................258 14.2.5 Threads ....................................................................................................259 14.2.6 Relationships among Basis Concepts .......................................................259 14.3 Model-Based Threads ...........................................................................................259 14.4 Use Case–Based Threads ..................................................................................... 264 14.4.1 Levels of Use Cases ................................................................................. 264 14.4.2 An Industrial Test Execution System .......................................................265 14.4.3 System-Level Test Cases ......................................................................... 268 Contents ◾ xiii 14.4.4 Converting Use Cases to Event-Driven Petri Nets .................................. 269 14.4.5 Converting Finite State Machines to Event-Driven Petri Nets .................270 14.4.6 Which View Best Serves System Testing? ................................................271 14.5 Long versus Short Use Cases ................................................................................271 14.6 How Many Use Cases? .........................................................................................274 14.6.1 Incidence with Input Events ....................................................................274 14.6.2 Incidence with Output Events .................................................................275 14.6.3 Incidence with All Port Events ................................................................ 277 14.6.4 Incidence with Classes ............................................................................ 277 14.7 Coverage Metrics for System Testing ................................................................... 277 14.7.1 Model-Based System Test Coverage ........................................................ 277 14.7.2 Specification-Based System Test Coverage .............................................. 278 14.7.2.1 Event-Based Thread Testing .................................................... 278 14.7.2.2 Port-Based Thread Testing ...................................................... 279 14.8 Supplemental Approaches to System Testing ....................................................... 279 14.8.1 Operational Profiles ................................................................................ 279 14.8.2 Risk-Based Testing ................................................................................. 282 14.9 Nonfunctional System Testing ............................................................................ 284 14.9.1 Stress Testing Strategies .......................................................................... 284 14.9.1.1 Compression ........................................................................... 284 14.9.1.2 Replication ............................................................................. 285 14.9.2 Mathematical Approaches ...................................................................... 286 14.9.2.1 Queuing Theory ..................................................................... 286 14.9.2.2 Reliability Models .................................................................. 286 14.9.2.3 Monte Carlo Testing............................................................... 286 14.10 Atomic System Function Testing Example .......................................................... 287 14.10.1 Identifying Input and Output Events ..................................................... 289 14.10.2 Identifying Atomic System Functions ..................................................... 290 14.10.3 Revised Atomic System Functions ...........................................................291 References ...................................................................................................................... 292 15 Object-Oriented Testing ...........................................................................................295 15.1 Issues in Testing Object-Oriented Software ......................................................... 295 15.1.1 Units for Object-Oriented Testing .......................................................... 295 15.1.2 Implications of Composition and Encapsulation .................................... 296 15.1.3 Implications of Inheritance ..................................................................... 297 15.1.4 Implications of Polymorphism ................................................................ 299 15.1.5 Levels of Object-Oriented Testing .......................................................... 299 15.1.6 Data Flow Testing for Object-Oriented Software ................................... 299 15.2 Example: ooNextDate ......................................................................................... 300 15.2.1 Class: CalendarUnit ................................................................................301 15.2.2 Class: testIt ............................................................................................. 302 15.2.3 Class: Date ............................................................................................. 302 15.2.4 Class: Day............................................................................................... 303 15.2.5 Class: Month .......................................................................................... 303 15.2.6 Class: Year .............................................................................................. 304 15.3 Object-Oriented Unit Testing.............................................................................. 304 xiv ◾ Contents 15.3.1 Methods as Units .................................................................................... 305 15.3.2 Classes as Units ...................................................................................... 305 15.3.2.1 Pseudocode for Windshield Wiper Class ................................ 306 15.3.2.2 Unit Testing for Windshield Wiper Class ............................... 306 15.4 Object-Oriented Integration Testing ....................................................................311 15.4.1 UML Support for Integration Testing .....................................................311 15.4.2 MM-Paths for Object-Oriented Software ................................................313 15.4.3 A Framework for Object-Oriented Data Flow Testing .............................318 15.4.3.1 Event-/Message-Driven Petri Nets ...........................................318 15.4.3.2 Inheritance-Induced Data Flow .............................................. 320 15.4.3.3 Message-Induced Data Flow ................................................... 320 15.4.3.4 Slices? ......................................................................................321 15.5 Object-Oriented System Testing ...........................................................................321 15.5.1 Currency Converter UML Description ...................................................321 15.5.1.1 Problem Statement...................................................................321 15.5.1.2 System Functions .....................................................................321 15.5.1.3 Presentation Layer .................................................................. 322 15.5.1.4 High-Level Use Cases ............................................................. 322 15.5.1.5 Essential Use Cases ................................................................. 323 15.5.1.6 Detailed GUI Definition .........................................................325 15.5.1.7 Expanded Essential Use Cases ................................................ 326 15.5.1.8 Real Use Cases ....................................................................... 328 15.5.2 UML-Based System Testing ................................................................... 328 15.5.3 StateChart-Based System Testing ............................................................329 References ...................................................................................................................... 330 16 Software Complexity ................................................................................................331 16.1 Unit-Level Complexity .........................................................................................331 16.1.1 Cyclomatic Complexity ...........................................................................332 16.1.1.1 “Cattle Pens” and Cyclomatic Complexity ...............................332 16.1.1.2 Node Outdegrees and Cyclomatic Complexity ........................332 16.1.1.3 Decisional Complexity ........................................................... 334 16.1.2 Computational Complexity .....................................................................335 16.1.2.1 Halstead’s Metrics ...................................................................335 16.1.2.2 Example: Day of Week with Zeller’s Congruence ................... 336 16.2 Integration-Level Complexity ...............................................................................338 16.2.1 Integration-Level Cyclomatic Complexity ...............................................339 16.2.2 Message Traffic Complexity .................................................................... 340 16.3 Software Complexity Example ............................................................................ 341 16.3.1 Unit-Level Cyclomatic Complexity ......................................................... 344 16.3.2 Message Integration-Level Cyclomatic Complexity ................................ 344 16.4 Object-Oriented Complexity ............................................................................... 344 16.4.1 WMC—Weighted Methods per Class .................................................... 344 16.4.2 DIT—Depth of Inheritance Tree ............................................................345 16.4.3 NOC—Number of Child Classes ...........................................................345 16.4.4 CBO—Coupling between Classes ..........................................................345 16.4.5 RFC—Response for Class .......................................................................345
The list of books you might like
