Table Of Content

TEAM LinG Software Testing and Continuous Quality Improvement TEAM LinG Series_A_master Page 1 Friday, January 23, 2004 8:49 AM Other CRC/Auerbach Publications in Software Development, Software Engineering, and Project Management The Complete Project Management Office Six Sigma Software Development Handbook Christine Tanytor Gerard M. Hill 0-8493-1193-4 0-8493-2173-5 Software Architecture Design Patterns Complex IT Project Management: in Java 16 Steps to Success Partha Kuchana Peter Schulte 0-8493-2142-5 0-8493-1932-3 Software Configuration Management Jessica Keyes Creating Components: Object Oriented, 0-8493-1976-5 Concurrent, and Distributed Computing in Java Software Engineering for Image Charles W. Kann Processing 0-8493-1499-2 Phillip A. Laplante 0-8493-1376-7 Dynamic Software Development: Manging Projects in Flux Software Engineering Handbook Timothy Wells Jessica Keyes 0-8493-129-2 0-8493-1479-8 The Hands-On Project Office: Software Engineering Measurement Guaranteeing ROI and On-Time Delivery John C. Munson 0-8493-1503-4 Richard M. Kesner 0-8493-1991-9 Software Engineering Processes: Principles and Applications Interpreting the CMMI®: A Process Yinxu Wang, Graham King, and Saba Zamir Improvement Approach 0-8493-2366-5 Margaret Kulpa and Kent Johnson 0-8493-1654-5 Software Metrics: A Guide to Planning, Analysis, and Application Introduction to Software Engineering C.R. Pandian Ronald Leach 0-8493-1661-8 0-8493-1445-3 Software Testing: A Craftsman’s ISO 9001:2000 for Software and Systems Approach, 2e Providers: An Engineering Approach Paul C. Jorgensen Robert Bamford and William John Deibler II 0-8493-0809-7 0-8493-2063-1 Software Testing and Continuous Quality The Laws of Software Process: Improvement, Second Edition A New Model for the Production William E. Lewis and Management of Software 0-8493-2524-2 Phillip G. Armour IS Management Handbook, 8th Edition 0-8493-1489-5 Carol V. Brown and Heikki Topi, Editors 0-8493-1595-9 Real Process Improvement Using the CMMI® Lightweight Enterprise Architectures Michael West Fenix Theuerkorn 0-8493-2109-3 0-9493-2114-X AUERBACH PUBLICATIONS www.auerbach-publications.com To Order Call: 1-800-272-7737 • Fax: 1-800-374-340T1EAM LinG E-mail: [email protected] Software Testing and Continuous Quality Improvement Second Edition William E. Lewis Gunasekaran Veerapillai, Technical Contributor AUERBACH PUBLICATIONS A CRC Press Company Boca Raton London New York Washington, D.C. TEAM LinG Library of Congress Cataloging-in-Publication Data Lewis, William E. Software testing and continuous quality improvement / William E. Lewis ; Gunasekaran Veerapillai, technical contributor.--2nd ed. p. cm. Includes bibliographical references and index. ISBN 0-8493-2524-2 (alk. paper) 1. Computer software--Testing. 2. Computer software--Quality control. I. Veerapillai, Gunasekaran. II. Title. QA76.76.T48L495 2004 005.1′4--dc22 2004052492 This book contains information obtained from authentic and highly regarded sources. Reprinted material is quoted with permission, and sources are indicated. A wide variety of references are listed. Reasonable efforts have been made to publish reliable data and information, but the author and the publisher cannot assume responsibility for the validity of all materials or for the consequences of their use. Neither this book nor any part may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, microfilming, and recording, or by any information storage or retrieval system, without prior permission in writing from the publisher. The consent of CRC Press LLC does not extend to copying for general distribution, for promotion, for creating new works, or for resale. Specific permission must be obtained in writing from CRC Press LLC for such copying. Direct all inquiries to CRC Press LLC, 2000 N.W. Corporate Blvd., Boca Raton, Florida 33431. Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation, without intent to infringe. Visit the Auerbach Web site at www.auerbach-publications.com © 2005 by CRC Press LLC Auerbach is an imprint of CRC Press LLC No claim to original U.S. Government works International Standard Book Number 0-8493-2524-2 Library of Congress Card Number 2004052492 Printed in the United States of America 1 2 3 4 5 6 7 8 9 0 TEAM LinG About the Authors William E. Lewis holds a B.A. in Mathematics and an M.S. in Operations Research and has 38 years experience in the computer industry. Currently he is the founder, president, and CEO of Smartware Technologies, Inc., a quality assurance consulting firm that specializes in software testing. He is the inventor of Test SmartTM, a patented software testing tool that creates optimized test cases/data based upon the requirements (see www.smart- waretechnologies.com for more information about the author). He is a certified quality analyst (CQA) and certified software test engineer (CSTE) sponsored by the Quality Assurance Institute (QAI) of Orlando, Florida. Over the years, he has presented several papers to con- ferences. In 2004 he presented a paper to QAI’s Annual International Infor- mation Technology Quality Conference, entitled “Cracking the Requirements/ Test Barrier.” He also speaks at meetings of the American Society for Qual- ity and the Association of Information Technology Practitioners. Mr. Lewis was a quality assurance manager for CitiGroup where he managed the testing group, documented all the software testing, quality assurance processes and procedures, actively participated in the CitiGroup CMM effort, and designed numerous WinRunner automation scripts. Mr. Lewis was a senior technology engineer for Technology Builders, Inc. of Atlanta, Georgia, where he trained and consulted in the requirements-based testing area, focusing on leading-edge testing methods and tools. Mr. Lewis was an assistant director with Ernst & Young, LLP, located in Las Colinas, Texas. He joined E & Y in 1994, authoring the company’s software configuration management, software testing, and application evolu- tionary handbooks, and helping to develop the navigator/fusion methodology application improvement route maps. He was the quality assurance manager for several application development projects and has extensive experience in test planning, test design, execution, evaluation, reporting, and automated testing. He was also the director of the ISO initiative, which resulted in ISO9000 international certification for Ernst & Young. TEAM LinG v Software Testing and Continuous Quality Improvement Lewis also worked for the Saudi Arabian Oil Company (Aramco) in Jed- dah, Saudi Arabia, on an overseas contract assignment as a quality assurance consultant. His duties included full integration and system testing, and he served on the automated tool selection committee and made rec- ommendations to management. He also created software testing standards and procedures. In 1998 Lewis retired from IBM after 28 years. His jobs included 12 years as a curriculum/course developer and instructor, and numerous years as a system programmer/analyst and performance analyst. An overseas assignment included service in Seoul, Korea, where he was the software engineering curriculum manager for the Korean Advanced Institute of Science and Technology (KAIST), which is considered the MIT of higher education in Korea. Another assignment was in Toronto, Canada, at IBM Canada’s head- quarters, where he was responsible for upgrading the corporate education program. In addition, he has traveled throughout the United States, Rome, Amsterdam, Southampton, Hong Kong, and Sydney, teaching software development and quality assurance classes with a specialty in software testing. He has also taught at the university level for five years as an adjunct pro- fessor. While so engaged he published a five-book series on computer prob- lem solving. For further information about the training and consulting services pro- vided by Smartware Technologies, Inc., contact: Smartware Technologies, Inc. 2713 Millington Drive Plano, Texas 75093 (972) 985-7546 Gunasekaran Veerapillai, a certified software quality analyst (CSQA), is also a project management professional (PMP) from the PMI USA. After his 15 years of retail banking experience with Canara Bank, India he was manager of the EDP section for 4 years at their IT department in Bangalore. He was in charge of many critical, internal software development, testing, and maintenance projects. He worked as project manager for testing projects with Thinksoft Global Services, a company that specializes in testing in the BFSI sector. Currently Guna is working as project manager in the Testing Center of Excellence of HCL Technologies (www.hcltechnologies.com), a level-5 CMM Company that has partnered itself with major test automation tool vendors such as Mercury Interactive and IBM Rational. Guna has success- fully turned out various testing projects for international bankers such as Citibank, Morgan Stanley, and Discover Financial. He also contributes arti- cles to software testing Web sites such as Sticky Minds. vi TEAM LinG Contents SECTION I SOFTWARE QUALITY IN PERSPECTIVE . . . . . . . . . . . . . . . 1 1 Quality Assurance Framework. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 What Is Quality?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Prevention versus Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Verification versus Validation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Software Quality Assurance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Components of Quality Assurance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Software Testing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Quality Control. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Software Configuration Management . . . . . . . . . . . . . . . . . . . . . . . . . 12 Elements of Software Configuration Management. . . . . . . . . . . . . . . 12 Component Identification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Version Control. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Configuration Building. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Change Control. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Software Quality Assurance Plan. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Steps to Develop and Implement a Software Quality Assurance Plan. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Step 1. Document the Plan. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Step 2. Obtain Management Acceptance . . . . . . . . . . . . . . . . . . . . 18 Step 3. Obtain Development Acceptance. . . . . . . . . . . . . . . . . . . . 18 Step 4. Plan for Implementation of the SQA Plan . . . . . . . . . . . . . 19 Step 5. Execute the SQA Plan. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Quality Standards. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 ISO9000. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Capability Maturity Model (CMM) . . . . . . . . . . . . . . . . . . . . . . . . . 20 Level 1 — Initial. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Level 2 — Repeatable. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Level 3 — Defined. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Level 4 — Managed. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Level 5 — Optimized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 PCMM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 CMMI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Malcom Baldrige National Quality Award . . . . . . . . . . . . . . . . . . . 24 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 TEAM LinG vii Software Testing and Continuous Quality Improvement 2 Overview of Testing Techniques . . . . . . . . . . . . . . . . . . . . . . . . . .29 Black-Box Testing (Functional). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 White-Box Testing (Structural). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Gray-Box Testing (Functional and Structural) . . . . . . . . . . . . . . . . . . 30 Manual versus Automated Testing . . . . . . . . . . . . . . . . . . . . . . . . . 31 Static versus Dynamic Testing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Taxonomy of Software Testing Techniques . . . . . . . . . . . . . . . . . . . . 32 3 Quality through Continuous Improvement Process . . . . . . . . . . .41 Contribution of Edward Deming. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Role of Statistical Methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Cause-and-Effect Diagram. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Flow Chart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Pareto Chart. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Run Chart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Histogram. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Scatter Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Control Chart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Deming’s 14 Quality Principles. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Point 1: Create Constancy of Purpose. . . . . . . . . . . . . . . . . . . . . . . 43 Point 2: Adopt the New Philosophy . . . . . . . . . . . . . . . . . . . . . . . . 44 Point 3: Cease Dependence on Mass Inspection. . . . . . . . . . . . . . 44 Point 4: End the Practice of Awarding Business on Price Tag Alone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 Point 5: Improve Constantly and Forever the System of Production and Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Point 6: Institute Training and Retraining . . . . . . . . . . . . . . . . . . . 45 Point 7: Institute Leadership . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Point 8: Drive Out Fear . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 Point 9: Break Down Barriers between Staff Areas . . . . . . . . . . . . 46 Point 10: Eliminate Slogans, Exhortations, and Targets for the Workforce. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Point 11: Eliminate Numerical Goals. . . . . . . . . . . . . . . . . . . . . . . . 47 Point 12: Remove Barriers to Pride of Workmanship. . . . . . . . . . 47 Point 13: Institute a Vigorous Program of Education and Retraining. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 Point 14: Take Action to Accomplish the Transformation. . . . . . 48 Continuous Improvement through the Plan, Do, Check, Act Process. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 Going around the PDCA Circle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 SECTION II LIFE CYCLE TESTING REVIEW . . . . . . . . . . . . . . . . . . . . .51 4 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .53 Waterfall Development Methodology . . . . . . . . . . . . . . . . . . . . . . . . . 53 Continuous Improvement “Phased” Approach . . . . . . . . . . . . . . . . . 54 viii TEAM LinG Contents Psychology of Life Cycle Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 Software Testing as a Continuous Improvement Process. . . . . . . . . 55 The Testing Bible: Software Test Plan. . . . . . . . . . . . . . . . . . . . . . . . . 58 Major Steps to Develop a Test Plan. . . . . . . . . . . . . . . . . . . . . . . . . . . 60 1. Define the Test Objectives. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 2. Develop the Test Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 3. Define the Test Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4. Develop the Test Specifications . . . . . . . . . . . . . . . . . . . . . . . . . 61 5. Schedule the Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 6. Review and Approve the Test Plan. . . . . . . . . . . . . . . . . . . . . . . 61 Components of a Test Plan. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 Technical Reviews as a Continuous Improvement Process. . . . . . . 61 Motivation for Technical Reviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 Types of Reviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 Structured Walkthroughs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 Inspections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 Participant Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 Steps for an Effective Review. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 1. Plan for the Review Process. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 2. Schedule the Review. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 3. Develop the Review Agenda. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 4. Create a Review Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 5 Verifying the Requirements Phase. . . . . . . . . . . . . . . . . . . . . . . . . 73 Testing the Requirements with Technical Reviews. . . . . . . . . . . . . . 74 Inspections and Walkthroughs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 Checklists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 Methodology Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 Requirements Traceability Matrix. . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 Building the System/Acceptance Test Plan . . . . . . . . . . . . . . . . . . . . 76 6 Verifying the Logical Design Phase . . . . . . . . . . . . . . . . . . . . . . . . 79 Data Model, Process Model, and the Linkage. . . . . . . . . . . . . . . . . . . 79 Testing the Logical Design with Technical Reviews . . . . . . . . . . . . . 80 Refining the System/Acceptance Test Plan . . . . . . . . . . . . . . . . . . . . 81 7 Verifying the Physical Design Phase . . . . . . . . . . . . . . . . . . . . . . . 83 Testing the Physical Design with Technical Reviews . . . . . . . . . . . . 83 Creating Integration Test Cases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 Methodology for Integration Testing. . . . . . . . . . . . . . . . . . . . . . . . . . 85 Step 1: Identify Unit Interfaces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 Step 2: Reconcile Interfaces for Completeness. . . . . . . . . . . . . . . 85 Step 3: Create Integration Test Conditions . . . . . . . . . . . . . . . . . . 86 Step 4: Evaluate the Completeness of Integration Test Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 TEAM LinG ix