IEEE Software (July/August)

76 Pages·2002·5.19 MB·English

from the editor
Editor in Chief: Steve McConnell (cid:2) Construx Software (cid:2) [email protected]
The Business of Software
Improvement
T
he return on investment in improved ported improvements in predictability simi-
software practices is well documented. lar to the results in Figure 1.5For a software
In 1994, James Herbsleb reported that products company, what is the business
the average “business value” (roughly value of improving schedule estimation ac-
the same as ROI) for 13 organizations curacy from plus or minus 100 percent to
that took on CMM-SW-based improve- plus or minus 10 percent? What is the value
ment programs was about 5 to 1, with the best of being able to make a commitment to cus-
organizations realizing returns of 9 to 1.1 In tomers six to 12 months in advance of a
1995, Neil C. Olsen reported similar returns scheduled completion date, with high confi-
for organizations that made signif- dence of delivering on that commitment?
icant investments in staffing, train- For a company that develops custom
ing, and work environments.2 In software, what is the business value of being
2000, Capers Jones reported that able to provide a fixed price bid with high
the ROI from process improve- confidence that the project will not signifi-
ment could easily go into double cantly overrun the bid?
digits (meaning returns greater For a retail sales organization, what is the
than 10 to 1).3A recent analysis by value of being able to plan cutover to a new
Watts Humphrey found that the system with pinpoint accuracy? What is the
ROI for improved software prac- value of knowing with confidence that cut-
tices could be in the neighborhood over will occur 1 October, as planned, with
of 4 to 1.4 little risk of overrunning to 15 November or
1 December?
Indirect benefits are even more Unlike the operational benefits that most
significant of the industry literature has focused on,
The ROI figures in the published litera- these indirect benefits open the door to ad-
ture are based on operational savings—that ditional revenue opportunities. These bene-
is, on reducing development cost per line of fits are based not on reducing costs, but on
code written or per function point delivered. increasing access to additional business. For
Although these savings are impressive, the top decision makers in organizations, these
greater business benefit might arise from the indirect benefits are often more compelling
significant indirect returns that arise from than the direct, operational benefits.
improved software practices. Better soft-
ware practices improve predictability of Organizational challenge
costs and schedules, reduce risk of cost and Considering the strong—even com-
schedule overruns, provide early warning of pelling—case for improving software prac-
problems, and support better management. tices, it might seem surprising that some or-
Many organizations that have focused on ganizations have not made a commitment to
improving their software practices have re- use best practices. I have recently been think-
Copyright © 2002 Steven C. McConnell. All Rights Reserved. July/August 2002 IEEE SOFTWARE 5
FROM THE EDITOR
600%
DEPARTMENT EDITORS
Bookshelf: Warren Keuffel, e 500%
[email protected] g
a
C{oAnnsdtyru, cDtPaiovrnaeg:} @mAnpadrtaiycg HmPuraontgitc rapanrmodmg Dreaarmvse,m Tehro.cmomas, percentesults 400% Predictability Predictability
CountryD Reespigonrt::d MD.meaeorptiitenrna Fd@oracw oMlmeor,pi tTuraht,eo rLu.uogcrhgetnWt Toerckhsn,ologies sults as a stimated r 320000%% impbreofvoermeent impraofvteerment
[email protected] ee
Loyal Opposition: Robert Glass, Computing Trends, al rof
u 100%
[email protected] ct
A
Manager: Don Reifer, Reifer Consultants,
[email protected] 0%
1 2 3
Quality Time: Jeffrey Voas, Cigital,
[email protected] SW-CMM level
STAFF
Senior Lead Editor
Figure 1. Project performance
Dale C. Strok
[email protected] ing a lot about why best practices compared to estimated
Group Managing Editor aren’t used. Several factors seem to be performance.5 This example
Crystal Chweh in play. demonstrates different projects
Associate Editors First, there is a basic technology in the US Air Force.
Jenny Ferrero, Shani Murray, and Dennis Taylor
transfer issue. Many software devel-
Staff Editors
Scott L. Andresen and Kathy Clark-Fisher opment best practices have been
Editorial Assistants available for decades, but only a few
Rebecca Deuel and Ty Manuel companies use them, and undergrad-
Magazine Assistants uate programs have not generally dustry. Companies didn’t need to focus
Dawn Craig, [email protected]
taught these best practices. The on operational improvements because
Pauline Hosillos
scarcity of experienced users of these that would have shifted too much fo-
Art Director
Toni Van Buskirk practices limits the rate at which cur- cus away from generating revenue. For
Cover Illustration rent users can train new users. Al- a time, improved software practices
Dirk Hagner
though a person might reasonably seemed to be more of a distraction
Technical Illustrator
assume that the average software or- than a help.
Alex Torres
Production Assistant ganization’s capability is halfway be- A final factor is that many organi-
Monette Velasco tween the worst organization’s capa- zations push responsibility for soft-
Production Artists bility and the best’s, in reality, the ware development improvement
Carmen Flores-Garvey and Larry Bauer
average software organization’s prac- down to the project level. In review-
Executive Director
David Hennage tices are much closer to the worst or- ing the “effort multiplier” factors in
Publisher ganization’s practices than the best’s. the Cocomo II estimation model,7 I
Angela Burgess
The result is that software developers was struck by how few of the factors
Assistant Publisher
who work in average organiza- are under the control of an individual
Dick Price
tions—which includes most develop- project manager. Of the 22 factors
Membership/Circulation Marketing Manager
Georgann Carter ers—have never seen a really well- Cocomo II uses to fine-tune a pro-
Advertising Assistant run software project, much less a ject’s base effort estimate, in my judg-
Debbie Sims
really well-run software organiza- ment only three are typically under
CONTRIBUTING EDITORS tion. The software industry faces the the individual project manager’s con-
Greg Goth, Keri Schreiner, problem of bootstrapping best prac- trol: documentation, architecture
Joan Taylor, and Margaret Weatherford
tices into common usage because of and risk resolution, and development
limited current usage of them. for reuse. Numerous factors are dic-
Editorial:All submissions are subject to editing for clarity,
A second factor is that recent eco- tated by the nature of the company’s
style, and space. Unless otherwise stated, bylined articles
and departments, as well as product and service descrip- nomic circumstances have prevented business—product complexity, re-
tions, reflect the author’s or firm’s opinion. Inclusion in
IEEE Softwaredoes not necessarily constitute endorsement software organizations from feeling quired reliability, platform volatility,
by the IEEE or the IEEE Computer Society.
any strong imperative to switch to bet- unprecedentedness of the software,
To Submit:Send 2 electronic versions (1 word-processed
and 1 postscript or PDF) of articles to Magazine Assistant, ter practices.6 Throughout the 1990s, and so on. A company cannot easily
IEEESoftware, 10662 Los Vaqueros Circle, PO Box 3014,
software-related companies rode a change these factors without chang-
Los Alamitos, CA 90720-1314; [email protected]. Ar-
ticles must be original and not exceed 5,400 words including technology wave that rewarded com- ing businesses. The remaining fac-
figures and tables, which count for 200 words each.
panies just for being in the software in- tors—staff capability, multisite devel-
6 IEEE SOFTWARE July/August 2002
FROM THE EDITOR EDITOR IN CHIEF:
Steve McConnell
10662 Los Vaqueros Circle
Los Alamitos, CA 90720-1314
opment, personnel continuity, pro- hear how they can help improve soft- [email protected]
cess maturity, and so on—can be in- ware projects. EDITOR IN CHIEF EMERITUS:
Alan M. Davis, Univ. of Colorado
fluenced by the organization but not What have you done to educate
at Colorado Springs
by individual projects. executives about better software
practices? What has worked well for ASSOCIATE EDITORS IN CHIEF
What can you do? you? I’d love to hear your comments Design: Maarten Boasson, Quaerendo Invenietis
We could hope that upper man- at [email protected]. [email protected]
Construction: Terry Bollinger, Mitre Corp.
agement, sales, and marketing staff
[email protected]
would read every issue of IEEE Soft- Requirements: Christof Ebert, Alcatel Telecom
[email protected]
ware cover to cover or educate them-
Management: Ann Miller, University of Missouri, Rolla
selves about the finer nuances of soft- [email protected]
ware development some other way. Quality: Jeffrey Voas, Cigital
[email protected]
But this isn’t likely to happen, so Experience Reports: Wolfgang Strigel,
leading software practitioners have References Software Productivity Center; [email protected]
an ongoing responsibility: the educa- 1. J. Herbsleb et al., Benefits of CMM Based
Software Process Improvement: Initial EDITORIAL BOARD
tion of nontechnical software project
Results, tech. report CMU/SEI-94-TR-13,
Don Bagert, Texas Tech University
stakeholders. Software practitioners Software Eng. Inst., Carnegie Mellon Univ.,
Richard Fairley, Oregon Graduate Institute
Pittsburgh, 1994.
sometimes perceive upper manage- Martin Fowler, ThoughtWorks
2. N.C. Olsen, “Survival of the Fastest: Improv- Robert Glass, Computing Trends
ment and other nontechnical staff to ing Service Velocity,” IEEE Software, vol. 12,
Andy Hunt, Pragmatic Programmers
be blocking the use of better prac- no. 5, Sept./Oct. 1995, pp. 28–38. Warren Keuffel, independent consultant
tices. We complain that they fail to 3. C. Jones, Software Assessments, Benchmarks, Brian Lawrence, Coyote Valley Software
and Best Practices, Addison Wesley, Reading, Karen Mackey, Cisco Systems
support better practices or even un- Mass., 2000. Deependra Moitra, Lucent Technologies, India
dermine them. I’ve generally found, 4. W. Humphrey, Winning with Software: An Don Reifer, Reifer Consultants
however, that upper management, Executive Strategy, Addison Wesley, Reading, Suzanne Robertson, Atlantic Systems Guild
Mass., 2001. Dave Thomas, Pragmatic Programmers
sales, marketing, product support, 5. P.K. Lawlis, R.M. Flowe, and J.B. Thordahl,
and other personnel are receptive to “A Correlational Study of the CMM and Soft-
INDUSTRY ADVISORY BOARD
ware Development Performance,” Crosstalk,
improved software practices when I
Sept. 1995. Robert Cochran, Catalyst Software (chair)
take the time to explain those prac- 6. S. McConnell, After the Gold Rush, Microsoft Annie Kuntzmann-Combelles, Q-Labs
tices to them. Indeed, they are Press, Redmond, Wash., 1999. Enrique Draier, PSINet
7. B. Boehm et al., Software Cost Estimation Eric Horvitz, Microsoft Research
acutely aware of the problems caused with Cocomo II, Addison Wesley, Reading, David Hsiao, Cisco Systems
by current practices and are eager to Mass., 2000. Takaya Ishida, Mitsubishi Electric Corp.
Dehua Ju, ASTI Shanghai
Donna Kasperson, Science Applications International
Pavle Knaflic, Hermes SoftLab
Wojtek Kozaczynski, Rational Software Corp.
Tomoo Matsubara, Matsubara Consulting
Call for Articles: Industry Experience Reports Masao Matsumoto, Univ. of Tsukuba
Dorothy McKinney, Lockheed Martin Space Systems
Nancy Mead, Software Engineering Institute
Stephen Mellor, Project Technology
Experience reports with lessons learned in industry are one way to Susan Mickel, AgileTV
Dave Moore, Vulcan Northwest
share successes or failures with others who likely face similar situa-
Melissa Murphy, Sandia National Laboratories
tions. We invite you to report on a technology or process you intro- Kiyoh Nakamura, Fujitsu
duced in your company, analyze its impact, and explore what you Grant Rule, Software Measurement Services
Girish Seshagiri, Advanced Information Services
would do differently the next time.
Chandra Shekaran, Microsoft
Martyn Thomas, Praxis
Rob Thomsett, The Thomsett Company
Articles should be 2,000 to 2,400 words with each illustration
John Vu, The Boeing Company
counting as 200 words. We also encourage you to submit up to 10 Simon Wright, Integrated Chipware
short bullet points on lessons learned and references to related Web Tsuneo Yamaura, Hitachi Software Engineering
sites. Submissions are reviewed by members of our Industry Advisory
MAGAZINE OPERATIONS COMMITTEE
Board and are subject to editing for style, clarity, and space.
George Cybenko (chair), JamesH. Aylor, Thomas J.
Bergin, Frank Ferrante, Forouzan Golshani, Rajesh
For detailed author guidelines, see computer.org/software/genres. Gupta, Steve McConnell, Ken Sakamura, M. Satya-
narayanan, Nigel Shadbolt, Munindar P. Singh,
htm#IndustryExperienceReport or contact [email protected]. Francis Sullivan, James J. Thomas
Submissions are accepted at any time.
PUBLICATIONS BOARD
Associate Editor: Wolfgang B. Strigel, Software Productivity Cen- Rangachar Kasturi (chair), Jean Bacon, Mark Chris-
tensen, George Cybenko, Gabriella Sannitti di Baja,
ter, [email protected]. Lee Giles, Thomas Keefe, Dick Kemmerer,
Anand Tripathi
July/August 2002 IEEE SOFTWARE 7
in the news
Features Editor: Scott L. Andresen (cid:2) [email protected]
Will the Cyber-UL
Concept Take Hold?
Greg Goth, [email protected]
O
ne of the most talked-about concepts in don’t know about. Would this null and void
network security—the idea of an Un- the UL? That’s the more challenging part.”
derwriters’ Laboratories-type seal of Yet many vendors and the public sector
approval for hardware and software— people in charge of the new effort believe it
is about to take full flight. Beginning 1 to be a critical step in providing incentive to
July, new products deemed to be part design and build better products.
of systems with US national security implica- “In reality, there is no such thing as a se-
tions must be certified under one of several in- cure system,” says Ron Ross, director of the
formation assurance schemes and tested by in- National Information Assurance Partner-
dependent laboratories working in ship, which oversees the new certification
concert with colleagues from the program. The NIAP is administered jointly
Whether the
federal government. by the National Institute for Standards and
new standards
However, whether the new Technology (NIST) and the National Secu-
will usher in standards will usher in an era of rity Agency. “What we’re talking about is
consistency and assurance for sys- putting in as much security and getting as
a new era of
tems developers and customers is much assurance about the system as we can
consistency is uncertain. While many vendors possibly afford, given our respective mis-
and security experts applaud the sions. Everybody’s mission is different,
uncertain.
effort as an important—and everybody’s perception of how much risk
workable—step in the quest for a they can handle is different. It’s the product
more secure Internet, others think evaluation that gives you more information
the concept is paradoxically simplistic and and confidence about what goes into the
confusing. system, and that’s a good starting point.”
“I think the idea of a UL-type approval is
very noble, but I think it will be very difficult Common Criteria for the common
to implement at this point,” says Marty Lind- good
ner, team leader of incident handling at the The linchpin of the new security and as-
Software Engineering Institute’s CERT Coor- surance effort is product certification under
dination Center. “Even if you did the UL stuff one of two standards: the Common Crite-
at the design phase, you’re only certifying ria, a graduated set of assurance levels rec-
that a perfect implementation of the code is ognized by 14 nations, or the Federal Infor-
believed to be secure. If you take what has mation Processing Standard (FIPS)-140.
been built and test that, you’re testing it Under the guidelines of a federal policy
based on what we know current vulnerabili- known as NSTISSP 11, federal purchasers
ties or vectors are. For example, buffer over- of network components that have national
flow would be a vector. That isn’t to say security implications must ensure these
somebody comes up with a clever way of do- products have received certification under
ing something six months from now that we either of these standards. Vendors say the
12 IEEE SOFTWARE July/August 2002 0740-7459/02/$17.00 © 2002 IEEE
IN THE NEWS
Common Criteria’s international
scope will strengthen the market.
“The Common Criteria is the big
one for a couple of reasons,” says
Mary Ann Davidson, chief security
officer at Oracle. “First of all, it’s an
ISO standard, so that carries some
weight. Second of all, we used to do
country-specific evaluations, which
were really expensive. We did ITSEC
[the old Western European stan-
dard], the Orange Book [the US De-
fense Department standard], and the
Russian criteria.
“Through the mutual recognition
provision of the Common Criteria, if
you get evaluation up to a certain
level, everyone says ‘That’s good
enough for me,’ as opposed to some-
thing developed by a US-specific
body, which may have other agendas.
This is a true international standard.”
Davidson testified before a US
House of Representatives Armed Ser-
vices Committee subcommittee that
the mandated security evaluations
will provide three main benefits:
(cid:2) A more secure product, through down the pike, it would be a multiplat- defines the scope of certification. We
the steps of the evaluation form, multirelease problem. That pays thought about including some of the
(cid:2) A secure development process, for the cost of the evaluation.” APIs that belong to Microsoft, for
through the evaluators’ review of An IBM product—a cryptographic example, which the lab said would
product security architecture, security chip—was the first to receive take years.”
functional, design, and test specifi- Common Criteria approval in No- Meeting Common Criteria specifi-
cations, which ensures a secure de- vember 2001. Ernie Ovies, product cations could offer an opportunity
velopment process is repeatable manager for the chip, says the effort for vendors of specialized compo-
(cid:2) A culture of security, through the will give customers assurance the nents entree into larger systems if
completion of multiple evalua- product does what it says. they cannot afford to develop and
tions, that makes security part of “When we put this chip on the evaluate them.
the corporate “DNA” over the motherboard, we wanted to make “Hardware, particularly for cryp-
long term sure our customers knew it had been tographic mechanisms, is rightly per-
verified independently. I don’t think ceived as more secure, so it has a
“I like that the Common Criteria it’s lip service. I think it’s going to be higher assurance level,” Davidson
forces a secure development process,” with us for a long time.” says. “The other nice thing about it is
Davidson says. “It’s not so much slap- Ovies estimated the process, from that if you have a card or chip with a
ping it on at the end as bells and whis- conceiving the chip’s design to receiv- cryptographic mechanism, it doesn’t
tles. Jumping through this hoop at the ing certification, took about 18 mean the software vendor is off the
end of your development process just months. During the design and eval- hook. It means you can plug some-
isn’t good enough. uation phases, IBM and the chip’s thing into your system that does
“It doesn’t mean you don’t have se- manufacturer, Atmel, worked closely some of the cryptographic mecha-
curity vulnerabilities, but you at least with the approved testing lab, Cy- nism and get a level of assurance
have to develop a process where people gnaCom. This collaboration might from that.”
have to think about security. Another have saved the designers much time The Common Criteria’s graduated
plus is that if the evaluators find a big and effort. levels of testing are called Evaluation
honking security hole, they won’t give “We developed a security target, Assurance Levels. Those deemed
you the seal of approval until you fix it. defining what part of it had the secu- most workable for commercial prod-
If you found one of those five years rity function in it,” Ovies says. “This ucts are Levels One through Four,
July/August 2002 IEEE SOFTWARE 13
IN THE NEWS
with the following requirements for abilities. Development environmen- NIAP Director Ross says the grad-
reaching each level as stated in Com- tal controls and TOE configuration uated Common Criteria standards
mon Criteria documentation: management are also required. are a significant advance over the
(cid:2) EAL4 (methodically designed, tested, previous Orange Book standards.
(cid:2) EAL1 (functionally tested) is applic- and reviewed) permits a developer “The Orange Book was fairly rigid in
able where some confidence in cor- to maximize assurance gained structure and was primarily devel-
rect operation is required but the from positive security engineering oped to define requirements for gen-
threats to security are not consid- based on good commercial devel- eral-purpose operating systems and
ered to be serious. Its value comes opment practices. Although rigor- the development of those systems.
when independent assurance is re- ous, these practices do not require The Common Criteria gives us great
quired to verify due that care was substantial specialist knowledge, flexibility to define requirements for
exercised with respect to the protec- skills, or other resources. EAL4 is security in key technologies like op-
tion of personal or similar informa- the highest level at which it is eco- erating systems, databases, firewalls,
tion. This level provides an evalua- nomically feasible to retrofit an ex- biometrics, and smart cards.”
tion of the Target of Evaluation isting product line. It is applicable Currently, there are seven accred-
(TOE) as made available to the con- in those circumstances where de- ited testing labs in the US, three in
sumer, including independent test- velopers or users require a moder- Canada and Germany, four in the
ing against a specification and an ate to high level of independently United Kingdom and France, and
examination of the guidance docu- assured security in conventional two in Australia.
mentation provided. commodity TOE, and are pre- Ross doesn’t foresee a land rush in
(cid:2) EAL2 (structurally tested) requires pared to incur additional security- labs asking to be certified.
the developer’s cooperation in specific engineering costs. An “I don’t know how many labs we
terms of design information and EAL4 evaluation provides an can sustain. I know NSTISSP 11 is dri-
test result delivery but should not analysis supported by the low-level ving a lot of companies to get their
demand more effort on the devel- design of the TOE’s modules, and product evaluated, so there may be a
oper’s part than is consistent with a subset of the implementation. large ramp-up because of that, but at
good commercial practice. As such, Testing is supported by an inde- some point we’ll reach a steady state in
it should not require a substantially pendent search for vulnerabilities. supply and demand in terms of the
increased investment of cost or Development controls are sup- evaluation process,” he says. “Some
time. EAL2 is applicable in those ported by a life-cycle model, iden- labs may get in the business and find
circumstances where developers or tification of tools, and automated there’s not enough work. Others may
users require a low to moderate configuration management. be overwhelmed. A lot of labs, though,
level of independently assured se- could use their NIAP approval as an
curity in the absence of the com- entree to do some consulting. That’s
plete development record. Such a allowed as long as they don’t evaluate
situation can arise when securing anything they’ve worked on. There are
legacy systems or where access to strict guidelines to protect against con-
the developer is limited. flict of interest.”
(cid:2) EAL3 (methodically tested and
checked) permits a conscientious Testing the testers
developer to gain maximum assur- “It takes about a Laboratories must submit to a
ance from positive security engi- three-step process to become accred-
year to become
neering at the design stage without ited. Those steps include
substantial alteration of existing accredited from
sound development practices. It is (cid:2) Developing a quality manual that
the time the lab
applicable in those circumstances will become the testing bible for
first submits a
where developers or users require the lab, pursuant to ISO standards
a moderate level of independently letter stating its (cid:2) Completing a proficiency test ad-
assured security and a thorough ministered by NIAP assessors
interest to the NIAP
investigation of the TOE and its de- (cid:2) An onsite assessment of the lab by
velopment without incurring sub- to the time examiners working for the Na-
stantial reengineering costs. An tional Voluntary Laboratory Ac-
it actually gets
EAL3 evaluation provides an analy- creditation Program, the NIST of-
sis supported by “gray box” testing, its certification.” fice responsible for the process
selective confirmation of the devel-
oper test results, and evidence of a “It takes about a year to become
developer search for obvious vulner- accredited from the time the lab first
14 IEEE SOFTWARE July/August 2002
IN THE NEWS
submits a letter stating its interest to
Useful URLs
the NIAP to the time it actually gets its
certification,” Ross says. “It takes a
long time to get a quality manual to-
Common Criteria: www.commoncriteria.org
gether if you don’t have one. The pro-
CERT Coordination Center: www.cert.org/nav/index_main.html
ficiency tests are fairly extensive. And
NIST’s Common Criteria: http://csrc.nist.gov/cc
the labs have ongoing business, so a
lot of the accreditation process takes NIST’s NVLAP: http://ts.nist.gov/ts/htdocs/210/214/214.htm
time away from normal business.”
Ken Kolstad is director of opera-
tions for InfoGard Laboratories, in
San Luis Obispo, California. Info- When they get into systems certifica- ing,” says cryptography expert Bruce
Gard, which was founded in 1993, tion, it will be another big step.” Schneier, cofounder of Counterpane
was the first lab in the US to become Internet Security. “And even worse,
FIPS certified but is still awaiting its Ball of confusion? the Common Criteria doesn’t even
Common Criteria accreditation. The likelihood that complex sys- mandate anything. It’s just a frame-
Kolstad says an early survey of its tems will eventually require some work. I can define a protection scheme
customers met with lukewarm en- sort of evaluation and certification of ‘Don’t bother with any security’ and
thusiasm for Common Criteria eval- might be the biggest area of uncer- then accurately claim that my system
uations, so InfoGard put its applica- tainty in the entire concept. For ex- conforms to the Common Criteria.”
tion on hold. A combination of ample, if a defense contractor clerk The NIAP’s Ross concedes there
events in the autumn of 2001, how- sends a federal agency counterpart a will be confusion and overlap.
ever, led those customers to recon- spreadsheet breakdown of costs for a “The danger is that the average per-
sider, and InfoGard is once again in vehicle part, will the spreadsheet and son doesn’t understand the difference
the accreditation process. email applications on their desktops between the Common Criteria, the
“Business has been swamped since need certification? If so, under which CIS, and CERT,” he says. “These pro-
9/11,” Kolstad says, explaining the program? Common Criteria? Or, grams do different things. The Com-
surge in security matters. “Our cus- perhaps, benchmarks released by the mon Criteria is a much more technical
tomer base is suddenly seeing a de- nonprofit Center for Internet Secu- evaluation. It looks at the internals of
mand overseas for Common Criteria- rity, or an evaluation done under these products, the way they’re de-
certified products. There’s also a big CERT’s Survivable Systems Analysis signed and developed, and the sophis-
demand for FIPS overseas, so that method? tication of the software development
business has increased substantially.” “This will all be incredibly confus- techniques that were used to build
Booz Allen Hamilton received its them. The CIS benchmarks look at
laboratory accreditation in March. what these products do once they’re
Steve Rome, recently named lab di- configured—when they come out of
rector, says customer interest in re- the box.
ceiving Common Criteria certification “You can spend a lot of time and
is steadily increasing. Joe Mahaffee, effort on a Common Criteria evalua-
who oversees the lab’s NIAP pro- tion and if that product is configured
gram, says interest is not confined to improperly or used by people who
“Our customer base is
companies within the US, but it is go- don’t understand it, you may just as
suddenly seeing a
ing to market its niche closer to well never have done it. So, I look at
home, focusing on products under demand overseas for these programs as being kind of com-
consideration for use by the federal plementary,” Ross says. “We all
Common Criteria-
government. know what each of us does, and I
Rome believes the future of certifi- certified products. think we need to do a better job of
cation will be centered not on indi- telling people. When I go out and talk
There’s also a big
vidual products but on systems. about the Common Criteria, I should
“I think the 1 July deadline shows demand for FIPS also talk about these other programs
the government is serious about mak- overseas, so that and how they relate to ours. Some
ing sure its acquisitions are products people might say it gives the other
business has increased
we know something about,” he says. guys a competitive advantage. I say it
“It’s not going to change the world. substantially.” doesn’t. We’re trying to position our-
The attitude of the NSA has always selves in the marketplace. We all have
been incremental steps toward adding a niche, and that niche must be made
security, and this is a good first one. very clear.”
July/August 2002 IEEE SOFTWARE 15
manager
Editor: Donald J. Reifer (cid:2) Reifer Consultants (cid:2) [email protected]
How Good Are Agile
Methods?
T
he software industry seems to be em- Table 1 summarizes the demographics of
bracing yet another change to the way the 32 organizations, representing 28 firms,
it does business. Because of their em- that responded (several large firms had more
phasis on agility and time-to-market, than one organization trying to use agile
many programming shops are moving techniques). To transfer a technology, these
to agile methods. Unlike more tradi- firms use it on a pilot to prove to themselves
tional approaches, these methods focus on that it works, use it on a pathfinder to de-
generating early releases of working products termine how to integrate the technology
using mostly collaborative tech- with their processes, and then move it onto
niques such as pair programming, production projects. As expected, five of the
refactoring, and having customers 14 firms that responded are involved in e-
work on site as team members. commerce and e-business applications. The
Programmers use these releases— information these early adopters supplied
which are working products, not gives us insight into how to tap the power of
prototypes—to demonstrate fea- these emerging practices.
tures and functions to stakehold- The 14 firms using agile methods cited a
ers involved in their use, market- laundry list of practices as agile: collective
ing, and support. ownership, concurrent development, contin-
This article surveys the experi- uous integration, customer collaboration,
ence software engineers in a wide daily standup meetings, product demos in-
range of industries have had in deploying stead of documents, Extreme Programming
agile methods. (XP), frequent product releases, full stake-
holder participation, individuals and interac-
The survey tions, just-in-time requirements, metaphors
Fads come and go, in software engineering instead of architectures, nightly product
as in everything else. Practitioners want to builds, pair programming, rapid application
know if agile methods are real or just more development, refactoring, retrospectives, sto-
hype. To answer that question, I surveyed 10 ries for requirements, team programming,
industry segments using the approach illus- and test-driven development.
trated in Figure 1. I designed the survey to The database’s 31 projects showed that
those firms pursuing agile methods were
(cid:2) Determine what practices early adopters motivated because they had a poor record of
of agile methods are using delivering acceptable products to market on
(cid:2) Assess the scope and conditions govern- time and within budget. Most projects were
ing their use relatively small (typically fewer than 10 par-
(cid:2) Evaluate the costs and benefits associated ticipants) and were pursued as pilots or
with their use pathfinders. All projects were in-house de-
16 IEEE SOFTWARE July/August 2002 0740-7459/02/$17.00 © 2002 IEEE
MANAGER
velopments (as opposed to con-
tracted out), lasting one year or less Interest IEEE Software/XP Agile Universe
and involving low-risk methods. Literature search
Furthermore, the firms character-
1 Firm up 5 Publish results
ized their projects as having stable re-
survey goals and conclusions
quirements, established architectures,
and a high degree of development flex- Goals Findings
ibility. Products under development
2 Develop 4 Develop
were mostly quick-to-market applica-
survey instrument findings
tions (generally Web-based and client-
Information from
server oriented). Teams were cohesive Questionnaire questionnaires and interviews
and staffed with motivated, experi-
Mailing 3 Canvas broad
enced performers, most of whom were lists spectrum of industry Interviews
relatively young and thus perhaps
more open to new ideas. Although
there was some skepticism, most prac- Figure 1. Survey approach showing steps taken to access industry
titioners involved with agile methods response to XP methods.
were enthusiastic about the prospects.
Although software engineers in incremental, or similar methods,as well tions even though their pro-cesses
the various industries differed on as on how informal or flexible the were mature. Also, most of the orga-
what constituted best agile practices, process should be. The engineers in nizations trying agile methods were
invariably they agreed that a project’s different industries disagreed on who modifying their processes to incorpo-
process must be cyclical and involve the stakeholders were and how deep rate those that worked into their way
builds and increments done in par- their involvement shouldbe. Opinions of doing business.
allel. Furthermore, they said, these differed as well on what practices fell
projects must involve collaborative or- under the category of agile meth- The results so far
ganizations that include participation ods—Extreme Programming, rapid In summarizing the results either
by all stakeholders during develop- application development, team pro- measured or observed by these early
ment. These projects always included gramming, and so on. adopter organizations, seven of the
full-time participation by customers or The biggest surprise was that most 14 organizations that used agile
users while the work was being done, responding organizations were at methods captured hard cost, produc-
rather than relying on reviews, and re- Level 2 or greater under the Software tivity, and quality data. Five of these
sulted in working product demos, not Capability Maturity Model (see Table had benchmarks that they could use
documents or prototypes that are of- 2). For the most part, these advanced for comparisons. Hard data included
ten thrown away. organizations were willing to try some-
Differences arose in the actual form thing new because they were having (cid:2) Productivity improvement: 15 to
of the process used, such as spiral, problems meeting delivery expecta- 23 percent average gain based on
published industry benchmarks.2
(cid:2) Cost reduction: 5 to 7 percent on
Table 1
average based on published indus-
try benchmarks.2
Characteristics of responding firms
(cid:2) Time-to-market compression: 25
Firms using Year State Average size
to 50 percent less time compared
Industry agile methods Projects first tried of progress (KESLOC)*
to previous projects in participat-
Aerospace 1 1 2001 Pathfinder 23
ing firms.
Computer 2 3 2000 Pilot 32
(cid:2) Quality improvement: Five firms
Consultants 1 2 2000 Pilot 25
had data showing that their defect
E-business 5 15 2000 Production 33
rates were on par with their other
Researchers 1 1 2000 Pilot 12
projects when products or appli-
Scientific 0 0 2001 Pilot N/A
cations were released.
Software 2 4 2000 Production 25
Telecom 2 5 2000 Production 42
These numbers normalize contribu-
Total 14 31 Average 31.8
tions of all participating firms inde-
*KESLOC = thousand equivalent source lines of code computed using formulas that normalize reused pendent of their CMM levels.
and modified code in terms of new lines of code (see Barry Boehm’s discussion of the mathematical ap- In addition, the seven organiza-
proach involved).1 tions that didn’t capture hard data
used soft data to justify their move to
July/August 2002 IEEE SOFTWARE 17
MANAGER
they must be considered separately.
Survey Summary and Recommendations For example, as we all well know, de-
creasing cost by accepting reduced
Questions asked and responses quality can accelerate schedule but
result in lost market share. And in-
What do users think agile methods are?
creasing productivity could increase
(cid:2) Devised list of variants and invariants based on user perceptions, not on a
a company’s cost as software staff is
search.
busily producing the wrong product.
Who’s using agile methods?
In such cases, rework increases as
(cid:2) Small, in-house teams developing software for quick-to-market applications.
does schedule.1
Do they provide added value?
(cid:2) Although reports from the field were positive, the sample was too small to
make any broad conclusions.
I
What are the issues plaguing users?
n the “Survey summary and recom-
(cid:2) Most issues revolve around classical problems in managing technology change.
mendations” sidebar, I’ve organized
my findings by the questions the
Recommendations
survey sought to answer. Recommen-
(cid:2) Clearly define what “agile methods” means. dations are aimed at addressing key is-
(cid:2) Build a business case for agile methods using “hard” data to justify the move. sues identified by early adopters. I
(cid:2) When adopting agile methods, recognize that you are changing the way hope this initial report from the field
your organization does business. on agile methods prompts others to
(cid:2) Provide those moving to agile methods with support for making the transition. put their experiences in the public do-
Support should include startup guidelines, “how to” checklists, and measure- main. I am currently preparing a pa-
ment wizards; a knowledge base of past experience accessible by all; and per detailing this survey’s findings for
education and training, including distance education and self-study courses. presentation at XP Agile Universe. If
you’re interested in this topic but can-
not attend, contact me for a copy of
agile methods. Most used some form dition, the “hard” data gathered that paper.
of survey to capture stakeholder might be tainted by the Hawthorne ef-
opinions, and all used recruitment, fect common in efforts of this type,
morale, and other intangibles to build which relates to the small sample size.
a case for trying and retaining agile (These were small, low-risk projects References
methods. All argued passionately for staffed by select teams under con- 1. D.J. Reifer, Making the Software Business
Care: Improvements by the Numbers, Addi-
continued use of agile methods based trolled situations, so the results might
son-Wesley, Reading, Mass., 2002.
on qualitative factors, and all pressed neither scale to larger projects nor re- 2. D.J. Reifer, “Let the Numbers Do the Talk-
for help in resolving the issues that re- flect higher-risk situations.) We will ing,” Crosstalk, Mar. 2002, pp. 4–8.
volved around technology transfer. just have to see if agile methods can
In any case, the jury is still out be- scale to address larger efforts.
cause the sample size (14 organiza- The percentages I’ve cited can be Donald J. Reiferis president of Reifer Consultants and a vis-
iting associate at the Center for Software Engineering at the Univer-
tions and 31 projects) is just too small deceptive. Although cost, schedule,
sity of Southern California. Contact him at [email protected]. He’d
to derive any firm conclusions. In ad- productivity, and quality are related, like to hear from you.
Table 2
Software CMM ratings of responding firms
Industry Number of agile projects Level 1 Level 2 Level 3 Level 4 Level 5 No Rating
Aerospace 1 1
Computer 3 3
Consultants 2 1 1
E-business 15 6 1 8
Researchers 1 1
Scientific 0
Software 4 2 2
Telecom 5 2 2 1
Totals 31 12 6 4 1 8
18 IEEE SOFTWARE July/August 2002