Organizational patterns of agile software development

488 Pages·2004·10.06 MB·English

by  James O. Coplien

Preface
We are satisfied by doing real work. Software is like a plant that
grows: You can’t predict its exact shape, or how big it will grow;
you can control its growth only to a limited degree. There are no
rules for this kind of thing—it’s never been done before.
- Charlie Anderson, Architect, Borland Quattro Pro for Win-
dows.
You will find no books on the bookshelf here that tell you how to
start up a new discipline. Software has been seeking its own way as a
relatively young discipline for the past 40 years. Every new discipline
struggles to find practices suitable to its survival and growth. Some-
times this struggle is incremental. Sometimes disciplines undergo
more substantial shifts in process, structure and values that break
more with the past to explore new ground. What Charlie Anderson
said above about the Borland Quattro Pro for Windows effort in partic-
ular applies to the rhythms of software development in general. Get
ready for change, for it will come tomorrow.
The most exciting advances in science go hand in hand with radical
social change. The move from classic physics to quantum physics pre-
1
2 Chapter
cipitated from a crisis in physics. We talk about the software crisis, yet
no individual crisis in software — let alone the Software Crisis, what-
ever that might be — has precipitated the same kinds of change that
we associate with great advances in science. Software development
has perhaps yet to face its first true crisis that leads to the first true
industry-wide systemic change.
But that doesn’t mean that software is static. We can identify dif-
ferent faces of change in software development over the past five
decades. Our interest in this book is what software development has
learned about itself from an organizational and social perspective.
Software development is perhaps working in its fourth social style of
system development. Yet what is really interesting about these social
styles is their ties to technical advances in the art. The first style of soft-
ware development goes back to the first computers that were pro-
grammed manually with console switches. The second style came
with the advent of programming languages that allowed scientists to
work individually or in small teams, interacting with the machine
through a language. In the third style, what we learned from hardware
design and manufacturing carried over into software. Formal pro-
cesses drove development, management was visible and explicit, and
both the system and the organizations that worked on the system were
highly hierarchical. Now we are in the fourth style: one that breaks
down hierarchy, that features dynamic social structures and communi-
cation paths, and that values immediacy. This fourth style often bears
the label “agile,” but that is just one of many characterizations of a
broad new way of developing software that has emerged over the past
decade.
Yet human endeavors tend to take the same shape century after cen-
tury, and the fact that human endeavors all have the common element
of human nature bounds the range of human undertakings. For
example, most organizations have leaders, and cliques, and their own
little rituals, their own nuances of meaning for terms of the trade, a
correspondence between physical space and organizational structure,
and hundreds more. The organization of any major human endeavor
follows basic laws of efficiency of communication, span of control,
xenophobia, specialization, and other sociological forces that drive
most large-scale projects to similar practices and structures. Much has
been made of the similarity between vernacular housing architecture
and the construction of software systems. [CoplienDevos2000]. The
3
same may be true for the organization of any social animals, but we
leave verification of that claim to readers better tooled in those disci-
plines than we are.
Going deeper yet, the systems of nature have common rhythms and
trends that underlie their emergent complexity. These properties come
from the structure of the organization: the deeply held relationships
that define the organization as a social entity. Senge [Senge1990] popu-
larized this aspect of social behavior in a discipline he called systems
thinking, a discipline that goes beyond our everyday disciplines that
are based on a simplistic relationship between cause and effect. Many
early attempts at software process improvement relied on this sim-
plistic cause-and-effect relationship, particularly as the third paradigm
of software development started to take hold in the industry. Most ISO
9000 process improvement efforts were run this way: find what we’re
doing wrong, find the place in the process that’s wrong, and make it
right. There was rarely any notion that the process as a whole might be
wrong and that, for example, a step-by-step process should be
replaced by a more reactive, agile process. And it was heresy to conjec-
ture that process itself might be the wrong formalism to capture the
crucial properties of efficient, effective systems.
When we started this work in the early 1990s, we started with docu-
mented research that showed serious shortcomings in process-based
approaches such as ISO 9000. We asked: if process isn’t the answer,
then what is? We chose organizational structure—and, in particular,
the structure of relationships between roles—as the basis for system
understanding. All systems are about relationships, and most disci-
plines that study systems study the relationships in those systems. The
idea worked. What’s better, the technique didn’t displace the process-
based approaches in existing organizations (it’s always hard to tell an
organization to stop doing something they think is helpful, anyhow),
but complemented them by adding insight into deep structure that
explained behavior at the process level. So if you already have a pro-
cess improvement program in place, this book can add an enriching
dimension that builds on your own culture and helps develop your
peoples’ insights into that culture.
Patterns provide a way to capture both the broad, invariant prac-
tices of socially built artifacts as well as the specialized practices of
individual disciplines, along with an understanding of how those
practices build on each other. Long before Alexander started using pat-
4 Chapter
terns for the field of architecture, anthropologists were using them to
describe human social structures [Kroeber1948]. The pattern lan-
guages in this book combine the timeless human structures that tran-
scend disciplines with the best practices of contemporary software
development. These patterns are all empirical: they capture the major
rhythms and structures of successful software-intensive organizations
today. Many of the patterns come from our own research, but we also
incorporated patterns from other authors working in the same field.
This is a collected work and in many ways reflects a community-wide
effort.
There are two equally valid views of this book: as a guide to organi-
zational improvement, and as a record of the “best typical” software
development structures of the fourth social paradigm of software
development. Most readers will use the book in the first sense. How-
ever, it is our hope that this book reflects a well-enough grounded
view of contemporary software development to serve as a touchstone
that records what life was like in software development organizations
in the late twentieth and early twenty-first centuries. Fifty years from
now, will this book be a sobering admonishment to the industry? Will
we just chuckle at how things used to be? Or will time leave this
fourth-generation culmination of software progress largely
untouched? In that spirit, we greet the rare reader who finds an
archival copy of this tome on a dusty bookshelf in the middle of the
twenty-first century, and we salute your efforts to use this under-
standing to further improve the lot of the organizations of your time
and place.
Our sense of history extends in the other direction as well. Christo-
pher Alexander’s book includes a picture at the beginning of every
pattern [Alexander1977]. Each picture sets a broad tone for the pattern
that follows. We wanted that same feeling for this book, and we strove
to include a picture for each pattern. We initially felt that the social net-
work diagrams would suffice as pictures, but that gave the book an
“academic” feel that left a bad taste in our mouths. Paul Bramble
turned us on to the prints and photographs division of the Library of
Congress (http://lcweb.loc.gov/rr/print/catalog.html), which pro-
vided a wealth of vintage photos. Most come from a collection of
depression-era photographs sponsored by the Farm Services Adminis-
tration. Some of the photos are strikingly poignant or relevant to the
patterns; that was a surprise, since they come from an era that predates
5
the software culture that is such a large part of this book. But we feel
that the age and human element of the photos lends an overall charm
to the book that totally would be lacking in the social network dia-
grams. They also give us a feel of the timelessness of the basic human
issues facing organizations of any culture, era, or ideology. This is not
a book about ideology, but about human nature. Last, the pictures
might help make the patterns more memorable for you: pictures are
powerful association tools.
Many of the pictures have a military theme. Please remember that
the purpose of patterns is human quality of life and comfort, and that
patterns help us capture as much learning from history’s tragedies as
from its moments of peak culture. Also remember that the earliest pat-
terns of human organizations (such as [Clavell1989]) have roots in mil-
itary organizational structure.
Acknowledgments
In doing this book we view ourselves as editors and chroniclers of
others’ work and ideas. There are literally thousands of people who
contributed to this book through their participation in the empirical
studies from which we mined the patterns. We don’t have all of their
names here, but we appreciate all of them for their time and energy.
Especially noteworthy were the Borland QPW Group, coordinated by
David Intersimone, a remarkably productive project at AT&T led by
Judy Tschirgi, highly effective projects at Schlumberger in Oslo, where
we were hosted by Lise Hvatum, and the group managed by Richard
Gabriel at ParcPlace Systems that was undergoing a sobering restruc-
turing at the time we visited Richard.
There are other people who put even more of their own energy into
this book by building things for us and doing things with us. Brendan
Cain was one of the original members of the Pasteur research effort at
Bell Laboratories, and he wrote many of the original analysis tools.
Anthropologist Peter B¸rgi was another early member of the research
team; he contributed many of the insights on schismogenesis and on
other direct parallels between the corporate world and the more “tra-
ditional” world of cultural anthropology. Tom Burrows’ early work on
the GIL and Romana environments at Bell Labs provided a platform
for many of the early tools. We are grateful to Steve North for the dot
tool that not only provides good supporting visuals that map out the
pattern languages, but which was a key research tool in its own right.
6 Chapter
We are particularly indebted to authors who let us reproduce their
patterns here. Pieces of the patterns of Alistair Cockburn, Ward Cun-
ningham, Bruce Whitenack and Steve Berczuk have all made their way
into this collection. Thanks for sharing, folks. Gerard Meszaros wrote
several patterns, including ARTIFACT OWNERSHIP, ARCHITECTURE DEFINI-
TION TEAM, and ARCHITECTURE ORGANIZATION, whose contents filtered
into many patterns in these pattern languages.
Other people steered us in the right direction. Diane Grinnell
pointed us to the references on organizational incest and its parallels to
dysfunctional constellations in family therapy. Tom Stone, then at
Addison-Wesley, pointed us to some dynamite references on organiza-
tional learning, and in particular the studies that came out of Royal
Dutch Shell. Bindu Rama Rao of Lucent pointed us to the work by
Kroeber, which gave us strong ties from patterns back to the world of
anthropology. Urvashi Kaul of Allstate developed pattern taxonomies
that shaped how we organized these patterns into pattern languages.
But most of all, we’re grateful to Moody Ahmad, who first gave us a
hint back in the mid-1980s that software development research should
be investigating not just the technical issues, but the human issues as
well.
Dozens of people reviewed these patterns in writers workshops at
pattern conferences and local pattern groups. Additionally, we enjoyed
the feedback of a team of focused and thorough reviewers who
weren’t afraid to give us the benefit of their opinion in places where
they felt we were misguided, and they were usually right. Gerhard
Ackermann at Siemens in Vienna offered a wealth of first-hand
insights in organizational growth and repair. Paul Bramble and Ian
Graham were our two main manuscript reviewers for the late versions
of the manuscript, but they offered a lot of useful advice on the organi-
zational of the book. Joshua Kerievsky pioneered the conversion of
these patterns to Alexandrian form with his outstanding editorial
efforts on SOLO VIRTUOSO, SIZE THE ORGANIZATION, and SELF SELECTING
TEAM. And many thanks to our favorite Mercenary Analyst, Betsy
Hanes Perry, for her contributions to PUBLIC CHARACTER. Other key
early reviewers included Jay Stagnone, ...
There are others who worked with us as partners along the way,
and whose editorial feedback and suggestions were fundamental to
the shape of the book. Martine Devos (then of Argo in Belgium, and
currently of Avaya Labs) and Steve Berczuk invested much of them-
7
selves in this work, and we are grateful for their energy and dedica-
tion.
We enjoyed a good technical and organizational infrastructure to
support out work. At Bell Labs, the research organizations managed
by Eric Sumner, Mary Zajac, and David Weiss actively supported this
work over a decade. That is a long time not only in Internet years but
even by research project standards. We honor their vision, patience,
and forbearance. David Weiss continued to support this work in a sim-
ilar capacity at Avaya Labs. Many thanks to UNIVERSIT‰T Karlsruhe for
the Wiki that we used to develop the book manuscript, and in partic-
ular to Dr. Helmut Goos. The support came in part under the auspices
of the IST 1999-14191 EasyComp joint research project of the European
Community, and we are grateful for that support. Research Chair John
Roddick and fellow professor Paul Calder sponsored the infrastruc-
ture and time for this work while Jim Coplien spent a summer (or was
it winter?) at Flinders University in Adelaide, Australia.
And of course, where would we be without our editorial support?
Alan Apt has always been there in the wings, but not bugging us too
much, supporting us in mighty ways. It’s been a joy working with
him.
This book was generated automatically from a Wiki web site using
tools that generated a MIF file for transmission to the publisher.
8 Chapter
9
PART I. History And Introduc-
tion
This book is about people — people who write software. No, this
isn’t a Dilbertesque look at our profession or an analysis of the minds
of cult figures in the profession. It is about teams of real people who
write real software. You see, over the last ten years or so, we have
studied how people work together to create software. And we have
seen that these organizations have a lot in common, whether they are
writing software for telephone systems, banks, or oil exploration. The
people issues shine through whatever application they are developing.
And that is comforting, in a way.
At the same time, though, it is disconcerting. For while organiza-
tions are inanimate, they take on a life of their own. We see that organi-
zations grow, learn, and sometimes even get sick! Yet they can heal
themselves, and can become healthy again. Of course, we are most
interested in the characteristics of the healthy organizations; perhaps
other organizations can learn from those experiences.
It is this notion of healing, repair and growth that are the founda-
tions of Agile development. O.K., we’ll be frank: we chose “Agile” for
the title out of marketing concerns. It seems to be the current term of
choice for the kinds of things we describe in this book. It is a term that
rolls off the tongue more easily that other clever names that clamor for
your attention on today’s bookshelves. This manuscript has been
10 Chapter
evolving piecemeal for over a decade, and the early pre-publication
manuscripts have been a foundation and source of inspiration for
many contemporary popular approaches. For example, Jeff Sutherland
notes that an early publication on this work, related to one of the major
case studies in this book, was one early influence on SCRUM
[Sutherland2003]. Ken Schwaber notes that the early background of
these pattern languages “were the genesis of some of the agile pro-
cesses” [Schwaber2003]. Gabriel’s early article [Gabriel1994] and later
book [Gabriel1996] discuss the successful application of these tech-
niques at ParcPlace Systems as a key part of a broader effort that trans-
formed the organization. And these patterns have the dubious
distinction of earning the criticism of one notable software person who
was the primary reviewer of the first published version of these pat-
terns. He noted that anyone who worked on organizational issues was
avoiding doing real work (which by his own admission was limited to
anything directly related to Smalltalk programming). He would later
go on to be one of the founders of Extreme Programming—a discipline
that builds in part on the patterns that have been in this language
(such as DEVELOPING IN PAIRS (4.2.28)) for almost a decade.
Yet this book is broader than so-called agile development. We are
really concerned with effective software development — the ability to
produce good software efficiently, time after time. Many of the organi-
zations we studied and learned from would not be considered “agile”,
but they were highly effective. The 5ESS development in AT&T, for
example, would fit nobody’s definition of agile. But year after year,
they produced software for a system that was not only one of the
largest software systems in the world, but among the world’s most
reliable. Yes, many of these patterns contribute to agility, but our chief
aim is effectiveness.
We have captured the good things organizations do and have
written them down as patterns. We hope these patterns will be as
interesting and useful to you as they have been to us. Many others
have found them useful: at conferences, we find that these patterns are
the foundation of improvement programs in many companies world-
wide.
We have divided this book into four parts:
• Part I: History and Introduction, the section you are reading
• Part II: The patterns themselves
• Part III: Foundations and History