Agile Data Warehousing Project Management

357 Pages·2012·9.633 MB·English

CHAPTER
1
What Is Agile Data
Warehousing?
How are agile approaches different than traditional methods?
What does it take to make agile succeed for data warehousing projects?
Where do we have to be careful with this new approach?
Faster, better, cheaper. That’s the promise that agile methods have been delivering
upon during the past decade for general application development. Although they are
an increasingly popular style of programming for transaction-processing or data-
capture applications, they have not been employed nearly as much for data ware-
housing/business intelligence (DWBI) applications. It usually takes a combination
of project disasters and existential threats to inspire corporate DWBI departments to
consider replacing their traditional development methods. Typically, the impetus to
change begins after several projects seriously overrun their go-live dates and/or fore-
casted delivery budgets. Next, the project sponsors in the company’s business depart-
ments start to grumble. “Our corporate data warehousing department is way too slow
and far too expensive.” Eventually, one end-user department finds a way to build an
analytical application using an outside services vendor and reports to the other busi-
ness leads that this new system required a fraction of corporate DWBI’s typical time
and cost brackets. “Should we send all of our business intelligence (BI) work to out-
side vendors?” the business departments begin asking themselves. It does not matter
to them that outside vendors trying to deliver fast frequently neglect to program the
many hidden attributes needed to make a system manageable, scalable, and extensible
over the long run. All the business sponsors saw was fast delivery of what they asked
for. If the business units are frustrated over time and cost, corporate DWBI will sur-
vive only if it finds a way to accelerate its deliveries and lower programming expense,
restoring its rapport with the customer. This type of predicament is exactly why cor-
porate DWBI should consider agile data warehousing techniques.
On the one hand, it is easy to understand why the popularity of agile for data
warehousing lags 10 years behind its usage for general applications. It is hard to
envision delivering any DWBI capabilities quickly. For data capture applications,
creating a new element requires simply creating a column for it in the database
and then dropping an entry field for it on the screen. To deliver a new warehousing
attribute, however, a team has to create several distinct programs to extract, scrub,
Agile Data Warehousing Project Management.
DOI: http://dx.doi.org/10.1016/B978-0-12-396463-2.00001-6 3
© 22001132 Elsevier Inc. All rights reserved.
4 CHAPTER 1 What Is Agile Data Warehousing?
integrate, and dimensionalize the data sets containing the element before it can be
placed on the end user’s console. Compared to the single transaction application
challenge that agile methods originally focused on, data warehousing projects are
trying to deliver a half-dozen new applications at once. They have too many archi-
tectural layers to manage for a team to update the data transform logic quickly in
order to satisfy a program sponsor’s latest functional whim.
On the other hand, data warehousing professionals need to be discussing agile
methods intently, because every year more business intelligence departments large
and small are experimenting with rapid delivery techniques for analytic and report-
ing applications. To succeed, they are adapting the generic agile approaches some-
what, but not beyond recognition. These adaptations make the resulting methods
one notch more complex than agile for transaction-capture systems, but they are no
less effective. In practice, agile methods applied properly to large data integration
and information visualization projects have lowered the development hours needed
and driven coding defects to zero. All this is accomplished while placing a steady
stream of new features before the development team’s business partner. By sav-
ing the customer time and money while steadily delivering increments of business
value, agile methods for BI projects go a long way toward solving the challenges
many DWBI departments have with pleasing their business customers.
For those readers who are new to agile concepts, this chapter begins with a
sketch of the method to be followed throughout most of this book. The next sec-
tions provide a high-level contrast between traditional development methods and
the agile approach, and a listing of the key innovative techniques that give agile
methods much of their delivery speed. After surveying evidence that agile methods
accelerate general application development, the presentation introduces a key set of
adaptations that will make agile a productive approach for data warehousing. Next,
the chapter outlines two fundamental challenges unique to data warehousing that
any development method must address in order to succeed. It then closes with a
guide to the remainder of the book and a second volume that will follow it.
A quick peek at an agile method
The practice of agile data warehousing is the application of several styles of itera-
tive and incremental development to the specific challenges of integrating and
presenting data for analytics and decision support. By adopting techniques such
as colocating programmers together in a single workspace and embedding a busi-
ness representative in the team to guide them, companies can build DWBI appli-
cations without a large portion of the time-consuming procedures and artifacts
typically required by formal software development methods. Working intently on
deliverables without investing time in a full suite of formal specifications neces-
sarily requires that developers focus only on a few deliverables at time. Building
only small pieces at a time, in turn, repeats the delivery process many times. These
repeated deliveries of small scopes place agile methods in the category of “iterative
and incremental development” methods for project management.
A quick peek at an agile method 5
When following agile methods, DWBI developers essentially roll up their sleeves
and work like they have only a few weeks before the system is due. They concentrate
on the most important features first and perform only those activities that directly gener-
ate fully shippable code, thus realizing a tremendous boost in delivery speed. Achieving
breakthrough programming speeds on a BI project will require developers to work dif-
ferently than most of them are trained, including the way they define requirements, esti-
mate work, design and code their systems, and communicate results to stakeholders,
plus the way they test and document the resulting system modules. To make iterative
and incremental delivery work, they will also need to change the physical environment
in which they work and the role of the project manager. Most traditional DWBI depart-
ments will find these changes disorienting for a while, but their disruption will be more
than compensated for by the increased programmer productivity they unleash.
Depending on how one counts, there are at least a dozen agile development styles
to choose from (see sidebar). They differ by the level of ongoing ceremonies they fol-
low during development and the amount of project planning they invest in before cod-
ing begins. By far the most popular flavor of agile is Scrum, first introduced in 1995
by Dr. Jeff Sutherland and Ken Schwaber. [Schwaber 2004] Scrum involves a small
amount of both ceremony and planning, making it fast for teams to learn and easy for
them to follow dependably. It has many other advantages, among them being that it
l Adroitly organizes a team of 6 to 10 developers
l Intuitively synchronizes coding efforts with repeated time boxes
l Embeds a business partner in the team to maximize customer engagement
l Appeals to business partners with its lightweight requirements artifacts
l Double estimates the work for accuracy using two units of measure
l Forecasts overall project duration and cost when necessary
l Includes regular self-optimizing efforts in every time box
l Readily absorbs techniques from other methods
AGILE DEVELOPMENT METHODS
Adaptive [Highsmith 1999]
Crystal [Cockburn 2004]
Disciplined Agile Delivery [Ambler 2012]
Dynamic Systems Development Method (DSDM) [Stapleton 2003]
Extreme Programming (XP) [Beck 2004]
Feature Driven Development (FDD) [Palmer 2002]
Lean Development [Poppendieck 2003]
Kanban [Anderson 2010]
Pragmatic [Hunt 1999]
Scrum [Cohn 2009]
Unified Processes (Essential, Open, Rational, etc.) [Jacobson, Booch, & Rumbaugh
1999]
6 CHAPTER 1 What Is Agile Data Warehousing?
Scrum has such mindshare that, unless one clarifies he is speaking of another
approach, Scrum is generally assumed to be the base method whenever one says
“agile.” Even if that assumption is right, however, the listener still has to interpret
the situation with care. Scrum teams are constantly optimizing their practices and
borrowing techniques from other sources so that they all quickly arrive at their own
particular development method. Over time Scrum teams can vary their practice
considerably, to the point of even dropping a key component or two such as the
time box. Given this diversity in implementations, this book refers to Scrum when
speaking of the precise method as defined by Sutherland and Schwaber. It employs
the more general term “agile” when the context involves an ongoing project that
may well have started with Scrum but then customized the method to better meet
the situation at hand.
Figure 1.1 depicts the simple, five-step structure of an iteration with which
Scrum teams build their applications. A team of 6 to 10 individuals—including an
embedded partner from the customer organization that will own the applications—
repeats this cycle every 2 to 8 weeks. The next chapter presents the iteration cycle
in detail. Here, the objective is to provide the reader with enough understanding of
an agile approach to contrast it with a traditional method.
As shown in Figure 1.1, a list of requirements drives the Scrum process. Typically
this list is described as a “backlog of user stories.” User stories are single sentences
authored by the business stating one of their functional needs. The embedded busi-
ness partner owns this list, keeping it sorted by each story’s importance to the busi-
ness. With this backlog available, Scrum teams repeatedly pull from the top as many
stories as they can manage in one time box, turning them into shippable software
modules that satisfy the stated needs. In practice, a minority of the stories on a back-
log include nonfunctional features, often stipulated for the application by the project
architect. These “architectural stories” call for reusable submodules and features sup-
porting quality attributes such as performance and scalability. Scrum does not pro-
vide a lot of guidance on where the original backlog of stories comes from. For that
reason, project planners need to situate the Scrum development process in a larger
project life cycle that will provide important engineering and project management
notions such as scope and funding, as well as data and process architecture.
The standard development iteration begins with a story conference where
the developers use a top-down estimating technique using what are called “story
points” to identify the handful of user stories at the top of the project’s backlog that
they can convert into shippable code during the iteration.
Next, the team performs task planning where it decomposes the targeted user
stories into development tasks, this time estimating the work bottom-up in terms of
labor hours in order to confirm that they have not taken on too much work for one
iteration.
After confirming they have targeted just the right amount of work, the team-
mates now dive into the development phase, where they are asked to self-organize
and create over the next couple of weeks the promised enhancement to the applica-
tion, working in the most productive way they can devise. The primary ceremony
A quick peek at an agile method 7
FIGURE 1.1
Structure of Scrum development iteration and duration of its phases.
that Scrum places upon them during this phase is that they check in with each other
in the morning via a short stand-up meeting, that is, it asks them to hold a daily
“scrum.”
At the end of the cycle, the team conducts a user demo where the business part-
ner on the team operates the portions of the application that the developers have just
completed, often with other business stakeholders looking on. For data integration
projects that have not delivered the information yet to a presentation layer, the team
will typically provide a simple front end (perhaps a quickly built, provisional BI
module) so that the business partner can independently explore the newly loaded
data tables. The business partner evaluates the enhanced application by considering
each user story targeted during the story conference, deciding whether the team has
delivered the functionality requested.
Finally, before beginning the cycle anew, the developers meet for a sprint ret-
rospective, where they discuss the good and bad aspects of the development cycle
they just completed and brainstorm new ways to work together during the next
cycle in order to smooth out any rough spots they may have encountered.
8 CHAPTER 1 What Is Agile Data Warehousing?
At this point, the team is ready to start another cycle. These iterations progress
as long as there are user stories on the project’s backlog and the sponsors continue
funding the project. During an iteration’s development phase, the team’s embedded
business partner may well have reshuffled the order of the stories in the backlog,
added some new one, and even discarded others. Such “requirements churn” does
not bother the developers because they are always working within the near-planning
horizon defined by the iteration’s time box. Because Scrum has the developers con-
stantly focused on only the top of the backlog, the business can steer the team in a
completely new direction every few weeks, heading to wherever the project needs to
go next. Such flexibility often makes business partners very fond of Scrum because
it allows the developers from the information technology (IT) department to become
very flexible and responsive.
The “disappointment cycle” of many traditional projects
In contrast to Scrum’s iterative approach to delivering systems, traditional soft-
ware engineering operates on a single-pass model. The most widely cited defini-
tion of this approach can be found in the first half of a 1970 white paper entitled
“Managing the Development of Large Software Systems” by a TRW researcher
named Dr. Winston Royce. [Royce 1970] This paper has been commonly inter-
preted to suggest that, in order to avoid conceptual errors and extensive reprogram-
ming of an application, all requirements should be gathered before design begins,
all design should be completed before programmers begin coding, and the bulk of
coding should be completed before serious testing can get underway. In this pro-
cess, each work phase should “fill up” with specifications before that information
spills over into the next phase—a notion that led many to call this approach a cas-
cade or a “waterfall” method.
Many people describe this waterfall process as the “big design up-front” strategy
because it requires enormous design specifications to be drafted and approved before
programming work can begin. [Ambler 2011] It has also been called a “plan-driven”
or “command and control” approach because the big design results in enormous pro-
ject plans, with possibly thousands of separate tasks, that project managers use to
drive the daily activities of the development teams they command. A further name for
this style of organizing development is the “big bang” approach because all the value
is theoretically dropped upon end users at the conclusion of the project.
Waterfall-style project organization can seem to be a safe approach for large
applications, especially those with multiple, intersecting data layers found in data
warehouses, because the engineers supposedly think out all aspects of the project
thoroughly ahead of time. Scrum, however, simply takes a few requirements off the
top of a list and converts them into code before the next set of features is even con-
sidered. In contrast to a waterfall method, Scrum can seem very tactical and ad hoc.
For these reasons, when software professionals first learn of agile, they often
decide that a waterfall method must be far more robust. Such conclusions are ironic
The “disappointment cycle” of many traditional projects 9
because waterfall methods have had over 40 years to prove themselves, but statis-
tics show that they struggle to deliver applications reliably. The Standish Group’s
seminal “Chaos” reports detailed the software industry’s track record in delivering
large systems using traditional methods. [Standish Group 1999] After surveying
the results of 8380 projects conducted by 365 major America companies, results
revealed that even small projects below $750,000 were unable to deliver applica-
tions on time, on budget, and with all the promised features more than 55% of the
time. As the size of the applications grew, the success rate fell steadily to 25% for
efforts over $3M and down to zero for projects over $10M (1999 dollars).
Data warehousing projects fall easily in the middle to upper reaches of the range
documented in the Standish study. Not surprisingly, they, too, have demonstrated
trouble reliably delivering value under traditional project management methods.
A survey performed in 1994—before agile methods were common—by the data
management industry’s primary trade magazine revealed that its readers’ data ware-
house projects averaged above $12M in cost and failed 65% of the time. [Cited in
Ericson 2006] Such statistics do not indicate that every waterfall-based data ware-
housing project is destined to fail. However, if the approach was as robust as people
often assume, plan-driven, big-bang project methods should have achieved a much
higher success rate in the 40 years since Royce’s paper first defined the approach.
Unfortunately, there is a good reason to believe that waterfalls will remain a
very risky manner in building large systems: the specifications flowing into any
one of its phases will always contain flaws. Being only human, the individuals pre-
paring these enormous artifacts will have less than perfect foresight, especially for
companies situated in a world market that is constantly changing. Moreover, in this
age of increasing global competition, the engineers producing these specifications
are also frequently overworked and given too little time to thoroughly research a
project’s problem domain. In this reality, developers working from these specifica-
tions will steadily encounter the unexpected. Lamentably, waterfall methods con-
tain no back step that allows the developers to substantially revisit requirements
if they encounter a major oversight while coding. Testing, compressed to fit into
the very last phase of development, cannot call for a major reconsideration of a
system’s design. With plan-driven project management, the team is locked into a
schedule. They must hurriedly span the gaps between their specifications and the
actual situation because, according to plan, work must move steadily downstream
so that the project meets its promised delivery date.
Rather than mitigating project risk, the waterfall approach actually inflates it.
Aiming for a big design up front, waterfall processes are heavy with documentation
and encumbered with numerous reviews in an effort to avoid oversights. However,
large specification documents acquire their own inertia by virtue of the long proce-
dures needed to update and reapprove them. With this inertia, errors committed dur-
ing requirements or design phases get “baked into” the application because it is either
too expensive or too late to substantively rework specifications when flaws are dis-
covered in them. The phases cannot validate nor contribute to one another’s success,
making each phase a single point of failure. The entire project will either succeed or
10 CHAPTER 1 What Is Agile Data Warehousing?
fail based on the quality of the requirements package taken in isolation, then the qual-
ity of the design taken alone, and then finally the quality of the coding as a whole.
Because little flaws cannot be simply corrected when they are discovered, their con-
sequences get amplified to where any one of them can threaten the entire undertaking.
In this light, an iterative approach mitigates risk in an important way. Scrum vali-
dates the engineering of each small slice of the project by pushing it through require-
ments, design, coding, and testing before work on the next set of stories from the
backlog begins. Errors detected in any of these engineering steps can provide important
guidance on how the team should pursue the remainder of the project. Scrum practition-
ers note that the method pushes errors to the surface early in the process while there is
still time to correct assumptions and the foundations of the project’s coding. (See, for
example, [Vingrys 2011].) In contrast to the tendency of a waterfall to allow errors to
undermine a whole project, the agile approach works to contain failures to just one pro-
ject slice at a time.
Being large undertakings by nature, data warehousing initiatives pursued with
a single-delivery paradigm demonstrate waterfall’s whole project failure pattern all
too often. A quick sampling of horror stories shared with the author by prospective
customers seeking a better approach to building business intelligence systems illus-
trates the point well:
l A major waste management company complained “We hired one of the world’s
top systems integrators to build our data warehouse. It cost us $2 million and
3 years of time. All we got was one report. Seems like a very expensive report.”
l A Fortune 500 telecommunications company related with alarm “We purchased
a ‘ready-to-use’ enterprise data model from our data warehouse appliance ven-
dor. Once it was in place, their consultants told us they would need 2 years to
customize it for our company and 3 years to populate it with data before we
could build our first dashboard.
l A Fortune 50 pharmaceuticals company lamented “Our electronic data manage-
ment project required 150 people for over 3 years and got so expensive it was
starting to hurt our share price.”
When large, plan-driven projects begin to fail, the managers start taking extraor-
dinary measures to rebalance budgets, deadlines, and intended features. As depicted
in Figure 1.2, such a project starts out with a firm forecast of cost, time, and scope.
As it becomes stressed by nasty surprises, time and cost estimates start to climb,
forcing sponsors to steadily accept less scope in order to keep the project afford-
able. After several such crises and delays, the company is eventually grateful to
push even a small subset of functions into production and declare some level of
victory, although the functionality delivered is far less than promised and the cost
often double or more than planned for.
Data warehousing/business intelligence departments turn this all-too-common
pattern of increasing costs and shrinking deliverables into a cycle of disappointment
because, still believing that a big design up front is the best strategy, they repeat this
basic approach for the next project. They believe the last project failed because they
The “disappointment cycle” of many traditional projects 11
Fewer Features Promised
as project Progresses
Est. After
Crisis 2 V
a
lu
e
o
f F
Original ea
Cost Estimate tu
Est. After re
s
Crisis 1 D
e
liv
e
“Realistic” re
Cost Estimate d
Time
FIGURE 1.2
All too often, waterfall methods yield to the project “disappointment cycle.”
did not manage it carefully enough, and start the next project with even more exten-
sive planning and detailed design.
We can rule out unique occurrences of unusual circumstances as the primary cause
for failed projects such as the ones sketched earlier. Unlike Tolstoy’s unhappy fami-
lies, failed waterfall projects in data warehousing all seem to be miserable for the same
basic reasons. On an average engagement, the project manager specifies work pack-
ages in excruciating detail with 5000 lines or more in the project plan. He directs and
cajoles the developers relentlessly. Everyone works nights and weekends, and still the
project runs months behind and millions over budget, delivering only a fraction of
what was promised. When the after-project, lessons-learned session is convened, par-
ticipants will voice the same rueful observations heard on so many previous efforts:
l We should have done better at gathering requirements.
l The business didn’t know what they wanted because they had never seen a
warehouse before.
l Design took too long, leaving no time for careful coding.
l Conflicting definitions for data elements made designs worthless.
l Coding went too slow so testing got started far too late to catch some crucial
defects.
l Programmers didn’t build what was specified.
l Dirty data created nightmares that undermined our schedule.
l By the time we finally delivered, the business had changed and the customer
wanted something else besides what we built.
12 CHAPTER 1 What Is Agile Data Warehousing?
With each pass through the disappointment cycle, IT and project manage-
ment typically respond to failure by heaping even more specifications, reviews,
and process controls on the development process. The process becomes opaque to
the project sponsor and projects take even longer to deliver, leaving the customer
increasingly alienated from the software creation process, steadily more frustrated
with corporate IT, and wondering ever more intently whether an outside service
vendor might have a better way to run a project.
The waterfall method was, in fact, a mistake
If the plan-driven, single-delivery approach to data warehousing provides such
problematic results, it is natural to ask how the strategy got started and why many
software development organizations continue to advocate it.
Sadly, the waterfall approach became the reigning paradigm only through a
series of mistakes originating with the U.S. military. Toward the end of the Viet
Nam war, the Department of Defense (DOD) was struggling to establish uniform
requirements for software development in order to achieve better results from its
systems integration contractors. The DOD issued a series of process specifications,
including DOD-STD-2167, issued in 1985, which made the single-pass waterfall
method as outlined in Dr. Royce’s white paper the standard approach for procured
systems. [DOD 1985]
As the military’s standard for evaluating contractor development processes,
DOD-STD-2167 proved to be particularly influential and soon transported the
single-pass approach into many further methods descriptions throughout the world.
The large-system contractors naturally trained their developers in the approach,
and these individuals carried 2167’s plan-driven approach to the successive com-
panies they worked for throughout the remainder of their careers. The resulting
waterfall methods proliferated, despite the fact that the military’s experience with
the approach was disappointing to say the least. A 1995 report on failure rates in a
sample of DOD projects revealed that given the $37 billion spent on these projects,
75% of the projects failed or were never used, and only 2% were utilized without
extensive modification. [Jarzombek 99]
Ironically, DOD-STD-2167 can only be viewed as a grievous misreading of
Royce’s paper. In its opening pages, Royce identifies the classic waterfall phases,
but describes them as only the “essential steps common to all computer program
developments, regardless of size or complexity.” In the second half of the docu-
ment, Royce clearly advocates an iterative path for progressing through these pro-
cess steps. Pursuing them sequentially from analysis through coding in a single
pass “is risky and invites failure,” he warned, suggesting that such a method would
work only for “the most uncomplicated projects.”
When interviewed in the mid-1990s, the author of DOD-STD-2167 in fact
admitted that he had been unaware of the notion of incremental delivery when he