Agile data warehousing project management : business intelligence systems using Scrum and XP

You have to make sense of enormous amounts of data, and while the notion of "agile data warehousing" might sound tricky, it can yield as much as a 3-to-1 speed advantage while cutting project costs in half. Bring this highly effective technique to your organization with the wisdom of agile data warehousing expert Ralph Hughes.

Agile Data Warehousing Project Management will give you a thorough introduction to the method as you would practice it in the project room to build a serious "data mart." Regardless of where you are today, this step-by-step implementation guide will prepare you to join or even lead a team in visualizing, building, and validating a single component to an enterprise data warehouse.

Author Notes

Ralph Hughes Chief Systems Architect Ceregenics, Inc. Denver, CO, USA

List of Figures	p. xiii
List of Tables	p. xv
Preface	p. xvii
Author's Bio	p. xxi
Part 1 An Introduction to Iterative Development
Chapter 1 What Is Agile Data Warehousing?	p. 3
A quick peek at an agile method	p. 4
The "disappointment cycle" of many traditional projects	p. 8
The waterfall method was, in fact, a mistake	p. 12
Agile's iterative and incremental delivery alternative	p. 14
Agile as an answer to waterfall's problems	p. 15
Agile methods provide better results	p. 18
Agile for data warehousing	p. 19
Data warehousing entails a "breadth of complexity"	p. 19
Adapted scrum handles the breadth of data warehousing well	p. 20
Managing data warehousing's "depth of complexity"	p. 22
Guide to this book and other materials	p. 26
Simplified treatment of data architecture for book 1	p. 28
Companion web site	p. 29
Where to be cautious with agile data warehousing	p. 30
Summary	p. 31
Chapter 2 Iterative Development in a Nutshell	p. 33
Starter concepts	p. 34
Three nested cycles	p. 35
The release cycle	p. 36
Development and daily cycles	p. 39
Shippable code and the definition of done	p. 40
Time-boxed development	p. 41
Caves and commons	p. 42
Product owners and scrum masters	p. 42
Improved role for the project manager	p. 45
Might a project manager serve as a scrum master?	p. 46
User stories and backlogs	p. 47
Estimating user stories in story points	p. 48
Iteration phase 1: story conferences	p. 50
Iteration phase 2: task planning	p. 52
Basis of estimate cards to escape repeating hard thinking	p. 53
Task planning doublechecks story planning	p. 54
Iteration phase 3: development phase	p. 55
Self-organization	p. 56
Daily scrums	p. 57
Accelerated programming	p. 59
Test-driven development	p. 62
Architectural compliance and "tech debt"	p. 63
Iteration phase 4: user demo	p. 65
Iteration phase 5: sprint retrospectives	p. 67
Retrospectives are vital	p. 70
Close collaboration is essential	p. 72
Selecting the optimal iteration length	p. 73
Nonstandard sprints	p. 74
Sprint 0	p. 75
Where did scrum come from?	p. 77
Distant history	p. 77
Scram emerges	p. 78
Summary	p. 79
Chapter 3 Streamlining Project Management	p. 81
Highly transparent task boards	p. 82
Task boards amplify project quality	p. 84
Task boards naturally integrate team efforts	p. 85
Scrum masters must monitor the task board	p. 86
Burndown charts reveal the team aggregate progress	p. 87
Detecting trouble with burndown charts	p. 89
Developers are not the burndown chart's victims	p. 91
Calculating velocity from burndown charts	p. 92
Common variations on burndown charts	p. 94
Setting capacity when the team delivers early	p. 94
Managing tech debt	p. 95
Managing miditeration scope creep	p. 96
Diagnosing problems with burndown chart patterns	p. 97
An early hill to climb	p. 98
Shallow glide paths	p. 99
Persistent inflation	p. 100
Should you extend a sprint if running late?	p. 102
Extending iterations is generally a bad idea	p. 102
Two instances where a changing time box might help	p. 103
Should teams track actual hours during a sprint?	p. 104
Eliminating hour estimation altogether	p. 105
Managing geographically distributed teams	p. 106
Consider whether fully capable subteams are possible	p. 108
Visualize the problem in terms of communication	p. 108
Choose geographical divisions to minimize the challenge	p. 109
Invest in a solid esprit de corp	p. 109
Provide repeated booster shots of colocation for individuals	p. 110
Invest in high-quality telepresence equipment	p. 110
Provide agile team group ware	p. 112
Summary	p. 112
Part 2 Defining Data Warehousing Projects for Iterative Development
Chapter 4 Authoring Better User Stories	p. 117
Traditional requirements gathering and its discontents	p. 118
Big, careful requirements not a solution	p. 120
A step in the right direction	p. 120
Agile's idea of "user stories"	p. 122
Advantages of user stories	p. 123
Identifying rather than documenting the requirements	p. 124
User story definition fundamentals	p. 125
Quick test for actionable user stories	p. 126
How small is small?	p. 127
Epics, themes, and stories	p. 128
Common techniques for writing good user stories	p. 130
Keep story writing simple	p. 132
Use stories to manage uncertainty	p. 133
Reverse story components	p. 134
Focus on understanding "who"	p. 134
Focus on understanding "what"	p. 135
Focus on understanding "why"	p. 137
Be wary of the remaining w's	p. 139
Add acceptance criteria to the story-writing conversations	p. 140
Summary	p. 141
Chapter 5 Deriving Initial Project Backlogs	p. 143
Value of the initial backlog	p. 144
Sketch of the sample project	p. 145
Fitting initial backlog work into a release cycle	p. 146
The handoff between enterprise and project architects	p. 148
Key observations	p. 152
User role modeling results	p. 154
Key persona definitions	p. 155
Carla in carp strategy	p. 155
Franklin in finance	p. 156
An example of an initial backlog interview	p. 157
Framing the project	p. 162
Finance is upstream	p. 164
Finance categorizes source data	p. 165
Customer segmentation	p. 165
Consolidated product hierarchies	p. 166
Sales channel	p. 166
Unit reporting	p. 167
Geographies	p. 168
Product usage	p. 168
Observations regarding initial backlog sessions	p. 170
Sometimes a lengthy process	p. 170
Detecting backlog components	p. 171
Managing user story components on the backlog	p. 173
Prioritizing stories	p. 173
Summary	p. 174
Chapter 6 Developer Stories for Data Integration	p. 175
Why developer stories are needed	p. 176
Introducing the "developer story"	p. 178
Format of the developer story	p. 179
Developer stories in the agile requirements management scheme	p. 180
Agile purists do not like developer stories	p. 181
Initial developer story workshops	p. 182
Developers workshop within software engineering cycles	p. 184
Data warehousing/business intelligence reference data architecture	p. 185
Forming backlogs with developer stories	p. 187
Evaluating good developer stories: DILBERT'S test	p. 190
Demonstrable	p. 190
Independent	p. 192
Layered	p. 192
Business valued	p. 192
Estimable	p. 194
Refinable	p. 194
Testable	p. 195
Small	p. 195
Secondary techniques when developer stories are still too large	p. 195
Decomposition by rows	p. 196
Decomposition by column sets	p. 198
Decomposition by column type	p. 200
Decomposition by tables	p. 201
Theoretical advantages of "small"	p. 203
Summary	p. 205
Chapter 7 Estimating and Segmenting Projects	p. 207
Failure of traditional estimation techniques	p. 208
Traditional estimating strategies	p. 209
Why waterfall teams underestimate	p. 211
Criteria for a better estimating approach	p. 213
An agile estimation approach	p. 215
Estimating within the iteration	p. 215
Estimating the overall project	p. 218
Quick story points via "estimation poker"	p. 219
Story points and ideal time	p. 223
Story points defined	p. 224
Ideal time defined	p. 224
The advantage of story points	p. 225
Estimation accuracy as an indicator of team performance	p. 227
Value pointing user stories	p. 228
Packaging stories into iterations and project plans	p. 229
Criteria for better story prioritization	p. 231
Segmenting projects into business-valued releases	p. 232
The data architectural process supporting project segmentation	p. 233
Artifacts employed for project segmentation	p. 234
Project Segmentation technique 1: dividing the star schema	p. 238
Project Segmentation technique 2: dividing the tiered integration model	p. 240
Project Segmentation technique 3: grouping waypoints on the categorized services model	p. 243
Embracing rework when it pays	p. 246
Summary	p. 247
Part 3 Adapting Iterative Development for Data Warehousing Projects
Chapter 8 Adapting Agile for Data Warehousing	p. 251
The context as development begins	p. 252
Data warehousing/business intelligence-specific team roles	p. 255
Project architect	p. 256
Data architect	p. 262
Systems analyst	p. 264
Systems tester	p. 265
The leadership subteam	p. 266
Resident and visiting "resources"	p. 267
New agile characteristics required	p. 268
Avoiding data churn within sprints	p. 269
Pipeline delivery for a sustainable pace	p. 273
New meaning for iteration 0 and iteration -1	p. 276
Pipeline requires two-step user demos	p. 278
Keeping pipelines from delaying defect correction	p. 279
Resolving pipelining's task board issues	p. 280
Pipelining as a buffer-based process	p. 283
Pipelining is controversial	p. 284
Continuous and automated integration testing	p. 285
High quality is a necessity	p. 287
Agile warehousing testing requirements	p. 288
The need for automation	p. 292
Requirements for a warehouse test engine	p. 293
Automated testing for front-end applications	p. 294
Evolutionary target schemas-the hard way	p. 297
Summary	p. 302
Chapter 9 Starting and Scaling Agile Data Warehousing	p. 303
Starting a scrum team	p. 303
Stage 1: time box and story points	p. 305
Stage 2: pipelined delivery	p. 306
Stage 3: developer stories and current estimates	p. 306
Stage 4: managed development data and test-driven development	p. 307
Stage 5: automatic and continuous integration testing	p. 307
Stage 6: pull-based collaboration	p. 309
Scaling agile	p. 309
Application complexity	p. 310
Geographical distribution	p. 311
Team size	p. 311
Compliance requirements	p. 311
Information technology governance	p. 312
Organizational culture	p. 312
Organizational distribution	p. 313
Coordinating multiple scrum teams	p. 314
Coordinating1 through scrum of scrums	p. 315
Matching milestones	p. 318
Balancing work between teams with earned-value reporting	p. 319
What is agile data warehousing?	p. 325
Communicating success	p. 328
Handoff quality	p. 329
Quality of estimates	p. 330
Defects by iteration	p. 330
Burn-up charts	p. 331
Cross-method comparison projects	p. 333
Cycle times and story point distribution	p. 334
Moving to pull-driven systems	p. 335
A glimpse at a pull-based approach	p. 335
Kanban advantages	p. 340
A more cautious view	p. 341
Stages of scrumban	p. 343
Summary	p. 344
References	p. 345
Index	p. 353

Available:*

On Order

Summary

Summary

Author Notes

Table of Contents