How Google tests software

Do you need to get it right, too? Then, learn from Google . Legendary testing expert James Whittaker, until recently a Google testing leader, and two top Google experts reveal exactly how Google tests software, offering brand-new best practices you can use even if you're not quite Google's size... yet!

Breakthrough Techniques You Can Actually Use

Discover 100% practical, amazingly scalable techniques for analyzing risk and planning tests...thinking like real users...implementing exploratory, black box, white box, and acceptance testing...getting usable feedback...tracking issues...choosing and creating tools...testing "Docs & Mocks," interfaces, classes, modules, libraries, binaries, services, and infrastructure...reviewing code and refactoring...using test hooks, presubmit scripts, queues, continuous builds, and more. With these techniques, you can transform testing from a bottleneck into an accelerator -and make your whole organization more productive!

Author Notes

James Whittaker: James is an Engineering Director at Google and has been responsible for testing Chrome, maps, and Google web apps. He used to work for Microsoft and was a professor before that. James is one of the best known names in testing.
Jason Arbon: Jason is a Test Engineer at Google and has been test lead for Google Desktop, Chrome, and Chrome OS and development lead for an array of open source test tools. He worked at Microsoft prior to joining Google.
Jeff Carollo: Jeff is a Software Engineer in Test and has focused on developer tools for code presubmit testing. He has consulted with dozens of internal Google development teams helping them improve initial code quality. He converted to Software Engineer in 2010 and leads development of Google+ APIs.

Alberto SavoiaPatrick Copeland

Foreword	p. xiii
Foreword	p. xvii
Preface	p. xxiii
Chapter 1 Introduction to Google Software Testing	p. 1
Quality≠Test	p. 5
Roles	p. 6
Organizational Structure	p. 8
Crawl, Walk, Run	p. 10
Types of Tests	p. 12
Chapter 2 The Software Engineer in Test	p. 15
The Life of an SET	p. 17
Development and Test Workflow	p. 17
Who Are These SETs Anyway?	p. 22
The Early Phase of a Project	p. 22
Team Structure	p. 24
Design Docs	p. 25
Interfaces and Protocols	p. 27
Automation Planning	p. 28
Testability	p. 29
SET Workflow: An Example	p. 32
Test Execution	p. 40
Test Size Definitions	p. 41
Use of Test Sizes in Shared Infrastructure	p. 44
Benefits of Test Sizes	p. 46
Test Runtime Requirements	p. 48
Case 1: Change in Common Library	p. 52
Test Certified	p. 54
An Interview with the Founders of the Test Certified Program	p. 57
Interviewing SETs	p. 62
An Interview with Tool Developer Ted Mao	p. 68
An Interview with Web Driver Creator Simon Stewart	p. 70
Chapter 3 The Test Engineer	p. 75
A User-Facing Test Role	p. 75
The Life of a TE	p. 76
Test Planning	p. 79
Risk	p. 97
Life of a Test Case	p. 108
Life of a Bug	p. 113
Recruiting TEs	p. 127
Test Leadership at Google	p. 134
Maintenance Mode Testing	p. 137
Quality Bots Experiment	p. 141
BITE Experiment	p. 153
Google Test Analytics	p. 163
Free Testing Workflow	p. 169
External Vendors	p. 173
An Interview with Google Docs TE Lindsay Webster	p. 175
An Interview with YouTube TE Apple Chow	p. 181
Chapter 4 The Test Engineering Manager	p. 187
The Life of a TEM	p. 187
Getting Projects and People	p. 189
Impact	p. 191
An Interview with Gmail TEM Ankit Mehta	p. 193
An Interview with Android TEM Hung Dang	p. 198
An Interview with Chrome TEM Joel Hynoski	p. 202
The Test Engineering Director	p. 206
An Interview with Search and Geo Test Director Shelton Mar	p. 207
An Interview with Engineering Tools Director Ashish Kumar	p. 211
An Interview with Google India Test Director Sujay Sahni	p. 214
An Interview with Engineering Manager Brad Green	p. 219
An Interview with James Whittaker	p. 222
Chapter 5 Improving How Google Tests Software	p. 229
Fatal Flaws in Google's Process	p. 229
The Future of the SET	p. 231
The Future of the TE	p. 233
The Future of the Test Director and Manager	p. 234
The Future of Test Infrastructure	p. 234
In Conclusion	p. 235
Appendix A Chrome OS Test Plan	p. 237
Overview of Themes	p. 237
Risk Analysis	p. 238
Per-Build Baseline Testing	p. 239
Per-LKG Day Testing	p. 239
Per-Release Testing	p. 239
Manual Versus Automation	p. 240
Dev Versus Test Quality Focus	p. 240
Release Channels	p. 240
User Input	p. 241
Test Case Repositories	p. 24l
Test Dashboarding	p. 241
Virtualization	p. 241
Performance	p. 242
Stress, Long-Running, and Stability	p. 242
Test Execution Framework (Autotest)	p. 242
OEMs	p. 242
Hardware Lab	p. 242
E2E Farm Automation	p. 243
Testing the Browser AppManager	p. 243
Browser Testability	p. 243
Hardware	p. 244
Timeline	p. 244
Primary Test Drivers	p. 246
Relevant Documents	p. 246
Appendix B Test Tours for Chrome	p. 247
The Shopping Tour	p. 247
The Student Tour	p. 248
Suggested Areas to Test	p. 248
The International Calling Tour	p. 249
Suggested Areas to Test	p. 249
The Landmark Tour	p. 249
Suggested Landmarks in Chrome	p. 249
The All Nighter Tour	p. 250
Suggested Areas to Test	p. 250
The Artisan's Tour	p. 251
Tools in Chrome	p. 251
The Bad Neighborhood Tour	p. 251
Bad Neighborhoods in Chrome OS	p. 251
The Personalization Tour	p. 252
Ways to Customize Chrome	p. 252
Appendix C Blog Posts on Tools and Code	p. 253
Take a BITE out of Bugs and Redundant Labor	p. 253
Unleash the QualityBots	p. 255
RPF: Google's Record Playback Framework	p. 257
Google Test Analytics-Now in Open Source	p. 260
Comprehensive	p. 260
Quick	p. 260
Actionable	p. 260
Sustained Value	p. 260
Index	p. 265

Available:*

On Order

Summary

Summary

Author Notes

Table of Contents