Table of Contents for
Site Reliability Engineering

Version ebook / Retour

Cover image for bash Cookbook, 2nd Edition Site Reliability Engineering by Jennifer Petoff Published by O'Reilly Media, Inc., 2016
  1. nav
  2. Cover
  3. Praise for Site Reliability Engineering
  4. Site Reliability Engineering
  5. Site Reliability Engineering
  6. Foreword
  7. Preface
  8. I. Introduction
  9. 1. Introduction
  10. 2. The Production Environment at Google, from the Viewpoint of an SRE
  11. II. Principles
  12. 3. Embracing Risk
  13. 4. Service Level Objectives
  14. 5. Eliminating Toil
  15. 6. Monitoring Distributed Systems
  16. 7. The Evolution of Automation at Google
  17. 8. Release Engineering
  18. 9. Simplicity
  19. III. Practices
  20. 10. Practical Alerting from Time-Series Data
  21. 11. Being On-Call
  22. 12. Effective Troubleshooting
  23. 13. Emergency Response
  24. 14. Managing Incidents
  25. 15. Postmortem Culture: Learning from Failure
  26. 16. Tracking Outages
  27. 17. Testing for Reliability
  28. 18. Software Engineering in SRE
  29. 19. Load Balancing at the Frontend
  30. 20. Load Balancing in the Datacenter
  31. 21. Handling Overload
  32. 22. Addressing Cascading Failures
  33. 23. Managing Critical State: Distributed Consensus for Reliability
  34. 24. Distributed Periodic Scheduling with Cron
  35. 25. Data Processing Pipelines
  36. 26. Data Integrity: What You Read Is What You Wrote
  37. 27. Reliable Product Launches at Scale
  38. IV. Management
  39. 28. Accelerating SREs to On-Call and Beyond
  40. 29. Dealing with Interrupts
  41. 30. Embedding an SRE to Recover from Operational Overload
  42. 31. Communication and Collaboration in SRE
  43. 32. The Evolving SRE Engagement Model
  44. V. Conclusions
  45. 33. Lessons Learned from Other Industries
  46. 34. Conclusion
  47. A. Availability Table
  48. B. A Collection of Best Practices for Production Services
  49. C. Example Incident State Document
  50. D. Example Postmortem
  51. E. Launch Coordination Checklist
  52. F. Example Production Meeting Minutes
  53. Bibliography
  54. Index
  55. About the Authors
  56. Colophon

Appendix C. Example Incident State Document

Shakespeare Sonnet++ Overload: 2015-10-21
Incident management info: http://incident-management-cheat-sheet

(Communications lead to keep summary updated.)
Summary: Shakespeare search service in cascading failure due to newly discovered sonnet not in search index.

Status: active, incident #465

Command Post(s): #shakespeare on IRC

Command Hierarchy (all responders)

  • Current Incident Commander: jennifer

    • Operations lead: docbrown

    • Planning lead: jennifer

    • Communications lead: jennifer

  • Next Incident Commander: to be determined

(Update at least every four hours and at handoff of Comms Lead role.)
Detailed Status (last updated at 2015-10-21 15:28 UTC by jennifer)

Exit Criteria:

  • New sonnet added to Shakespeare search corpus TODO

  • Within availability (99.99%) and latency (99%ile < 100 ms) SLOs for 30+ minutes TODO

TODO list and bugs filed:

  • Run MapReduce job to reindex Shakespeare corpus DONE

  • Borrow emergency resources to bring up extra capacity DONE

  • Enable flux capacitor to balance load between clusters (Bug 5554823) TODO

Incident timeline (most recent first: times are in UTC)

  • 2015-10-21 15:28 UTC jennifer

    • Increasing serving capacity globally by 2x

  • 2015-10-21 15:21 UTC jennifer

    • Directing all traffic to USA-2 sacrificial cluster and draining traffic from other clusters so they can recover from cascading failure while spinning up more tasks

    • MapReduce index job complete, awaiting Bigtable replication to all clusters

  • 2015-10-21 15:10 UTC martym

    • Adding new sonnet to Shakespeare corpus and starting index MapReduce

  • 2015-10-21 15:04 UTC martym

    • Obtains text of newly discovered sonnet from shakespeare-discuss@ mailing list

  • 2015-10-21 15:01 UTC docbrown

    • Incident declared due to cascading failure

  • 2015-10-21 14:55 UTC docbrown

    • Pager storm, ManyHttp500s in all clusters