GitHub 8月17日宕机7小时47分，官方复盘与后续计划
The August 17 outage, and the work ahead
On August 17, GitHub experienced an outage that lasted 7 hours and 47 minutes. It disrupted github.com, authentication, GitHub Actions, APIs, pull requests, issues, and Copilot, affecting developers and organizations around the world. If you were trying to ship software that day, we let you down.
8月17日，GitHub经历了一次持续7小时47分钟的中断。这次中断影响了github.com、身份验证、GitHub Actions、API、拉取请求、问题跟踪以及Copilot，波及全球的开发者和组织。如果你当天正试图发布软件，我们让你失望了。
This was our second significant incident in August, following an actions failure on August 6. In March and April, I shared the work underway to improve GitHub’s reliability. We have made progress, but these incidents make clear that we must accelerate this work.
这是我们在8月份发生的第二次重大事件，此前在8月6日有一次Actions故障。在3月和4月，我分享了正在进行的提升GitHub可靠性的工作。我们取得了一些进展，但这些事件清楚地表明，我们必须加快这项工作。
What happened
发生了什么
Our investigation found that the outage began when traffic reached a new peak, and a critical infrastructure component in our Central US data center failed to scale with it. The resulting capacity pressure spread through our systems, causing authentication failures and disrupting multiple GitHub services.
我们的调查发现，中断始于流量达到新的峰值，而我们美国中部数据中心的一个关键基础设施组件未能随之扩展。由此产生的容量压力蔓延至整个系统，导致身份验证失败，并中断了多项GitHub服务。
Recovery required several coordinated actions. Teams rerouted traffic, isolated affected infrastructure, and restored services in stages. Most GitHub services recovered earlier that day, but some Copilot services took longer. Errors in those services triggered a client-side retry loop that increased traffic during recovery. We had to mitigate that behavior before we could safely restore traffic. The full root cause analysis includes a detailed technical timeline.
恢复需要多项协调行动。团队重新路由了流量，隔离了受影响的基础设施，并分阶段恢复了服务。当天早些时候，大多数GitHub服务已恢复，但部分Copilot服务耗时更长。这些服务中的错误触发了客户端重试循环，在恢复期间增加了流量。我们必须先缓解这一行为，才能安全地恢复流量。完整的根本原因分析包含详细的技术时间线。
Neither outage was caused by a code or configuration change. Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity. Since April, monthly commits have grown from 1.4 billion to 2.9 billion. That growth explains the pressure on our systems, but it does not excuse these outages.
这两次中断都不是由代码或配置更改引起的。这两起事件的核心都是容量故障。我们在需求超过容量之前未能扩展关键组件。自4月以来，月度提交量从14亿增长到29亿。这种增长解释了系统承受的压力，但这并不能为这些中断开脱。
What we have done and what comes next
我们已做的和接下来的计划
As part of the reliability commitments we made earlier this year, we have focused on three priorities: adding capacity, improving efficiency, and removing architectural bottlenecks. We have since added more than 3 million CPU cores, 120 petabytes of high-speed storage, and significant network capacity. We installed as much hardware as available power allowed in our existing data centers while accelerating our migration to Azure.
作为今年早些时候我们做出的可靠性承诺的一部分，我们专注于三个优先事项：增加容量、提高效率和消除架构瓶颈。此后，我们增加了超过300万个CPU核心、120PB的高速存储以及大量的网络容量。我们在现有数据中心内尽可能安装硬件，同时加速向Azure的迁移。
Today, Azure serves roughly 58% of GitHub’s platform load and half of all Git operations, up from 12% of platform load in May. This expanded footprint has also supported the growth in GitHub Actions job runs shown below.
如今，Azure承担了GitHub约58%的平台负载和一半的Git操作，而5月份时平台负载仅为12%。这一扩展的足迹也支持了下方所示的GitHub Actions作业运行量的增长。
Azure’s infrastructure and capacity have also accelerated our work to scale the largest monorepos. Our next milestone is an architecture that scales read capacity linearly with the number of readers, enabling unlimited read operations. We will roll it out gradually, beginning with the largest monorepos.
Azure的基础设施和容量也加速了我们扩展最大单体仓库的工作。我们的下一个里程碑是构建一种架构，使读取容量随读者数量线性扩展，从而实现无限读取操作。我们将从最大的单体仓库开始，逐步推出这一架构。
Scale is not our only challenge. As the pace and complexity of change increased, our existing operational practices did not keep up. We have redirected teams and resources toward availability and invested in stronger testing, safer rollouts, better observability, and more effective alerting. We have made progress, but this work is not complete.
规模并非我们唯一的挑战。随着变更速度和复杂性的增加，我们现有的运维实践已跟不上步伐。我们已将团队和资源转向可用性，并投资于更强大的测试、更安全的发布、更好的可观测性和更有效的告警。我们已取得进展，但这项工作尚未完成。
In addition, we are also isolating critical systems and removing shared dependencies between them. This work is designed to reduce the likelihood of an outage and limit its impact when one occurs.
此外，我们还在隔离关键系统，并移除它们之间的共享依赖。这项工作旨在降低故障发生的可能性，并在故障发生时限制其影响。
We learn from every outage and add new work to our availability workstream. The August 6 and August 17 incidents led to two immediate changes. First, we are applying consistent retry limits, retry budgets, and variable timeouts across service-to-service interactions to prevent retry storms and cascading load. Second, we are reviewing lower-priority CPU and memory alerts to identify components that could fail during sudden traffic spikes.
我们从每次故障中学习，并将新工作纳入可用性工作流。8月6日和8月17日的事件导致了两个立即的变更。首先，我们在服务间交互中应用一致的重试限制、重试预算和可变超时，以防止重试风暴和级联负载。其次，我们正在审查低优先级的CPU和内存告警，以识别在突发流量高峰期间可能失败的组件。
Our commitment to high availability isn’t just a technical promise. The developer community depends on GitHub to build, ship, and operate their work. That is only possible if you can rely on us, and on August 17, you couldn’t. It is our responsibility to fix that. We’ll earn your trust through the scaling and reliability of the platform.
我们对高可用性的承诺不仅仅是技术上的保证。开发者社区依赖GitHub来构建、发布和运营他们的工作。只有当你能够信赖我们时，这才成为可能，而在8月17日，你无法信赖我们。修复这个问题是我们的责任。我们将通过平台的扩展性和可靠性来赢得你的信任。
The post The August 17 outage, and the work ahead appeared first on The GitHub Blog.
文章《8月17日故障及后续工作》最初出现在GitHub博客上。
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力