构建韧性系统：灾备、RPO/RTO 与 AI 时代的可靠性规划

Last week’s guest on the podcast was Sam Newman, who had released a book I’ve personally long awaited titled Building Resilient Distributed Systems. It’s an especially timely book for many of us software engineers, as we’re seeing more outages with nearly all services and systems – much of it caused by more code deploys thanks to AI agents writing more code; and more bugs causing more incidents in production.

Sam’s new book, Building Resilient Distributed Systems, in print form

Sam’s book being “fresh out of the oven,” it naturally also touches on what AI changes as we design resilient systems. In fact, Sam told me that he took a total of three years to finishe the book – instead of the two he originally planned – because he went back and reworked all parts of the book, updating parts where AI is relevant to talk about.

I asked Sam if he would consider sharing an excerpt from the book, with sections especially relevant for anyone building systems that ought to be reliable. Sam graciously agreed, and today’s issue covers a few important topics, spread across two chapters from his book. We get into:

• Disaster recovery (DR). Recovering from catastrophic failures: planning for cloud region/data center/site-wide outages, the concept of RPO and RTO, DRO objectives and SLOs.
• Disaster Recovery Patterns. Backup-and-restore, pilot light failover, warm failover and multisite active-active.
• Infrastructure availability when planning for DR. Ongoing GPU and CPU shortages mean you cannot expect infrastructure availability at all times, so you need to understand constraints your vendors have, and to prioritize critical services.
• Resiliency of AI-powered software. Model provider resilience, the challenge of models changing, and the need for observability.
• How to handle uncertainty. Tactics like being multi-model and multi-vendor, making costs transparent, and frequently analyzing vendors.

Even though this article is pretty long, we cover less than a full chapter from the book's 16 chapters. If you build reliable systems for a living, I warmly recommend the book itself, and you can get it from the publisher.

My usual disclaimer: as with all my recommendations, I was not paid to recommend this book, and none of the links are affiliate ones. See my ethics statement for more details.

The excerpts below are from Building Resilient Distributed Systems, by Sam Newman. Copyright © 2026 Sam Newman. Published by O’Reilly Media, Inc. Used with permission.

1. Disaster Recovery

Excerpt from Chapter 10: Disaster Recovery.

We’ve already discussed how to handle a variety of failures that impact individual service instances. But what happens when a catastrophic failure causes widespread outage of large parts of your system? Fire, flood, electrical supply issues, and errors in the underlying cloud infrastructure are all outside of your control, and at one fell swoop, they can render your attempts to make your system more robust totally redundant.

Handling these situations falls under the auspices of disaster recovery (DR), a set of procedures that should kick in when something really bad happens. As with much of what we’ve looked at in this book, trade-offs abound. As you reduce your period of acceptable downtime or data loss, cost and complexity increase, so picking the right approach for your system can be a delicate balancing act. In this chapter, I’ll give you some concrete targets to work with to help drive your own DR approach.

Cloud Region, Data Center, or Site?

The exact terminology you use to discuss your infrastructure will likely vary based on implementation details, and this in turn can make discussions around generally applicable patterns more difficult. When running on the public cloud, for example, you’ll probably be familiar with the concept of a cloud region, which is where your cloud resources live. When running on a private cloud or in a more traditional on-premises setup, you’ll instead think in terms of data centers.

As we discuss the various options around handling DR, the patterns that we’ll see later in this chapter can be used for either public or private cloud scenarios. Therefore, to keep things simple, I’ll use the term site as a general descriptor of where your computing infrastructure lives.

There are some specific issues to consider around mixing cloud vendors as part of a DR strategy, and we’ll look more at those in Chapter 11.

How Good Does Your DR Need to Be?

A spectrum of DR options are available to you, and they can range greatly in terms of cost, complexity, and recovery. To help you decide which DR approach is most appropriate, we should first try to understand how to determine the appropriate requirements for your context.

A simple way to define technical requirements around DR is to make use of two concrete measurable targets, recovery point objective and recovery time objective. In Figure 10-1, you can see how these objectives relate to an incident:

Figure 10-1. The key objectives to define as part of selecting an appropriate DR strategy

Recovery Point Objective

The recovery point objective (RPO) defines the maximum amount of time that can have passed since the data was last backed up or synchronized. Put another way, RPO defines the upper duration of potential data loss. If the data for your system backs up once an hour, then it’s possible to lose up to an hour of data if your system has an incident. Therefore, your RPO would be an hour.

Recovery Time Objective

The recovery time objective (RTO) defines how long it’s acceptable for your system to be unavailable in the wake of an incident. If all your computers became unavailable, perhaps because of a data center–wide outage, how long would it take you to have your services running in an alternative data center? Your RTO includes all the time needed to recover data from the last recovery point, make computing resources available for your workloads, and redirect traffic to your new site—so there are more variables to consider in terms of bringing your system back online than just time.

You Can’t Have Everything

If asked what RPO and RTO you’d want in a vacuum, you’d probably give a very simple answer: you’d want an RPO of zero (i.e., a guarantee of no data loss), and an RTO of zero too. This is an understandable wish, but as you reduce your RPO and RTO, cost and complexity increase dramatically—and in technical terms, it’s not possible to guarantee a zero RPO or RTO in any case.

An additional complication is that different types of workloads might require different RPOs and RTOs. Consider a bank, for example. Having a low RPO to ensure that payment information isn’t lost would be key, and having a low RTO to ensure that people can still take money out of their account is vital. On the other hand, the part of the system that enables new customers to sign up for an account could have a higher RTO and RPO, as the impact to the business is less severe.

So when deciding on the appropriate RPO and RTO, there needs to be a discussion with stakeholders regarding end-user needs and regulatory and legislative requirements, which are balanced against the cost and complexity of implementing the required solution. For example, your product owner might initially say they need an RPO of 10 minutes, but if you could tell them it would take months of work to implement (during which other work has to be deprioritized), they might reduce their target, perhaps accepting an RPO that is easier for you to deliver. Your target RPO and RTO should also exceed anything specified in any service level agreements you have.

DR Objectives and SLOs

At first glance, RPO and RTO would appear to make for good service level objectives (SLOs). After all, they are both important measures that directly impact user experience. However, there are some challenges around managing RPO and RTO as SLOs.