Code Yellow 机制：八周零宕机的集中攻坚法

Towards the end of 2025, it seemed that we couldn’t get through a week at Provet without some kind of infrastructure degradation. Outages were piling up, they were hard to diagnose, and customers were losing patience.

When a major AWS outage hit in October, our customers didn’t distinguish between Amazon’s problem and ours: months of flakiness had already eroded their trust.

So we declared a Code Yellow, the first time we’d done it: a company-wide written announcement explaining the concept, the structure, the exit criteria, and the reasoning. The target was simple: tighten our SLAs and achieve zero downtime for eight consecutive weeks.

The good news was that we hit it, with only one brief degradation incident in the entire period, and the contrast with where we’d been just weeks earlier was extraordinary.

This article is about that experience, and about the broader practice of Code Yellows and Code Reds: what they are, how to run one well, when to call one, and when to end one.

Here’s what we’re going to cover:

• What Code Yellow and Code Red actually mean, and where the terms come from.
• How we ran ours at Provet: the structure, the work, and what we learned.
• A generalised template you can adapt for your own organisation.
• When to call one, when to escalate, and the failure modes to watch for.

If you find this topic interesting, here are some complementary articles from the archive:

• One bottleneck at a time explores why fixing one binding constraint compounds faster than optimising everything at once, which is what a Code Yellow forces you to do.
• The beauty of constraints covers how deliberate limitation unlocks unconventional thinking.
• Going direct is about communicating without intermediaries, which becomes essential during an escalation.
• Invert, always invert explores defining failure first, which connects directly to setting exit criteria before you begin.

Let’s get going.

What Code Yellow and Code Red actually mean

Before I share our experience, it’s worth defining the vocabulary.

The terms Code Yellow and Code Red originated at Google. As Steven Levy describes in In The Plex, the name came from a yellow tank top owned by engineering director Wayne Rosing: whoever wore the shirt became the designated leader and could tap anyone at Google on the shoulder, pulling them off their current project to help.

The practice has since spread across the industry: LinkedIn, Meta, Shopify, Instacart, and OpenAI all use variations of it. And us of course, and maybe even you too.

The core idea is simple. A Code Yellow is a formal declaration that something is seriously wrong and needs concentrated, cross-functional attention now, before it becomes catastrophic, while a Code Red is the next level up: an existential threat that demands everything stops until it is fixed.

Here’s the distinction:

The real power lies in having a shared language at the company, not in the table’s definitions. When everyone in your organisation understands what “Code Yellow” means, you don’t need to explain the severity, negotiate priority, or convince people to shift their focus, because the concept itself does the work for you.

Compare that to “we’re really concerned about uptime” or “this needs to be a top priority,” both of which are vague enough to be ignored, or for other functions like product to argue that their own priorities are more important.

As LinkedIn’s SRE team described it, the goal is to get a team “out of a reactive mode where they are running from crisis to crisis and into a proactive state.” That’s precisely what it felt like for us, and what the next section is about.

How we ran ours

Let me dig into the Provet Code Yellow in more detail.

The trigger was a pattern, not a single incident. As described in the opening of this article, throughout the back end of 2025, we’d had multiple outages that were frustratingly difficult to diagnose, and that difficulty itself was a signal: it pointed to deeper infrastructure problems rather than isolated bugs.

When the October AWS outage hit, our customers perceived it as our problem because months of instability had already conditioned them to expect that this again was all our fault.

As an exec team, we agreed that something bolder was needed, and the CEO signed off. I announced the initiative to the entire company in writing, and since it was our first Code Yellow, the announcement explained the concept itself alongside the specifics: what we were targeting, how long it would take, what the exit criteria were, and why we were doing it now.

The framing was deliberate. Uptime metrics are getting worse, we are losing trust with customers, and we need to act now before this becomes a Code Red.

The first week was intense and, honestly, chaotic: daily stand-ups, daily war rooms, and a full audit of monitoring, speed measurements, logs, and alerting.

The scope of what needed fixing was larger than anyone had anticipated, and prioritising within the Code Yellow itself generated real debate, though we found a surprising amount of low-hanging fruit (e.g. implementing additional timeouts and circuit breakers) and those quick wins built early momentum.

As the immediate fixes gave way to longer-term infrastructure work (migrating services to better technologies, rearchitecting for resilience), the cadence naturally slowed from daily to bi-weekly, then weekly.

Our approach was to walk the software stack from the bottom up: starting at the databases and what was connected to them, analysing access patterns and API calls, then moving further up the stack to look at rate limiting. Who was calling our API? What slow queries existed? What should the timeouts be across the stack as a result?

We also walked the stack from the outside in, looking at monitoring and alerting and identifying gaps that needed fixing.

Throughout, we maintained async updates in a public Slack channel called #code-yellow-uptime, deliberately over-sharing so anyone in the company could see exactly what was happening. Written shares followed a similar cadence to the above.

That transparency, the proximity and visibility it created to the actual work, was one of the most valuable parts of the whole exercise.

One powerful structural decision of Code Yellows is what is called “tap on the shoulder” authority. If an endpoint was particularly slow for our biggest customers, anyone working on the Code Yellow could tap the domain team that owned it and tell them to reprioritise to speed it up immediately. That kind of authority only works when the entire company understands and supports the escalation, which is why the concept and upfront communication mattered so much.

Getting people to deprioritise their roadmap work was surprisingly easy in our case, though I recognise that’s partly a function of context. At a company of our size, the instability was felt by everyone, and after the AWS outage, everyone saw fixing it as the number one priority. After all, a direct correlation exists between the stability of our product and the happiness of our customers.

I suspect deprioritisation is harder at larger organisations where the pain isn’t evenly distributed. If that’s your world, the communication piece becomes even more critical: the problem statement needs to make the impact concrete and company-wide, and getting exec buy-in and sign off is critical.