Multi-Zone Failover: Healthy Servers, Stuck Jobs

00:00:02 — Maya
Welcome to Development Radio. I'm Maya, here with Alex. Today we're talking about cloud availability, through a very ordinary moment: uploading a file and waiting for the little progress bar. Alex, do you leave that tab open, or do you trust the notification?

00:00:19 — Alex
I leave it open. Then I go and do something else, which is what I could have done in the first place. But the open tab makes me feel involved. Like I'm supervising the upload from another room.

00:00:30 — Maya
I leave it open until the first few products appear. Then I can leave it alone. I've already spent the morning checking columns and fixing names; I want to hand the file over and let the computer have a turn.

00:00:41 — Alex
When you say the first few, are we talking three products, or are you still there when the whole catalog arrives?

00:00:48 — Maya
Three would be sensible. In practice I find one that has a very long name and wait for that one. I want to see how it looks before I go and do something else.

00:00:58 — Alex
Ah. So there's a test product now. I thought we were discussing whether to close a browser tab.

00:01:03 — Maya
We are. I just have a little ceremony before I close it. You supervise from another room; I inspect the awkward name. Either way, I'd like to get on with the rest of the day.

00:01:13 — Alex
Google's preliminary report, published on October twenty-second, gives us a useful reason to look behind that handoff. During the London outage on October twentieth, a power interruption took one availability zone offline. Some workloads in other zones were affected too.

00:01:29 — Maya
The report describes both pressure on the surviving capacity and calls waiting for validation from the offline zone. Those are different problems.

00:01:38 — Alex
Google calls the second problem "synchronous control-plane validation dependencies across zonal boundaries." Synchronous means the operation waits for that answer before proceeding. The investigation is preliminary; this isn’t a claim that every multi-zone workload failed.

00:01:54 — Maya
Let's follow that wait. If our workers run in three zones, why can't our service start the next job?

00:02:00 — Alex
Now imagine a small product-import service of our own. A shop uploads a product file. Before any worker gets that file, an admission service checks the selected import definition. That definition says how to read the columns and where the results go. Workers run in three zones.

00:02:17 — Maya
Give me one import already running. Its worker has the accepted instructions, and it can still reach the file and the destination. Being alive wouldn't be enough if its next read needed the failed zone.

00:02:29 — Alex
Its file and destination are still reachable. Now one zone becomes unreachable. The running import keeps processing. The shop submits its next file, using the same definition. That new import waits at admission. No worker has received it.

00:02:44 — Maya
Is that because the two surviving zones are full?

00:02:47 — Alex
They have room for this file. But admission asks the definition store whether the definition can be used. In this design, that required answer still comes from the zone we lost.

00:02:57 — Maya
Then adding another worker won't start this import. A worker is already available. The new file is waiting for a decision that can't arrive.

00:03:06 — Alex
That's the trap. We distributed the workers, but starting their next job still depends on the failed zone. The three-zone diagram described where we could do the work. It didn't show everything needed to let the work begin.

00:03:19 — Maya
And the person at the shop sees one file making progress and the next one going nowhere. I'd understand why they thought the second file was wrong. But changing its columns wouldn't help with this wait.

00:03:31 — Alex
No. They've already done their part. One import is processing; the next is waiting for admission. That's the distinction I'd want the service to show. There's an available worker on the other side of the gate, but this file hasn't reached it.

00:03:44 — Maya
I'd keep a local copy of that import definition. The shop used it for the first file. We know how to read those columns. Why fetch the same instructions again before the second file?

00:03:55 — Alex
For the instructions, that could be reasonable. But what did we promise when we published that definition? Can somebody disable it immediately, or is a published version valid for a stated period?

00:04:06 — Maya
In our original service, disabling it blocks new admissions. It doesn't cancel imports already admitted. So a copy tells me how to run the import, but not whether admission can accept the next file.

00:04:18 — Alex
And that distinction is why I'd keep new admissions waiting for now. The live check is buying a current decision. Removing it during an outage would quietly change the promise to whoever disabled that definition.

00:04:30 — Maya
I agree we can't quietly change it. But keeping every new file waiting is a choice too. The workers are available, the instructions haven't changed, and the shop may need today's product catalog online. I'd want to know whether immediate disable is actually the contract the product needs.

00:04:47 — Alex
Then let's consider a different promise. Suppose we agree, before an outage, that approval of this definition lasts five minutes from issuance. The service already holds that approval and its instructions before the outage. While the approval is unexpired, admission can finish this definition check locally. It doesn't skip the other checks for a new import.

00:05:09 — Maya
Then if somebody disables the definition while that approval is still valid, admission can still accept a new file under that approval. That's the cost of keeping this part of admission available.

00:05:19 — Alex
Yes. And if the definition store is still unavailable when the approval expires, the next import waits again. We don't extend the approval just because refreshing it is inconvenient. The five-minute contract buys a bounded period of admission, not an unlimited outage escape.

00:05:36 — Maya
I would push for that option if the shop can tolerate the disable delay. But our original service hasn't made that agreement. In the service we started with, the first import is still running and the next one still waits. We have a design choice to make before changing that answer.

00:05:51 — Alex
And if we need immediate disable and still want to admit new imports through that failure, the live admission decision has to remain available through it. That's the component to investigate. Duplicating workers doesn't duplicate that decision.

00:06:05 — Maya
Which changes what I'd draw on our three-zone picture. I'd follow this file into admission, through the definition check, and only then to a worker. The line leaving the surviving zone matters as much as the worker sitting inside it.

00:06:18 — Alex
Suppose we can keep that definition check available. Is the shop now able to get all its files through? We've been concentrating on this one waiting file.

00:06:27 — Maya
Not necessarily. We gave our surviving workers room for this next file. We haven't proved they have room for everything the failed zone was doing. If we fix admission and flood them with work, we could overload the surviving workers next.

00:06:40 — Alex
Then I'd keep enough worker capacity ready before the failure. Creating machines during it would give us another service to wait for—the one that places and configures them.

00:06:50 — Maya
I'd want to price that before promising it. Keeping room ready costs money even on days when all three zones work. If the shop doesn't want to pay for that much spare capacity, we need to agree how much new work can wait.

00:07:02 — Alex
That's the choice. AWS calls preparing to continue without depending on emergency changes static stability. For this service, we'd agree how much new work the surviving workers can accept, and tell the shop to wait before submitting more. Fixing admission isn't permission to overwhelm them.

00:07:20 — Maya
I'd rather say this running import can keep making progress, and explain why the next one is waiting, than advertise three zones as a promise that every operation keeps working. But if fresh imports are the business-critical part, that narrower promise isn't enough.

00:07:36 — Alex
In a test environment, take one real new job through your architecture with a zone unavailable. Follow it from submission to the first worker. Find the first required answer that can no longer arrive, and decide what the service is allowed to do while that answer is missing. Test that exact behavior, including the capacity left behind.

00:07:56 — Maya
And make the waiting state something the person with the file can understand. They shouldn't have to keep fixing a perfectly good spreadsheet because our admission check is unavailable.

00:08:06 — Alex
You can keep the progress tab open if you like. At least it should tell you what you're waiting for. Thanks for listening to Development Radio.