Before You Replace the Engine, Rehearse the Failure
An evergreen two-host workshop on migration design, uncertain retries, and making a technical proposal useful to another team. The examples are hypothetical; this pilot is not a live report on the organization's week.
Try this: For one proposed change, prepare a five-line decision note: user outcome, current baseline, smallest experiment, failure/rollback condition, and the person who should challenge the plan.
Chapters
00:00:00 What Are We Improving?
00:01:59 Design the Migration
00:05:05 Rehearse the Rollback
00:06:00 When Retrying Is Unsafe
00:08:16 Bring People With You
00:09:53 Your Five-Line Decision
Sources
AWS Prescriptive Guidance: Strangler fig pattern: https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/strangler-fig.html
AWS Builders' Library: Timeouts, retries, and backoff with jitter: https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/
AWS Builders' Library: Making retries safe with idempotent APIs: https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/
Website and show notes: https://developmentradio.dev
Spotify: https://open.spotify.com/show/3trD3hKUAviXCiSvCOzxDm
YouTube: https://www.youtube.com/@DevelopmentRadio
**00:00:00 — Alex:** I've brought a migration plan. It's elegant, modern, and has precisely one minor weakness.
**00:00:06 — Maya:** It hasn't met the existing system?
**00:00:08 — Alex:** It prefers to think of the existing system as a temporary scheduling inconvenience.
**00:00:13 — Maya:** Welcome to Development Radio. I'm Maya, with Alex. In this weekly-format pilot, we're going to put that plan through a rehearsal: one product decision, one failure that changes the design, and one conversation that might get the work adopted.
**00:00:28 — Alex:** Our product is fictional: a service that scans uploaded documents and shows customers the results. The current path works, but some large uploads take too long. A new processing engine looks promising. So naturally, my first slide says replace the engine.
**00:00:45 — Maya:** And my first question is: which waiting time are we trying to remove? The customer might be waiting for upload, queue admission, scanning, or a result page that refreshes late. A faster engine only helps the part it actually touches.
**00:00:59 — Alex:** Fair. Let's say we measured enough examples to see that scanning dominates for a particular class of large documents. We want those customers to receive useful results sooner. That's our starting claim, and we'd keep the measurements beside it.
**00:01:13 — Maya:** Better. Now we can compare options. Tune the current engine. Change how work is split. Try the replacement on a narrow slice. Even decide that another delay matters more. The new technology has to earn its place in that comparison.
**00:01:27 — Alex:** But a small slice can be misleading. If I only choose easy documents, the demo wins and the difficult cases remain difficult. So the trial needs examples that represent the actual pain, including inputs where we expect the candidate to struggle.
**00:01:42 — Maya:** And the baseline needs to be real. Record the old path under comparable conditions. Keep correctness, cost, and operator effort beside latency. A graph that gets faster because we quietly stopped doing part of the job is a very expensive magic trick.
**00:01:59 — Alex:** That trick does produce beautiful graphs. Let's talk about the migration shape. AWS describes the strangler fig pattern as incremental replacement: a routing boundary sends selected work to the new implementation while the existing path continues serving the rest.
**00:02:15 — Maya:** Before we get attached to the replacement, there's another practical question. Could a capability already in our stack solve this? Maybe we haven't used its batch mode, or we haven't measured a newer supported configuration. Familiar technology can contain unfamiliar options. Research shouldn't begin and end with launch announcements.
**00:02:35 — Alex:** I'd put those options in the same experiment. But I'd still verify their limits in current documentation. A feature existing somewhere in a product doesn't mean it's available in our version, supports our workload, or has an operating cost we're happy to carry.
**00:02:50 — Maya:** And we can learn before sending a customer down the new path. For our fictional scanner, a controlled copy of representative inputs could produce comparison results that never become customer-visible. That gives us a place to investigate disagreements while the existing path remains authoritative.
**00:03:07 — Alex:** With one large asterisk. A copied request mustn't quietly send another notification, change shared state, or expose documents to a destination we haven't approved. Calling something a shadow test doesn't remove its side effects. We have to trace where the copy actually goes.
**00:03:25 — Maya:** Then sort the disagreements. A deliberate improvement is different from a missing field, a timing difference, or a result we simply don't understand. One overall agreement percentage can hide the only category that matters to the customer. Pick examples and explain them.
**00:03:42 — Alex:** That also gives a colleague something concrete to teach from. They can walk us through a surprising old result and the constraint behind it. We learn the domain before deleting the behavior. Some historical complexity is accidental; some is an undocumented product requirement wearing a terrible variable name.
**00:04:00 — Maya:** Useful pattern, but not a commandment to make everything a microservice. In our example, the seam could initially be an ordinary interface inside one application. We should choose the smallest boundary that lets us compare the two behaviors honestly.
**00:04:15 — Alex:** I'd start with a route we can identify and switch deliberately. Maybe one document category in a controlled environment. The trial should tell us whether the new engine preserves the result contract, not just whether it can process a file.
**00:04:28 — Maya:** What belongs in that contract? Say it without the word compatibility.
**00:04:33 — Alex:** The customer sees the same meaning for the same fields. A missing result stays distinguishable from a clean result. Failures arrive in a form the caller understands. If the new engine changes any of those things, the adapter or the product decision must account for it.
**00:04:49 — Maya:** Now we're designing something another person can review. The AWS guide also warns that the routing layer can become a bottleneck or a failure point, and that coordinating old and new data brings consistency problems. Incremental doesn't mean consequence-free.
**00:05:05 — Alex:** Here's where my original slide gets nervous. It says rollback is simple: flip the flag back. But suppose the new engine has already written results the old path can't interpret. The flag restores traffic routing. It doesn't restore the old world.
**00:05:20 — Maya:** So rehearse a specific rollback. After one new result is written, what does the old reader do? After a batch is partly complete, who owns the remainder? And can an operator tell which path produced a confusing result without becoming a detective?
**00:05:35 — Alex:** I want to push back on the size of this exercise. We can invent failure cases forever. A small prototype shouldn't need the paperwork of an airport.
**00:05:44 — Maya:** Agreed. The depth should follow the consequence. A disposable local prototype needs a clear learning question. A customer-facing switch needs evidence for the failure paths it introduces. We don't make the prototype safe by writing more pages; we keep its effects contained.
**00:06:00 — Alex:** That's a useful boundary. Fast learning first, then stronger evidence as exposure grows. And now our rehearsal gets its first interruption: a request to the new engine times out. Do we send it again?
**00:06:12 — Maya:** Maybe. The timeout tells the caller it didn't receive a timely answer. It doesn't prove the operation never happened. The AWS retry guidance makes that distinction, and also explains why retries can add pressure to a system that's already struggling.
**00:06:27 — Alex:** In our document example, the engine might have finished, stored the result, and lost the response on the way back. Retrying could waste compute, create conflicting results, or trigger a second notification. The answer depends on the effect we're repeating.
**00:06:41 — Maya:** And please don't respond by adding exponential backoff to every layer. Backoff spaces attempts out. Jitter varies their timing. Those help manage retry traffic; they don't decide whether repeating a successful operation is semantically safe.
**00:06:57 — Alex:** For that, explicit request identity is one tool. The AWS idempotency article describes using a caller's request identifier to recognize repeated intent. The important design question is what one request means, and what response a repeat should receive.
**00:07:14 — Maya:** Let's make that uncomfortable. The same document is submitted twice. Is that a duplicate delivery of one request, or did the customer intentionally ask for a new scan with changed settings? Hashing the document alone might erase a legitimate second action.
**00:07:30 — Alex:** So our product semantics belong in the identity. We'd distinguish a repeated attempt from a new requested scan. Then we'd describe what happens if processing is still running, has completed, or failed after producing part of the visible result.
**00:07:45 — Maya:** We now have a more interesting experiment than a speed contest. Run a representative document, interrupt the response, repeat the same request, and inspect the customer-visible result. Then try a deliberately new request. Those should exercise different intentions.
**00:08:02 — Alex:** In a controlled environment, with effects we can clean up. And if we can't safely reproduce a production path, we should record that limit. A successful simulation is useful evidence. It isn't a secret substitute for observing the real integration.
**00:08:16 — Maya:** Our migration plan now has a seam, a comparison, and a recovery question. Here's the final obstacle: another team owns part of the existing path. You need their time. How do you open that conversation?
**00:08:30 — Alex:** I would like to introduce a revolutionary new architecture.
**00:08:33 — Maya:** Their calendars have just moved to another country.
**00:08:36 — Alex:** All right. I'd bring the customer problem and the baseline, then show two options. One improves the existing path. One tests the new engine. I'd explain the part I can investigate myself and the specific assumption where I need their experience.
**00:08:52 — Maya:** That's easier to respond to. Also ask what your proposal makes harder for them. They may own a support obligation, a compatibility promise, or a release constraint missing from your diagram. Discovering that early is progress, even if it changes your favorite option.
**00:09:08 — Alex:** And give someone else a meaningful part of the experiment. Not just a chore list: let them own the comparison criteria or challenge the failure assumptions. If the work only moves when I personally explain every detail, I've built a dependency on myself.
**00:09:23 — Maya:** There's a career lesson hiding in that. A useful story about this work would connect the original uncertainty, the people who helped resolve it, the decision, and the observed result. It should also preserve the limits: what we tested, what we didn't, and what changed our mind.
**00:09:40 — Alex:** So less I championed the latest engine, more we found which change helped these customers, and made the decision easier to trust. If the old engine wins the experiment, that can still be a good engineering outcome.
**00:09:53 — Maya:** Exactly. For your next move, choose one proposed change and write five lines. The user outcome. The current baseline. The smallest experiment. The failure or rollback condition. And the person who should challenge the plan before it grows.
**00:10:09 — Alex:** Take that note into a short conversation. Ask which line is least convincing. That answer is your next investigation. You don't need to arrive with an empire of slides; you need to make the next decision clearer.
**00:10:21 — Maya:** That's our weekly pilot. New technology earns its place through a useful comparison, and a migration earns confidence through a specific recovery story. I'm Maya.
**00:10:31 — Alex:** I'm Alex. My slide now has a baseline, a rollback question, and no logo.
**00:10:37 — Maya:** Personal growth. We love to hear it. See you next time on Development Radio.