**00:00:02 — Alex:** Welcome to Development Radio. I'm Alex, here with Maya. Today we're talking about PostgreSQL backups. Maya, when you're moving house, how much unpacking has to happen before you're allowed to stop for the evening?

**00:00:15 — Maya:** As soon as the bed is made. The clean sheets go in my open-first box. Kitchen, books, cables: those can wait. I don't want to unpack an entire home before I can go to bed.

**00:00:26 — Alex:** I would still have three boxes marked important. Stopping would feel like admitting I have no idea what important meant when I packed them.

**00:00:34 — Maya:** Sleep important, paperwork important, and something you promised you'd fix?

**00:00:39 — Alex:** The last one travels with me. At this point it has a right to vote.

**00:00:44 — Maya:** This is why I pack the sheets where I can find them. There will always be another box. I'm not spending the first night negotiating with every one of them.

**00:00:52 — Alex:** Are you really stopping with everything still in boxes? I don't think I'd sleep until I'd found the plates.

**00:00:58 — Maya:** I absolutely am. Bed made, door locked, done for the evening. You can conduct your plate search quietly.

**00:01:05 — Alex:** I was about to offer to help, but I can see the terms and conditions are strict.

**00:01:10 — Maya:** You can help. Sheets first, plate expedition afterwards. And you brought up moving boxes, so I suspect you had a destination in mind.

**00:01:19 — Alex:** To a small change in this week's pgBackRest release. It's made me reconsider what we want a backup job to call done.

**00:01:26 — Maya:** When a backup finds damaged data, should it stop, or finish saving what it can?

**00:01:31 — Alex:** Let's put ourselves on a recovery call at an imaginary wholesaler. The original orders database is unavailable. Its latest backup finished overnight, and the team is restoring that copy to a replacement server. They're using pgBackRest two point sixty, which came out on October eighteenth.

**00:01:47 — Maya:** And the restore stops on a page-checksum error. The replacement database is incomplete. We can't just start it and put the order desk back to work.

**00:01:55 — Alex:** They're waiting to find out what they can ship. Our dashboard says the backup completed, which sounded reassuring until we needed it.

**00:02:03 — Maya:** I'd want to know whether the backup was damaged in storage, or whether we backed up damage that was already there. Those leave us with different places to investigate.

**00:02:12 — Alex:** In our case, the backup's own record contains a page-checksum warning. That warning was written during the overnight backup. Nobody followed it up. The damaged page wasn't a new discovery on this recovery call.

**00:02:24 — Maya:** Then the backup already knew. Why did the recovery team have to discover it again?

**00:02:28 — Alex:** The release notes put it this way: "Previously the restore command did not report invalid page checksums at all." Version two point sixty changes the restore behavior. It stops by default when the backup has those errors.

**00:02:42 — Maya:** So an upgrade can expose the problem at restore time. The new version didn't put the damaged page in our overnight copy.

**00:02:49 — Alex:** And the backup command still warns by default. It can finish collecting the data while recording the bad pages. Our case has PostgreSQL page checksums enabled, so that check was available.

**00:03:00 — Maya:** We should stay with those checks for a moment. Someone on the call could reasonably say, we verified the backup files, how can they be corrupt?

**00:03:08 — Alex:** pgBackRest calculates a checksum for each backup file, and checks it again when restoring or verifying. That can tell us whether the saved file still matches the file that was captured. But imagine the database page was already damaged before we copied it.

**00:03:23 — Maya:** Then we can copy the damage faithfully. The file we get back matches the file we saved.

**00:03:28 — Alex:** Yes. The database page has its own checksum, written with its contents. A mismatch there can reveal damage inside the source data. Checking that page and checking our saved file answer different questions.

**00:03:41 — Maya:** A backup can preserve damaged data perfectly. That's an uncomfortable kind of success.

**00:03:47 — Alex:** I know. It's the copy you wish you didn't need. But other data might be recoverable, and the record of the damage helps us investigate. I wouldn't want the word broken to make somebody throw away the only copy we have.

**00:04:00 — Maya:** Neither would I. But if the backup saw a damaged page, I want that job to fail. Somebody should have been looking at this before the original server disappeared.

**00:04:09 — Alex:** Would you want it to stop copying at the first bad page? We might still preserve other data from that server.

**00:04:15 — Maya:** I'm asking for the pager to go off. Why does that require throwing the copy away?

**00:04:20 — Alex:** It doesn't. I heard stop the backup. You meant stop telling the team that recovery is covered.

**00:04:26 — Maya:** Yes. Keep what we can save. But if our only alert watches the command's exit status, a warning can disappear under a successful backup. Whoever arrives tomorrow reads the reassuring part.

**00:04:38 — Alex:** Then we need to decide what happens to that warning. In this release, enabling the backup command's checksum-error option actually terminates the backup when validation fails. That's one policy. Finishing the capture and separately alerting on the integrity problem is another.

**00:04:54 — Maya:** I'd accept continuing capture if that separate alert is real. It needs to identify this backup and reach whoever can investigate. I don't want a warning counter that everyone assumes somebody else watches.

**00:05:06 — Alex:** Would you choose continuing capture before we'd tested that alert?

**00:05:10 — Maya:** No. That's where I'm less comfortable than you are. The next person will still see a successful job. Until we know the warning reaches them, I'd choose the failed job and accept the interrupted capture.

**00:05:21 — Alex:** I still hesitate, because we may never get another chance to copy that server. But I can't give you a person who's actually watching the warning either.

**00:05:30 — Maya:** Then that's the thing to test before we rely on that policy. Use a known warning in an isolated exercise: does it reach the person responsible, and does the recovery record say this backup needs investigation? Either way, we keep the saved evidence and investigate the damage.

**00:05:47 — Alex:** Back on our recovery call, the original server is still unavailable. We have a saved backup with recorded damage, and the replacement is incomplete because restore stopped. Changing tomorrow's alert won't recover today's orders.

**00:06:01 — Maya:** Right. And the operator can see a setting that turns this restore error into a warning. I can imagine the pressure to use it. Everyone wants the order desk working again.

**00:06:11 — Alex:** It lets the restore continue past that reported condition. It doesn't repair the damaged page. A completed restore after that change would still need investigation before anyone trusted the database with orders.

**00:06:24 — Maya:** Would you ever use that option?

**00:06:25 — Alex:** Potentially, on a separate recovery copy, to investigate or salvage what we can. I would keep that purpose explicit. Extracting some recoverable data and reopening the order desk are different decisions.

**00:06:38 — Maya:** The person at the desk doesn't need a database process to be running. They need to know which orders are there and whether they can act on them. We don't have that answer just because a command finished.

**00:06:49 — Alex:** No. And rerunning the same restore without changing the evidence won't make that checksum warning go away. We need a different recovery starting point, or an investigation into this copy. The red result has given us a reason to stop treating it as routine.

**00:07:04 — Maya:** Suppose we have an older backup without the recorded page errors. I'd look at that before overriding the restore check. But older sounds like telling the business to lose everything since then.

**00:07:16 — Alex:** It doesn't necessarily mean that. An older physical backup can be brought forward using PostgreSQL's archived change records, the write-ahead log. We need the continuous usable log sequence for that recovery. Having an older backup on the list doesn't prove we have it.

**00:07:32 — Maya:** So I'd ask what we can actually recover to before promising the order desk a cutoff time. The timestamp on a backup is one piece of that answer.

**00:07:41 — Alex:** Right. The order desk still can't open, and someone has to tell them that. I'd rather give them an honest account of what we're checking than a reassuring backup timestamp.

**00:07:50 — Maya:** Which makes me want a quieter version of this conversation before there's an order desk waiting. If we run a restore drill, where do you stop the timer?

**00:07:59 — Alex:** After we check that the recovered application can do the job we're restoring it for. For these orders, that means inspecting the expected records and checking a representative order workflow in isolation. PostgreSQL's recovery instructions explicitly separate finishing recovery from inspecting the contents and allowing users back in.

**00:08:18 — Maya:** I'd get the order desk to help choose that check. Engineers can prove the database answers a query. The people packing boxes know which missing detail would send the wrong thing to a customer.

**00:08:29 — Alex:** And if the drill stops at the same checksum error, that's a finding to investigate. We don't keep rerunning it until we can put a green tick in the report.

**00:08:38 — Maya:** This week, take one recent backup through an isolated restore drill. Bring its warning record with it. Check the recovered application state before calling the drill complete, and record anything that stops you.

**00:08:50 — Alex:** Your open-first box is doing more work than I gave it credit for.

**00:08:53 — Maya:** The sheets are in there. Future me can negotiate for plates tomorrow.