Your Backups Probably Don't Work: What a Restore Drill Shows

Most small SaaS teams have backups. Very few have restored one. I'm Nazmul Khalid, a lead software engineer who builds and runs SaaS products for founders, and on every system I run the backup isn't done until a restore has been tested on a schedule.

Here's what that looked like on HMS, a multi-tenant hospital management system I architected for WebCode Technology, and what the drills found that a green backup job never would have.

Why most backups fail when you need them

Most backups fail because nobody has ever restored one. The backup job says "success", which only means a file was written. Whether that file brings the product back is a separate question that usually gets answered for the first time during an outage.

The ways it goes wrong are rarely dramatic. The backup is on the same server that just died. The copy elsewhere is a day old. The restore needs a password that only one person had. The database won't start because the new server has different settings. None of these show up until you try.

What a restore drill is

A restore drill is a scheduled rehearsal: take the latest backup, restore it somewhere disposable, check the result, and record whether it passed. It turns "we have backups" into "we restored one on Sunday and it matched".

On HMS the setup is:

• Every database change is archived within a minute (Postgres write-ahead log, or WAL), so at most 60 seconds of writes can be lost on the machine itself.
• A nightly full backup at 02:15, keeping the last seven, copied to a storage bucket off the server (S3-compatible object storage).
• A drill every Sunday at 03:30 that restores each database from the bucket, not from the local copy, into a throwaway container and checks it.
• Every run is recorded, and a failed run emails the on-call address with the output of the step that failed. A run that failed without anyone being told counts as a problem of its own.

The status check is simple enough to watch from a monitor: last nightly backup within 36 hours, last off-site drill within 8 days, last off-site push within 30 minutes.

The drill that could never fail

The first problem the drills found was in the drill itself. It compared row counts per table before and after a restore. But it connected as a database role that row-level security was filtering, so it counted zero rows on both sides and reported a match. Any restore would have passed, even an empty one.

A check that can't fail isn't a check. The fix was to run the drill's counts as an admin user that sees every row, and then to prove the check could fail: plant known data, restore it, and confirm the counts and hashes matched.

If you take one thing from this article, take that one. Before you trust a backup check, make it fail on purpose once.

What losing the whole server showed

The weekly drill doesn't cover the worst case: the server is gone, disk and all. So I rehearsed that separately, with one command that uses nothing from the lost machine. It pulls the latest backup and every change after it from the bucket, restores each database, starts the API and stops the clock when a hospital login works.

The first rehearsal took 164 seconds, and it found two more problems.

The download was the downtime, not the restore. Of those 164 seconds, one database spent 145 pulling change files and 11 replaying them. Postgres writes change files in fixed 16 MB pieces, and with an archive every minute, a 209 MB database had about 6.5 GB of change files in the bucket. Compressing them before upload (a nearly empty piece shrinks to a few KB) and sending only what the bucket lacks brought the same rehearsal down to 42 seconds.

The off-site copy was a day behind. Changes were archived every minute, but only sent off the machine with the nightly backup. So on the machine you'd lose a minute at most, and if you lost the machine you'd lose up to a day. Changes now go to the bucket every five minutes, and the status flags the bucket if it falls more than 30 minutes behind. In the rehearsal, the restored data was from 3 to 9 seconds before the loss.

A third, smaller one: the restore used to read database settings from the live server, which in a real disaster doesn't exist. Those settings now travel to the bucket with the backups.

What has to live somewhere else

A restore also needs things that aren't data. When I wrote the recovery runbook, the list was short but every item was essential:

• The code at the deployed version, in git.
• The bucket's address and credentials, in a password manager, not only on the server. Without them the backups exist and can't be reached.
• The app's secrets (signing keys, payment and SMS gateway credentials), in the same password manager.
• Uploaded files (documents, PDFs), which were never on the server in the first place: they live in object storage.

If any of those only exist on the machine that died, the backup doesn't matter.

What a small SaaS should check this week

You don't need a hospital system to do this. Five questions cover most of it:

1. When did you last restore a backup, and to where? If the answer is "never", that's the first job.
2. Is there a copy off the server, and how far behind is it?
3. Does anything alert you when a backup or a restore fails, or doesn't run at all?
4. Could someone else restore it using only a written runbook and a password manager?
5. How long would it take? Measure it once, so the number is real.

HMS is an MVP that hasn't gone into production yet, and these numbers come from rehearsals on a staging setup, not a real outage. That's the point: the time to find these problems is before a hospital, or your first paying customer, depends on them.

If you'd like this set up on your own product, backups, a weekly drill and alerts are part of my DevOps work for startups, and server care with backup checks starts from $150 a month per server. You can also see how I work and what it costs.