Personal explorations · Orchestration · verification · operations
A backup you have not restored is a hope
The nightly pipeline that checks my homelab’s database backup by actually restoring part of it.
Personal project · data platform
Restore it, then believe it
The dump runs at 03:00 and the check at 04:00. It confirms the file is recent and intact, loads one database into a scratch copy, compares it with live, and drops the copy.
A backup nobody has restored is a hope. This turns it into a fact every morning.
Catch the job that stopped
The newest dump is 31 hours old, so the nightly job has not run. Nothing errored; it simply stopped. The first check fails, the rest are skipped, and an alert goes out.
26 hours allows one late run to pass while a dead schedule still gets caught.
The failure only a restore can find
A dump taken with a newer database client looks perfect. The archive is intact and all six databases are there. Then the restore stops on its first statement, because the dump carries a setting the older server rejects.
This is the case the restore step exists for. Every check before it said yes.
A check nobody hears is not a check
A different job, the nightly blog comment summaries, failed 483 times in a row from March to August. Its alerts went to a notification service that had been deleted, so every failure went nowhere.
Failures now go to a channel a person reads. When I build a check, I also check where its alarm lands.
The steps, thresholds and the 483 count come from the homelab’s own pipeline code. Database names are left out.
Two machines, one platform
At home I run a small two-node Kubernetes cluster: a mini PC and a GPU workstation. It hosts my blog, my chat agent, Airflow, monitoring, and LLM tracing, all deployed through GitOps with encrypted secrets and single sign-on in front of every interactive service.
It is where I practice the parts of data engineering that only show up when you are the one on call.
The backup that looked healthy
A scheduled job dumps the databases every night at 03:00. For a long time that was the whole story, and that was the problem. A backup can be on time, the right size, and still impossible to restore.
Restore it, every night
At 04:00 a pipeline checks the newest dump. Is it under 26 hours old? Does the archive pass an integrity test? Are all six databases inside it?
Then it does the part a cron job cannot. It really restores one database into a scratch database, stopping at the first error, compares the result with the live database, and drops the scratch copy whatever happens.
Why the restore step earns its keep
The failure it catches looks perfect. A dump taken with a newer Postgres client contains a setting the older server rejects. The file is fresh, intact, and complete. It fails only when you try to restore it, which is exactly when you can least afford to find out.
Play “The wrong client” above and watch every early check pass.
Put the check where the failure is not
The dump stays a plain scheduled job on purpose. Airflow depends on the same database server, so an Airflow-driven backup could not run during the failure it exists for. Verification has the opposite shape: if Airflow is down, I lose the check, not the backup.
Alert on the alerting
A different job taught me the last lesson. Comment summaries failed 483 times in a row, from March to August, and nobody knew. Its alerts went to a notification service that had been deleted. Failures now go somewhere a person reads, and a quiet channel is something I check, not something I trust.
Read how the platform is built ↗A green status is a claim. A restore is evidence.
Let’s talk ↗