Your preparation
0 of 0 safeguards readyMake the next decision with evidence
Stop corruption from spreading, preserve the earliest clean recovery paths, and choose repair or restore based on verified business consistency.
Capture before evidence disappears
- Capture engine errors, checksums, failed queries, affected pages or tables, replication state, storage health, schema, version, and the first known bad record.
- Preserve snapshots, replicas, logs, write-ahead logs, exports, and application events before automated retention or failover changes them.
- Map affected records to customers, transactions, derived indexes, caches, files, and external systems that may contain corroborating state.
Decisions that change the response
| Question | Act when | Action |
|---|---|---|
| Stop writes? | New writes can propagate or obscure corruption and no safe partition can be isolated. | Enter read-only or maintenance mode and protect replicas from promotion or replay. |
| Repair or restore? | Repair can prove logical consistency without discarding valid history. | Repair only in a clone; otherwise restore to a clean point and reconcile the missing interval. |
Proof that recovery worked
- Engine integrity checks, schema constraints, relationships, counts, totals, and sampled customer workflows pass.
- Replicas and backups are recreated from verified clean state rather than inheriting corruption.
- Every reconstructed or lost transaction in the gap has a documented business outcome.
Controls to put in place
- Run integrity checks and restore tests that include application-level totals and invariants.
- Keep point-in-time recovery and isolated copies beyond the likely corruption-detection delay.
- Use checksums, constraints, idempotent event records, and reconciliation against payment or vendor systems.
Corrupt synthetic rows in a staging copy. Detect the invariant failure, freeze writes, compare repair with point-in-time restore, reconcile the gap, and rebuild a replica.
Use database recovery specialists before in-place repair, replica promotion, or destructive maintenance when the corruption boundary is unknown.
What this means
The application may appear online while reading or writing incorrect data. Continuing normal traffic can spread corruption and make recovery harder.
Warning signs
- Integrity checks, replication, or database logs report corruption.
- Totals, relationships, or customer records contradict source events.
- Reads differ across replicas or repeated queries.
- Errors begin after a migration, restore, storage failure, or faulty job.
Recover now
First 15 minutes
- Stop or restrict writes that could spread the damage.
- Capture database, application, and infrastructure logs plus exact detection time.
- Identify the last known-good point and affected tables or records.
- Clone data for investigation instead of experimenting on the only production copy.
Today
- Choose repair, point-in-time recovery, or replay based on measured scope.
- Restore into an isolated environment and run integrity and business checks.
- Reconcile changes made after the restore point from trusted event sources.
- Reopen writes gradually and monitor invariant failures.
Verify recovery
- Database-native integrity checks pass.
- Business totals and relationships match independent records.
- Replicas agree and new writes remain consistent.
- A sample of affected customers is verified.
Prepare now
Access
- Destructive database actions require restricted roles and review.
Backups and evidence
- Point-in-time recovery and logical exports cover critical data.
- Business invariants are tested independently of database health.
Contacts and ownership
- A decision owner can pause writes and approve restoration.
Practice
- A recent restore test included application-level reconciliation.
Common mistakes
- Trusting a successful restore without checking business meaning.
- Running repair commands directly on the only copy.
- Leaving writes open while measuring scope.
Sources
Reported by the people who handled it
Lessons from real incidents
These are operator postmortems and official incident reports, condensed into one lesson you can reuse.October 21 post-incident analysis
TL;DRA network partition left GitHub's database clusters with conflicting writes across two sites. The team paused webhooks and Pages builds, chose to fail forward, restored and synchronized replicas, and processed queued work only after the serving topology was stable.
Learning to reuseState the priority explicitly: data integrity can matter more than feature availability or a fast return to green. Pause secondary writers and queues until the source of truth is settled.
Read the full report(opens in a new tab)