Your preparation
0 of 0 safeguards readyMake the next decision with evidence
Keep the business's essential workflow moving, preserve requests for replay, and distinguish provider recovery from your own integration failures.
Capture before evidence disappears
- Capture provider status, request IDs, response codes, timeouts, webhook gaps, queue depth, affected features, and the last known-good transaction.
- List data held only by the vendor, pending writes, callbacks, exports, tokens, and manual work performed during the outage.
- Measure customer and internal impact by workflow rather than treating every vendor error as equal.
Decisions that change the response
| Question | Act when | Action |
|---|---|---|
| Queue, degrade, or disable? | The operation's idempotency, ordering, expiry, and customer expectation are known. | Queue safe work, degrade read paths, and disable actions that could duplicate or lose state. |
| Use a substitute? | The fallback has compatible data, security, legal, and operational behavior. | Switch only the pre-approved workflow; avoid improvised data uploads to a new vendor. |
Proof that recovery worked
- New requests follow the declared degraded behavior and customers see honest status.
- Queued work replays once, in order where required, with duplicate and failure reconciliation.
- Provider recovery is confirmed through your own end-to-end transaction, not status alone.
Controls to put in place
- Define timeout, retry, idempotency, queue, and circuit-breaker behavior for each critical vendor operation.
- Export irreplaceable vendor data and keep alternate contact and status routes.
- Practice the degraded workflow and vendor recovery reconciliation.
Return timeouts and delayed webhooks from a vendor stub. Enter degraded mode, queue safe work, communicate impact, restore the dependency, and reconcile duplicates.
Use provider support and account contacts when the public status lacks your failure; involve legal or procurement when contractual service, data access, or exit rights are at risk.
What this means
The provider may recover quickly, but your business still needs a decision: wait, degrade gracefully, switch to manual work, or activate a fallback. Avoid repeatedly changing your system while the provider is unstable.
Separate the provider’s public status from your actual customer impact. One may recover before the other.
Warning signs
- Provider requests time out, fail, or return partial results.
- Queues and retries grow while your own infrastructure appears healthy.
- The provider status page reports an incident or maintenance.
- Customers report missing email, payments, files, authentication, or support replies.
- Your integration starts duplicating work after retries.
Recover now
First 15 minutes
- Confirm the dependency. Compare provider status, your logs, a direct test, and reports from another network.
- Stop retry amplification. Add backoff, pause nonessential jobs, and cap queues before they overload your systems.
- Protect customer actions. Preserve accepted requests locally and show a truthful pending state. Do not say something succeeded when only the provider call started.
- Activate the smallest fallback. Use cached reads, manual processing, a secondary channel, or a temporary feature disablement.
- Open one incident timeline. Record provider updates, your impact, queue size, customer messages, and changes.
Today
- Decide how long you can wait before switching to a deeper fallback.
- Keep provider-bound work idempotent so replay does not duplicate charges, emails, orders, or files.
- Update customers when the outage blocks an important action.
- Reconcile queued and partially completed work after the provider recovers.
- Remove temporary bypasses carefully and watch for delayed webhooks or duplicate events.
Verify recovery
- The provider status is healthy and direct integration tests succeed.
- Queues are draining at a controlled rate.
- Accepted customer actions are completed exactly once.
- Missing, delayed, and duplicate records have been reconciled.
- Temporary manual work and bypasses are closed.
- Customer-facing status matches actual service.
Prepare now
Dependency
- Every critical provider has an owner, status page, support route, and incident threshold.
- The product has explicit behavior for provider timeout, partial success, and retry.
- Critical operations are idempotent.
- A local record exists before sending irreplaceable work to a provider.
Fallback
- The business has a manual or degraded path for the most important customer action.
- Queues, retry limits, and circuit breakers are tested.
- Provider data needed for continuity is exported regularly.
- Customer status can be updated without the failed provider.
Practice
- A drill has disabled the provider integration and safely replayed queued work.
Common mistakes
- Retrying every request immediately. You overload both systems and create duplicates.
- Trusting the public status page alone. Your account or region may still be broken.
- Switching providers during a short outage. A rushed migration can create a second incident.
- Losing accepted customer actions. Save them before the provider call.
- Replaying without idempotency. Recovery creates duplicate charges or messages.
Sources
Reported by the people who handled it
Lessons from real incidents
These are operator postmortems and official incident reports, condensed into one lesson you can reuse.Summary of the June 8 outage
TL;DRA valid customer configuration activated a latent software bug and caused 85% of Fastly's network to return errors. Fastly detected the disruption within a minute, isolated and disabled the triggering configuration, and restored most service within 49 minutes.
Learning to reuseA routine provider-side change can trigger a systemic failure. Detect provider impact from your own critical paths, bound retries, and have a degraded mode that does not require diagnosing the vendor's root cause first.
Read the full report(opens in a new tab)