How you might know
- Customers report problems right after a release.
- Data looks missing or wrong.
- An AI agent or teammate made a change to the live system.
Do this now
Confirm the change is the likely cause
Check what changed and when the problems started. If they line up, it is likely the cause.
Check whether rollback is safe for data
Changes to the database may not be reversible. Ask: if we undo this, do we lose anything customers added since?
Take a backup or snapshot first
Copy the current state so you do not lose evidence or data.
Roll back the smallest thing that restores safety
Revert the specific release or feature, not the whole system, if you can.
Verify it worked
Test the main flows and check customer reports. Confirm monitoring is normal.
Tell people what happened
Customers and your team should know what changed and whether any data was affected.
Do not do this
- Do not roll back blindly if data changed in ways that cannot be undone.
- Do not skip verification.
- Do not let an agent fix its own mistake on live data without a person checking.
Who to tell
- Incident lead
- Anyone who relied on the new behavior
- Customers if data was affected
What to save before you clean up
- Deployment and migration logs
- A copy of the data before the rollback
In the next 24 hours
- Write the timeline while it is fresh.
- Decide how to re-release safely.
- Add a rollback plan to every risky release.
Official help for the platforms involved
Links open each provider's own documentation, so you do not have to search for it.
What to write in the post mortem afterwards
- Why the change shipped without a rollback plan
- How quickly rollback was decided
- How to test or gate risky changes
Finished this one?
Add it to What I learned and earn 30 Trust. You can open your list from the badge at the bottom of the page.