RR-002 recommended rollback-strategy

Rollback tested regularly

Rollback procedure tested at least quarterly with multiple team members

Question to ask

"When was the last time you actually ran a rollback drill?"

Verification guide

Severity: Recommended

An untested rollback procedure may not work when you need it. Regular testing ensures the team is practiced and the procedure is current.

Check automatically:

  1. Look for rollback testing records:
# Check for rollback testing in CI/CD or docs
grep -riE "rollback.*test|test.*rollback|disaster.*drill|recovery.*test" .github/ docs/ runbooks/ --include="*.yml" --include="*.md" 2>/dev/null

# Check for scheduled drills
grep -riE "drill|exercise|regularly|monthly|quarterly" docs/ runbooks/ --include="*.md" 2>/dev/null | grep -iE "rollback|recovery|disaster"

Ask user:

  • "When was the last time you tested a rollback? (Last month = good, never = bad)"
  • "Do you have scheduled disaster recovery drills?"
  • "Has everyone on the team done a rollback at least once?"

Cross-reference with:

  • RR-001 (documented procedure) - can't test what isn't documented
  • RR-006 (recovery procedure tested) - same principle applies to full recovery
  • Section 35 (incident response) - drills should include rollback scenarios

Pass criteria:

  • Rollback tested at least quarterly (or after major infra changes)
  • Multiple team members have done it (not just one person)
  • Last test date is documented

Fail criteria:

  • Never tested ("we'll figure it out when we need it")
  • Only one person has ever done it
  • Last test was > 1 year ago

Evidence to capture:

  • Date of last rollback test
  • How many team members have performed a rollback
  • Whether there's a schedule for regular testing

Section

34. Rollback & Recovery

API & Security