Off-site backup storage (outside primary provider)

Severity: Recommended (Critical when serious money involved)

Backups must be stored with a different provider than production. If AWS has a catastrophic failure, AWS backups won't help you.

Check automatically:

1. Look for external backup destinations:

# Check for different provider storage in backup scripts/config
grep -rE "backblaze|b2://|wasabi|r2://|idrive|rsync\.net|minio" --include="*.tf" --include="*.yml" --include="*.sh" 2>/dev/null

# If AWS production, check for GCP/Azure backup targets
grep -rE "gs://|google.*storage|azure.*blob|wasb://" --include="*.tf" --include="*.yml" --include="*.sh" 2>/dev/null

# If GCP production, check for AWS/Azure backup targets
grep -rE "s3://|aws.*s3" --include="*.tf" --include="*.yml" --include="*.sh" 2>/dev/null

2. Check for S3 cross-region replication (partial credit - still AWS):

# This is better than nothing but doesn't satisfy "outside provider"
aws s3api get-bucket-replication --bucket BACKUP_BUCKET 2>/dev/null

3. Check for backup export jobs:

# RDS export to S3 (then synced elsewhere?)
aws rds describe-export-tasks --query "ExportTasks[].{ID:ExportTaskIdentifier,Status:Status,S3Bucket:S3Bucket}" --output table 2>/dev/null

# Look for sync scripts
grep -rE "rclone|aws s3 sync|gsutil rsync" --include="*.sh" --include="*.yml" 2>/dev/null

4. Check Terraform for external backup resources:

# Backblaze B2
grep -rE "b2_bucket|backblaze" --include="*.tf" 2>/dev/null

# Cloudflare R2
grep -rE "cloudflare_r2" --include="*.tf" 2>/dev/null

# Wasabi
grep -rE "wasabi" --include="*.tf" 2>/dev/null

5. Check for read replica with different provider (rare but ideal):

# Self-hosted replica pulling from managed DB
grep -rE "primary_conninfo.*amazonaws|primary_conninfo.*cloud\.google" --include="*.conf" 2>/dev/null

Ask user:

• "If your primary cloud provider (AWS/GCP/Azure) had a catastrophic failure, where would you restore from?"
• "Are backups stored with a completely separate provider?"
• "What's your process for syncing backups to external storage?"

Low-cost external backup options:

• Backblaze B2 (~$5/TB/month)
• Wasabi (~$7/TB/month, no egress fees)
• Cloudflare R2 (no egress fees)
• rsync.net (SSH-based, good for pg_dump files)
• Scheduled SQL dumps uploaded via cron

Cross-reference with:

• HA-003 (backups exist - this item is about WHERE they're stored)
• HA-005 (PITR - off-site should include WAL archives if using PITR)
• Section 34 (Disaster Recovery - off-site enables DR)

Pass criteria:

• Backups exist with a different provider than production
• Sync process is automated (not manual)
• External backup freshness verified (not stale)
• Restoration from external backups tested

Fail criteria:

• All backups within same cloud provider (even if cross-region)
• "We use S3 cross-region replication" (still AWS-dependent)
• Manual sync process that hasn't run in months
• No plan for provider-wide outage

Evidence to capture:

• External backup destination (provider, bucket/path)
• Sync mechanism (rclone, custom script, managed service)
• Sync frequency
• Last verified external backup timestamp
• Last external restore test (if any)

---