Deep health endpoint with dependency checks

Severity: Critical

Check automatically:

1. Search for deep health endpoint definitions:

   # Look for deep/ready/detailed health routes
   grep -rE "(health/deep|health/ready|health/detailed|healthcheck|readiness)" \
     --include="*.ts" --include="*.js" --include="*.py" --include="*.go" --include="*.rb"
   
   # Look for DB connectivity checks in health-related files
   grep -rE "(ping|connect|isAlive|isReady|checkConnection)" \
     $(find . -name "*health*" -type f 2>/dev/null) 2>/dev/null
   
   # Look for multiple service checks
   grep -rE "(database|redis|postgres|mysql|mongo|rabbit|kafka)" \
     $(find . -name "*health*" -type f 2>/dev/null) 2>/dev/null
   

2. Verify authentication requirement:

   # Should return 401/403 without auth
   curl -s -o /dev/null -w "%{http_code}" http://localhost:PORT/health/deep
   
   # Common auth patterns - one of these should work
   curl -s -o /dev/null -w "%{http_code}" -H "Authorization: Bearer TOKEN" http://localhost:PORT/health/deep
   curl -s -o /dev/null -w "%{http_code}" -H "X-Health-Token: TOKEN" http://localhost:PORT/health/deep
   curl -s -o /dev/null -w "%{http_code}" -H "X-API-Key: TOKEN" http://localhost:PORT/health/deep
   

3. Verify dependency checks (with valid auth):

   curl -s -H "X-Health-Token: TOKEN" http://localhost:PORT/health/deep
   

   Expected response should include status of each external service:

   {
     "status": "healthy",
     "checks": {
       "database": { "status": "healthy" },
       "redis": { "status": "healthy" }
     }
   }
   

4. Test failure reporting - stop a dependency and verify deep health reports it:

   # Stop a dependency
   docker compose stop redis
   
   # Deep health should return 503 with details
   curl -s -w "\nHTTP: %{http_code}" -H "X-Health-Token: TOKEN" http://localhost:PORT/health/deep
   
   # Restore
   docker compose start redis
   

5. Verify basic health still works when dependency is down:

   docker compose stop redis
   curl -s -o /dev/null -w "%{http_code}" http://localhost:PORT/health  # Should still be 200
   docker compose start redis
   

Cross-reference with:

• HEALTH-001 - basic health should still return 200 when dependencies are down
• Section 5 (Database) - deep health verifies DB connectivity
• Section 6 (Resilience) - validates graceful degradation behavior

Pass criteria:

• Deep health endpoint exists
• Requires authentication (token, API key, or header)
• Returns 401/403 without valid auth
• Checks all external services the app depends on (database, cache, queues, external APIs)
• Reports individual service status (not just overall pass/fail)
• Returns HTTP 503 when any dependency is unhealthy
• Response body includes which service is down and why
• Basic health (HEALTH-001) still works when dependencies are down

Fail criteria:

• No deep health endpoint
• Deep health is unauthenticated (exposes infrastructure details)
• Only checks "is app running" (same as basic health)
• Returns 200 when a dependency is down
• Doesn't report which specific service is down
• Crashes or hangs when a dependency is unavailable
• Missing checks for known dependencies

If endpoint found but no auth, ask user: "Deep health endpoint at [path] doesn't require authentication. This exposes infrastructure details (database status, service names, potential error messages). Is this intentional? Recommend protecting with a token or API key."

If no deep health found, ask user: "No deep health endpoint found. What external services does this app depend on? If any (database, Redis, external APIs), recommend adding a protected endpoint that verifies connectivity to each and returns 503 with details when any is unavailable."

Evidence to capture:

• Endpoint path (or none)
• Authentication method required (header name, token type)
• List of services checked
• Sample response when healthy
• Sample response when a dependency is down (should be 503)
• Whether basic health remains 200 when dependency is down

---