# Supabase / RPC Outage Runbook

## Incident description
Supabase rejects writes or RPCs fail, blocking lifecycle persistence, ledger updates, or reconciliation state.

## Symptoms
- RPC errors logged (e.g., `timeout`, `503`, `service_role` rejection).
- `/internal/health` reports `monitorLoopHealthy` or `tradingLoopHealthy` false.
- `bot_state.json` writes or Supabase insert logs fail.

## Metrics to check
- `/internal/health` ? `tradingLoopHealthy`, `monitorLoopHealthy`, `lockContentionCount`.
- `/metrics` ? `supabase_rpc_errors_total`, `lifecycle_persist_failures_total`.

## Immediate mitigation
1. Pause new ENTRY submissions until Supabase is writable.
2. Queue lifecycle persistence retries and avoid dropping them; persistent message store may be required.
3. Notify Supabase support and confirm any service-wide issues.
4. Monitor whether reconciliation loops can still run with read-only access.

## Expected self-recovery
- Supabase recovers; pending RPC retries succeed, and reconciliation cleans up backlog.
- Metrics for RPC errors fall back to zero.

## When to escalate
- Outage exceeds 10 minutes with STATUS still failing.
- Supabase confirms incident impacting service_role writes.
- Lost trades or ledger inconsistencies appear post-recovery.
Escalate via docs/runbooks/database-outage.md to platform reliability.

## What NOT to do
- Do not switch to a secondary database manually; the system expects single Supabase project.
- Do not delete pending lifecycle entries; they must be replayed once writes succeed.
- Do not ignore RPC failure alerts; they indicate blocked capital and lifecycle flows.
