2.1 KiB
2.1 KiB
Capital Invariant Violation Runbook
Incident description
The ledger calculation allocated - reserved_for_orders - reserved_for_positions + realized_pnl dropped below zero for a profile, indicating over-allocation or stale reservations.
Symptoms
capitalInvariantViolationsmetric increments in/metricsand/internal/health.- Critical log entry from the capital watchdog describing the negative delta and profile_id.
- Dashboard may show unexpected negative available capital or trading loop halting entries for that profile.
Metrics to check
/internal/health?capitalInvariantViolations,tradingLoopHealthy,lockContentionCount./metrics?capital_invariant_violations_total,available_capital_gauge,reconciliation_mismatch_count.
Immediate mitigation
- Identify affected profile_id from the log or metric labels.
- Confirm no new ENTRY orders are being placed for that profile (tradingLoopHealthy may be false).
- Trigger a reconciliation cycle for the profile (if manual trigger exists, otherwise wait for scheduled run). Ensure reconciliation lock is available before forcing.
- Notify downstream ops that trading for the profile is paused until invariant clears.
Expected self-recovery
- Reconciliation loop recalculates reservations from exchange open orders/positions and updates ledger fields, restoring the invariant.
- Capital watchdog logs the repair and metrics return to zero once
available_capitalis non-negative.
When to escalate
- Metric stays non-zero after three reconciliation cycles or 15 minutes.
- Trading loop remains unhealthy while other profiles are functioning.
- Capital watchdog logs repeated violations for different profiles within an hour. Refer to docs/runbooks/invariant-violation.md for escalation steps.
What NOT to do
- Do not manually edit ledger rows via Supabase without consulting runbooks.
- Do not restart the trading loop: the rebuild is deterministic; restarting may reintroduce transient errors.
- Do not ignore the metric; unresolved violations can lead to over-leveraging other profiles.