35 lines
2.1 KiB
Markdown
35 lines
2.1 KiB
Markdown
# Capital Invariant Violation Runbook
|
|
|
|
## Incident description
|
|
The ledger calculation `allocated - reserved_for_orders - reserved_for_positions + realized_pnl` dropped below zero for a profile, indicating over-allocation or stale reservations.
|
|
|
|
## Symptoms
|
|
- `capitalInvariantViolations` metric increments in `/metrics` and `/internal/health`.
|
|
- Critical log entry from the capital watchdog describing the negative delta and profile_id.
|
|
- Dashboard may show unexpected negative available capital or trading loop halting entries for that profile.
|
|
|
|
## Metrics to check
|
|
- `/internal/health` ? `capitalInvariantViolations`, `tradingLoopHealthy`, `lockContentionCount`.
|
|
- `/metrics` ? `capital_invariant_violations_total`, `available_capital_gauge`, `reconciliation_mismatch_count`.
|
|
|
|
## Immediate mitigation
|
|
1. Identify affected profile_id from the log or metric labels.
|
|
2. Confirm no new ENTRY orders are being placed for that profile (tradingLoopHealthy may be false).
|
|
3. Trigger a reconciliation cycle for the profile (if manual trigger exists, otherwise wait for scheduled run). Ensure reconciliation lock is available before forcing.
|
|
4. Notify downstream ops that trading for the profile is paused until invariant clears.
|
|
|
|
## Expected self-recovery
|
|
- Reconciliation loop recalculates reservations from exchange open orders/positions and updates ledger fields, restoring the invariant.
|
|
- Capital watchdog logs the repair and metrics return to zero once `available_capital` is non-negative.
|
|
|
|
## When to escalate
|
|
- Metric stays non-zero after three reconciliation cycles or 15 minutes.
|
|
- Trading loop remains unhealthy while other profiles are functioning.
|
|
- Capital watchdog logs repeated violations for different profiles within an hour.
|
|
Refer to docs/runbooks/invariant-violation.md for escalation steps.
|
|
|
|
## What NOT to do
|
|
- Do not manually edit ledger rows via Supabase without consulting runbooks.
|
|
- Do not restart the trading loop: the rebuild is deterministic; restarting may reintroduce transient errors.
|
|
- Do not ignore the metric; unresolved violations can lead to over-leveraging other profiles.
|