learning_ai_invt_trdg/backend/runbooks/reconciliation-exit-backfill.md

# Reconciliation EXIT Backfill Rollout Checklist

## Scope Guardrails
- No entry/exit/risk/signal logic changes.
- Data-only repair: insert compensating `EXIT` rows only.
- No deletes. Rollback uses status updates only.
- Backfill must be paused-mode gated and auditable by `batch_id`.

## Feature Flags (default safe state)
- `ENABLE_RECON_EXIT_BACKFILL=false`
- `RECON_EXIT_BACKFILL_DRY_RUN=true`
- `RECON_EXIT_BACKFILL_REQUIRE_PAUSE=true`
- `RECON_EXIT_BACKFILL_DUST_ABS_QTY=0.001`
- `RECON_EXIT_BACKFILL_DUST_REL_PCT=0.002`
- `RECON_EXIT_BACKFILL_LOOKBACK_HOURS=72`
- `EXCHANGE_STATE_MISMATCH_THROTTLE_MS=300000`

## 1. Detect Affected `trade_id` Values
```sql
WITH lifecycle AS (
  SELECT
    profile_id,
    symbol,
    trade_id,
    SUM(CASE WHEN action = 'ENTRY' THEN COALESCE(quantity, qty, 0) ELSE 0 END) AS entry_qty,
    SUM(CASE WHEN action = 'EXIT'  THEN COALESCE(quantity, qty, 0) ELSE 0 END) AS exit_qty
  FROM orders
  WHERE profile_id = '<<PROFILE_ID>>'
    AND status IN ('filled', 'partially_filled', 'partially-filled')
    AND trade_id IS NOT NULL
  GROUP BY profile_id, symbol, trade_id
)
SELECT
  profile_id,
  symbol,
  trade_id,
  entry_qty,
  exit_qty,
  (entry_qty - exit_qty) AS open_qty
FROM lifecycle
WHERE (entry_qty - exit_qty) > 0
ORDER BY symbol, trade_id;
```

## 2. Allow Criteria for Backfill `EXIT` Row
Backfill is allowed only when all are true:
- Profile is paused (`/internal/trading/status` returns `PAUSED`).
- Exchange is flat for symbol.
- No pending lifecycle blocker row exists for that `trade_id`.
- Close qty is supported by exchange fill evidence OR unresolved remainder is dust.
- Idempotency key resolves to unique deterministic `order_id`.

## 3. Idempotency Rules
- Deterministic `order_id`:
  - `BFILL-<md5(profile_id:trade_id:exchange_order_id:filled_at)>`
- Insert path:
  - `ON CONFLICT (order_id) DO NOTHING` (implemented via upsert + `ignoreDuplicates`).
- Re-runs are safe: existing backfill `order_id` rows are skipped.
- Exchange `client_order_id` (if present) is stored in `reconciliation_backfill_audit.exchange_client_order_id`.

## 4. Dust Threshold Handling
- Dust threshold per trade:
  - `MAX(dust_abs_qty, open_qty * dust_rel_pct)`
- Default:
  - `dust_abs_qty = 0.001`
  - `dust_rel_pct = 0.2%`
- Auto-close allowed:
  - evidence-covered qty always
  - remainder only if `remainder <= threshold`
- No-Go:
  - `remainder > threshold` with missing exchange fill evidence.

## 5. Dry-Run Execution and Verification
1. Set:
   - `ENABLE_RECON_EXIT_BACKFILL=true`
   - `RECON_EXIT_BACKFILL_DRY_RUN=true`
   - `RECON_EXIT_BACKFILL_REQUIRE_PAUSE=true`
2. Pause trading from admin.
3. Let one reconciliation cycle run.
4. Verify audit-only effect:
```sql
SELECT
  batch_id,
  profile_id,
  symbol,
  trade_id,
  decision,
  reason,
  filled_qty,
  backfill_order_id,
  created_at
FROM reconciliation_backfill_audit
WHERE profile_id = '<<PROFILE_ID>>'
ORDER BY created_at DESC
LIMIT 200;
```
5. Confirm `orders` table unchanged in dry-run:
```sql
SELECT COUNT(*) AS dry_run_backfill_rows
FROM orders
WHERE profile_id = '<<PROFILE_ID>>'
  AND order_id LIKE 'BFILL-%';
```

## 6. Go / No-Go Before Apply
Go only if:
- Pause state confirmed.
- Dry-run shows expected `DRY_RUN` or `PENDING_APPLY` candidates.
- No unexpected `NO_GO` due to non-flat exchange or pending blockers.
- Candidate qty aligns with known exchange fills.
- Audit table writes succeed.

No-Go if any:
- Exchange not flat for target symbol.
- Large unmatched remainder (`missing_fill_evidence_for_large_remainder`).
- Audit table unavailable/write failure.
- Pending blocker rows for target `trade_id`.

## 7. Apply Step (still paused)
1. Set `RECON_EXIT_BACKFILL_DRY_RUN=false`.
2. Run one reconciliation cycle.
3. Validate inserted rows:
```sql
SELECT
  order_id,
  profile_id,
  symbol,
  trade_id,
  action,
  status,
  qty,
  price,
  filled_at,
  created_at
FROM orders
WHERE profile_id = '<<PROFILE_ID>>'
  AND order_id LIKE 'BFILL-%'
ORDER BY created_at DESC;
```

## 8. Post-Fix Validation (ghost positions removed)
Open lifecycle should be zero/near-zero:
```sql
WITH lifecycle AS (
  SELECT
    profile_id,
    symbol,
    trade_id,
    SUM(CASE WHEN action = 'ENTRY' THEN COALESCE(quantity, qty, 0) ELSE 0 END) AS entry_qty,
    SUM(CASE WHEN action = 'EXIT'  THEN COALESCE(quantity, qty, 0) ELSE 0 END) AS exit_qty
  FROM orders
  WHERE profile_id = '<<PROFILE_ID>>'
    AND status IN ('filled', 'partially_filled', 'partially-filled')
    AND trade_id IS NOT NULL
  GROUP BY profile_id, symbol, trade_id
)
SELECT *
FROM lifecycle
WHERE (entry_qty - exit_qty) > 0.000001
ORDER BY symbol, trade_id;
```

Mismatch noise check (audit + events):
```sql
SELECT
  decision,
  COUNT(*) AS rows
FROM reconciliation_backfill_audit
WHERE profile_id = '<<PROFILE_ID>>'
GROUP BY decision
ORDER BY decision;
```

## 9. Rollback (Reversible, Non-Destructive)
Rollback is status-only, never delete:
```sql
UPDATE orders
SET
  status = 'canceled',
  updated_at = now()
WHERE order_id IN (
  SELECT backfill_order_id
  FROM reconciliation_backfill_audit
  WHERE batch_id = '<<BATCH_ID>>'
    AND decision IN ('APPLIED', 'SKIP_EXISTING')
    AND backfill_order_id IS NOT NULL
);
```
Record rollback marker:
```sql
UPDATE reconciliation_backfill_audit
SET reverted_at = now()
WHERE batch_id = '<<BATCH_ID>>';
```