learning_ai_invt_trdg/backend/runbooks/reconciliation-exit-backfill.md

Reconciliation EXIT Backfill Rollout Checklist

Scope Guardrails

• No entry/exit/risk/signal logic changes.
• Data-only repair: insert compensating EXIT rows only.
• No deletes. Rollback uses status updates only.
• Backfill must be paused-mode gated and auditable by batch_id.

Feature Flags (default safe state)

• ENABLE_RECON_EXIT_BACKFILL=false
• RECON_EXIT_BACKFILL_DRY_RUN=true
• RECON_EXIT_BACKFILL_REQUIRE_PAUSE=true
• RECON_EXIT_BACKFILL_DUST_ABS_QTY=0.001
• RECON_EXIT_BACKFILL_DUST_REL_PCT=0.002
• RECON_EXIT_BACKFILL_LOOKBACK_HOURS=72
• EXCHANGE_STATE_MISMATCH_THROTTLE_MS=300000

1. Detect Affected trade_id Values

WITH lifecycle AS (
  SELECT
    profile_id,
    symbol,
    trade_id,
    SUM(CASE WHEN action = 'ENTRY' THEN COALESCE(quantity, qty, 0) ELSE 0 END) AS entry_qty,
    SUM(CASE WHEN action = 'EXIT'  THEN COALESCE(quantity, qty, 0) ELSE 0 END) AS exit_qty
  FROM orders
  WHERE profile_id = '<<PROFILE_ID>>'
    AND status IN ('filled', 'partially_filled', 'partially-filled')
    AND trade_id IS NOT NULL
  GROUP BY profile_id, symbol, trade_id
)
SELECT
  profile_id,
  symbol,
  trade_id,
  entry_qty,
  exit_qty,
  (entry_qty - exit_qty) AS open_qty
FROM lifecycle
WHERE (entry_qty - exit_qty) > 0
ORDER BY symbol, trade_id;

2. Allow Criteria for Backfill EXIT Row

Backfill is allowed only when all are true:

• Profile is paused (/internal/trading/status returns PAUSED).
• Exchange is flat for symbol.
• No pending lifecycle blocker row exists for that trade_id.
• Close qty is supported by exchange fill evidence OR unresolved remainder is dust.
• Idempotency key resolves to unique deterministic order_id.

3. Idempotency Rules

• Deterministic order_id:
  • BFILL-<md5(profile_id:trade_id:exchange_order_id:filled_at)>
• Insert path:
  • ON CONFLICT (order_id) DO NOTHING (implemented via upsert + ignoreDuplicates).
• Re-runs are safe: existing backfill order_id rows are skipped.
• Exchange client_order_id (if present) is stored in reconciliation_backfill_audit.exchange_client_order_id.

4. Dust Threshold Handling

• Dust threshold per trade:
  • MAX(dust_abs_qty, open_qty * dust_rel_pct)
• Default:
  • dust_abs_qty = 0.001
  • dust_rel_pct = 0.2%
• Auto-close allowed:
  • evidence-covered qty always
  • remainder only if remainder <= threshold
• No-Go:
  • remainder > threshold with missing exchange fill evidence.

5. Dry-Run Execution and Verification

1. Set:
   • ENABLE_RECON_EXIT_BACKFILL=true
   • RECON_EXIT_BACKFILL_DRY_RUN=true
   • RECON_EXIT_BACKFILL_REQUIRE_PAUSE=true
2. Pause trading from admin.
3. Let one reconciliation cycle run.
4. Verify audit-only effect:

SELECT
  batch_id,
  profile_id,
  symbol,
  trade_id,
  decision,
  reason,
  filled_qty,
  backfill_order_id,
  created_at
FROM reconciliation_backfill_audit
WHERE profile_id = '<<PROFILE_ID>>'
ORDER BY created_at DESC
LIMIT 200;

5. Confirm orders table unchanged in dry-run:

SELECT COUNT(*) AS dry_run_backfill_rows
FROM orders
WHERE profile_id = '<<PROFILE_ID>>'
  AND order_id LIKE 'BFILL-%';

6. Go / No-Go Before Apply

Go only if:

• Pause state confirmed.
• Dry-run shows expected DRY_RUN or PENDING_APPLY candidates.
• No unexpected NO_GO due to non-flat exchange or pending blockers.
• Candidate qty aligns with known exchange fills.
• Audit table writes succeed.

No-Go if any:

• Exchange not flat for target symbol.
• Large unmatched remainder (missing_fill_evidence_for_large_remainder).
• Audit table unavailable/write failure.
• Pending blocker rows for target trade_id.

7. Apply Step (still paused)

1. Set RECON_EXIT_BACKFILL_DRY_RUN=false.
2. Run one reconciliation cycle.
3. Validate inserted rows:

SELECT
  order_id,
  profile_id,
  symbol,
  trade_id,
  action,
  status,
  qty,
  price,
  filled_at,
  created_at
FROM orders
WHERE profile_id = '<<PROFILE_ID>>'
  AND order_id LIKE 'BFILL-%'
ORDER BY created_at DESC;

8. Post-Fix Validation (ghost positions removed)

Open lifecycle should be zero/near-zero:

WITH lifecycle AS (
  SELECT
    profile_id,
    symbol,
    trade_id,
    SUM(CASE WHEN action = 'ENTRY' THEN COALESCE(quantity, qty, 0) ELSE 0 END) AS entry_qty,
    SUM(CASE WHEN action = 'EXIT'  THEN COALESCE(quantity, qty, 0) ELSE 0 END) AS exit_qty
  FROM orders
  WHERE profile_id = '<<PROFILE_ID>>'
    AND status IN ('filled', 'partially_filled', 'partially-filled')
    AND trade_id IS NOT NULL
  GROUP BY profile_id, symbol, trade_id
)
SELECT *
FROM lifecycle
WHERE (entry_qty - exit_qty) > 0.000001
ORDER BY symbol, trade_id;

Mismatch noise check (audit + events):

SELECT
  decision,
  COUNT(*) AS rows
FROM reconciliation_backfill_audit
WHERE profile_id = '<<PROFILE_ID>>'
GROUP BY decision
ORDER BY decision;

9. Rollback (Reversible, Non-Destructive)

Rollback is status-only, never delete:

UPDATE orders
SET
  status = 'canceled',
  updated_at = now()
WHERE order_id IN (
  SELECT backfill_order_id
  FROM reconciliation_backfill_audit
  WHERE batch_id = '<<BATCH_ID>>'
    AND decision IN ('APPLIED', 'SKIP_EXISTING')
    AND backfill_order_id IS NOT NULL
);

Record rollback marker:

UPDATE reconciliation_backfill_audit
SET reverted_at = now()
WHERE batch_id = '<<BATCH_ID>>';