# Trading Control State Persistence

## Where is the Pause/Resume State Stored?

The trading control state (PAUSED or RUNNING) is persisted in **three locations** to ensure reliability and recovery:

---

## 1. In-Memory (Primary Runtime State)

**Location**: `HealthTracker` singleton in `healthTracker.ts`

```typescript
// bytelyst-trading-bot-service/src/services/healthTracker.ts

export class HealthTracker {
    private tradingControl: TradingControlSnapshot = {
        mode: 'RUNNING',
        lastChangedBy: 'system',
        lastChangedAt: Date.now()
    };

    public isPaused(): boolean {
        return this.tradingControl.mode === 'PAUSED';
    }

    public recordTradingControl(update: TradingControlSnapshot): void {
        this.tradingControl = update;
        // Triggers persistence to disk and database
    }
}
```

**Purpose**: Fast access for enforcement points (AutoTrader, TradeExecutor)

---

## 2. Disk Storage (Local Persistence)

**Location**: `bot_state.json` in the bot service root directory

**File Path**: `c:\Users\sarav\project\bytelyst.ai\trading\bytelyst-trading-bot-service\bot_state.json`

**Structure**:
```json
{
  "health": {
    "tradingControl": {
      "mode": "PAUSED",
      "lastChangedBy": "admin@example.com",
      "lastChangedAt": 1708200000000,
      "reason": "Manual admin pause"
    },
    "tradingLoopHealthy": true,
    "reconciliationLoopHealthy": true,
    // ... other health metrics
  },
  "symbols": { ... },
  "orders": [ ... ],
  "history": [ ... ]
}
```

**How it's saved**:
```typescript
// bytelyst-trading-bot-service/src/services/apiServer.ts

private saveState(): void {
    const snapshot = healthTracker.getSnapshot();
    const stateToSave = {
        health: snapshot,  // Includes tradingControl
        symbols: this.state.symbols,
        orders: this.state.orders,
        history: this.state.history,
        settings: this.state.settings
    };
    
    fs.writeFileSync(
        path.join(process.cwd(), 'bot_state.json'),
        JSON.stringify(stateToSave, null, 2)
    );
}
```

**When it's saved**:
- After every pause/resume action
- Periodically (every 30 seconds)
- On graceful shutdown

**Purpose**: Survive bot restarts, local backup

---

### 3. Database Storage (Cloud Backup)
- **Location**: Supabase `bot_state_snapshots` table
- **Purpose**: Multi-instance recovery, cloud backup, and audit trail
- **Updated**: Throttled writes based on `DB_SNAPSHOT_INTERVAL_MS` (Default: 5 mins)
- `user_id` (UUID, foreign key to users)
- `state` (JSONB) - Contains full bot state including tradingControl
- `created_at` (timestamp)

**How it's saved**:
```typescript
// bytelyst-trading-bot-service/src/services/apiServer.ts

private async persistSnapshotToDb(): Promise<void> {
    if (!config.ENABLE_DB_SNAPSHOTS) return;

    const now = Date.now();
    const elapsed = now - this.lastSnapshotWriteAt;
    if (elapsed < config.DB_SNAPSHOT_INTERVAL_MS) {
        // Throttled...
        return;
    }

    const stateToSave = this.getPersistableState();
    await supabaseService.saveBotStateSnapshot(ownerId, stateToSave);
    this.lastSnapshotWriteAt = Date.now();
}
```

**When it's saved**:
- **Throttled Periodically**: Every `DB_SNAPSHOT_INTERVAL_MS` (Default: 300,000ms / 5 minutes)
- **On Startup/Shutdown**: Forced sync if enabled
- **Note**: The manual "Pause/Resume" action triggers a request to save, but the actual DB write is still subject to the throttle to prevent IOPS spikes.

**Purpose**: 
- Multi-instance recovery
- Cloud backup
- Audit trail

---

## State Recovery on Bot Restart

When the bot starts, it loads the state in this order:

```typescript
// bytelyst-trading-bot-service/src/services/apiServer.ts

public async loadState(): Promise<void> {
    try {
        // 1. Try to load from Supabase (preferred)
        const dbSnapshot = await supabaseService.getLatestBotSnapshot();
        if (dbSnapshot?.snapshot_data?.health?.tradingControl) {
            healthTracker.recordTradingControl(
                dbSnapshot.snapshot_data.health.tradingControl
            );
            logger.info('[LoadState] Restored trading control from Supabase:', 
                dbSnapshot.snapshot_data.health.tradingControl.mode);
            return;
        }
    } catch (err) {
        logger.warn('[LoadState] Failed to load from Supabase, trying disk');
    }

    try {
        // 2. Fallback to disk (bot_state.json)
        const filePath = path.join(process.cwd(), 'bot_state.json');
        if (fs.existsSync(filePath)) {
            const fileData = JSON.parse(fs.readFileSync(filePath, 'utf-8'));
            if (fileData.health?.tradingControl) {
                healthTracker.recordTradingControl(fileData.health.tradingControl);
                logger.info('[LoadState] Restored trading control from disk:', 
                    fileData.health.tradingControl.mode);
                return;
            }
        }
    } catch (err) {
        logger.warn('[LoadState] Failed to load from disk');
    }

    // 3. Default to RUNNING if no state found
    logger.info('[LoadState] No saved state found, defaulting to RUNNING');
    healthTracker.recordTradingControl({
        mode: 'RUNNING',
        lastChangedBy: 'system',
        lastChangedAt: Date.now(),
        reason: 'Bot startup - no previous state'
    });
}
```

---

## Persistence Flow Diagram

```
Admin clicks "Pause" button
    │
    ▼
POST /internal/trading/pause
    │
    ▼
healthTracker.recordTradingControl({ mode: 'PAUSED' })
    │
    ├─────────────────────────────────────────┐
    │                                         │
    ▼                                         ▼
1. In-Memory Update                    2. Trigger Persistence
   (Immediate)                            (Async)
   │                                         │
   ├─ tradingControl.mode = 'PAUSED'        ├─ apiServer.saveState()
   └─ isPaused() returns true               │  └─ Write to bot_state.json
                                             │
                                             └─ apiServer.persistSnapshotToSupabase()
                                                └─ Upsert to bot_snapshots table

Bot Restart
    │
    ▼
apiServer.loadState()
    │
    ├─ Try Supabase (preferred)
    │  └─ SELECT * FROM bot_snapshots ORDER BY updated_at DESC LIMIT 1
    │     └─ Extract health.tradingControl
    │
    ├─ Fallback to Disk
    │  └─ Read bot_state.json
    │     └─ Extract health.tradingControl
    │
    └─ Default to RUNNING
       └─ If no state found
```

---

## 4. Global Configuration (Neural Persistence Settings)

To prevent database overload while maintaining state safety, the bot includes a synchronization throttling mechanism. These settings are managed via the **Admin Tab > Database Synchronization** panel.

### Settings stored in `bot_config` table:

| Key | Default | Description |
|-----|---------|-------------|
| `ENABLE_DB_SNAPSHOTS` | `true` | When `false`, the bot will not write snapshots to Supabase at all. |
| `DB_SNAPSHOT_INTERVAL_MS` | `300000` | Minimum time (in ms) to wait between database writes. |

### Throttling Logic:
1. The bot saves state to local `bot_state.json` **effectively immediately** (debounced 1.5s).
2. The bot checks if `ENABLE_DB_SNAPSHOTS` is true.
3. The bot checks if enough time has passed since `lastSnapshotWriteAt`.
4. Only if both pass is a write sent to Supabase.

---

## Verification

### Check Current State

**Option 1: API Call**
```bash
curl -H "Authorization: Bearer <token>" \
  http://localhost:5000/internal/trading/status

# Response:
{
  "mode": "PAUSED",
  "lastChangedBy": "admin@example.com",
  "lastChangedAt": 1708200000000,
  "reason": "Manual admin pause"
}
```

**Option 2: Check bot_state.json**
```bash
# Windows PowerShell
Get-Content "c:\Users\sarav\project\bytelyst.ai\trading\bytelyst-trading-bot-service\bot_state.json" | ConvertFrom-Json | Select-Object -ExpandProperty health | Select-Object -ExpandProperty tradingControl
```

**Option 3: Check Supabase**
```sql
SELECT snapshot_data->'health'->'tradingControl' 
FROM bot_snapshots 
ORDER BY updated_at DESC 
LIMIT 1;
```

**Option 4: Check Logs**
```bash
# Look for these log entries
[Admin] Trading PAUSED by admin@example.com. Reason: Manual admin pause
[Admin] Trading RESUMED by admin@example.com.
[LoadState] Restored trading control from Supabase: PAUSED
```

---

## Important Notes

### Persistence Guarantees

✅ **Immediate**: In-memory state updated instantly  
✅ **Durable**: Disk and database writes happen within 1 second  
✅ **Recoverable**: State survives bot restarts  
✅ **Auditable**: All changes logged with timestamp and user  

### Failure Scenarios

| Scenario | Behavior |
|----------|----------|
| Disk write fails | State still in memory, Supabase backup available |
| Supabase write fails | State still in memory and on disk |
| Both writes fail | State in memory, will retry on next save cycle |
| Bot crashes | State recovered from Supabase or disk on restart |
| Supabase unavailable on restart | Falls back to disk (bot_state.json) |
| Both unavailable on restart | Defaults to RUNNING mode |

### State Consistency

- **Single Source of Truth**: In-memory state in HealthTracker
- **Persistence is Async**: Writes don't block trading operations
- **Recovery is Synchronous**: State loaded before trading starts
- **No Race Conditions**: All writes go through HealthTracker singleton

---

## File Locations Summary

| Storage | Location | Purpose |
|---------|----------|---------|
| In-Memory | `healthTracker.ts` singleton | Fast runtime access |
| Disk | `bot_state.json` | Local persistence |
| Database | Supabase `bot_snapshots` table | Cloud backup |
| Logs | `combined.log` | Audit trail |

---

## Code References

### Save State
- **File**: `bytelyst-trading-bot-service/src/services/apiServer.ts`
- **Method**: `saveState()` (line ~1700)
- **Method**: `persistSnapshotToSupabase()` (line ~1750)

### Load State
- **File**: `bytelyst-trading-bot-service/src/services/apiServer.ts`
- **Method**: `loadState()` (line ~1800)

### Trading Control State
- **File**: `bytelyst-trading-bot-service/src/services/healthTracker.ts`
- **Property**: `tradingControl: TradingControlSnapshot`
- **Method**: `recordTradingControl()`
- **Method**: `isPaused()`

### Pause/Resume Endpoints
- **File**: `bytelyst-trading-bot-service/src/services/apiServer.ts`
- **Endpoint**: `POST /internal/trading/pause` (line 1054)
- **Endpoint**: `POST /internal/trading/resume` (line 1073)