# Fleet module — agent gigafactory coordinator (Phase 2 foundation)

Product-agnostic, cloud-agnostic coordinator for distributed agent jobs. This is
the durable backend that supersedes the single-host stand-ins built in the
`agent-queue` (devops-tools) repo. Everything runs on the `@bytelyst/datastore`
abstraction, so all tests execute on `DB_PROVIDER=memory` (no Cosmos/network).

Spec: `../learning_ai_devops_tools/agent-queue/docs/GIGAFACTORY/GIGAFACTORY_ROADMAP.md`
(§4 core contract, §7 scheduler/claim, §8 factory/lease/heartbeat, §13 containers,
§18 failure model, §25 durability/recovery, §26 insights).

## Containers (partition keys)

| Container           | PK           | Purpose                                                                                                                                 |
| ------------------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------- |
| `fleet_jobs`        | `/productId` | durable job: `manifestSnapshot`, verbatim `bodyMd`, `stage`, `idempotencyKey`, `deps`, `checkpoint`, `priority`, `rev`, `leaseEpoch`, … |
| `fleet_runs`        | `/jobId`     | one execution attempt: engine, timings, `result`, `insights` (tokens/cost/diff)                                                         |
| `fleet_leases`      | `/jobId`     | single-holder lease: `holderFactoryId`, `expiresAt`, `leaseEpoch`, `status`                                                             |
| `fleet_factories`   | `/productId` | registered worker host: `capabilities`, `health`, `load`, `seatLimit`, `lastHeartbeatAt`                                                |
| `fleet_profiles`    | `/productId` | immutable, versioned profile snapshot                                                                                                   |
| `fleet_events`      | `/jobId`     | append-only audit/event stream (monotonic `seq`)                                                                                        |
| `fleet_artifacts`   | `/jobId`     | pointers to blob-stored artifacts (no inline logs)                                                                                      |
| `fleet_queue_state` | `/productId` | Phase-4 M0 RU gate: monotonic `version` bumped on job create + every stage change (cheap "work changed" signal)                         |

Every document carries `productId`. Containers are registered in
`lib/cosmos-init.ts`.

## Concurrency protocol

**Optimistic concurrency (`rev`).** Jobs and leases carry a monotonic `rev` token.
`repository.revUpdate*` is a compare-and-swap: it writes only if the stored `rev`
still equals the caller's expected `rev`, else it reports `conflict` and writes
nothing. In production (Cosmos) this maps to `_etag` / `If-Match`; on the memory
provider it is enforced by re-reading `rev` immediately before the write, which is
exact for the sequential calls the coordinator and tests make.

**Atomic claim (`claimNextJob`).** Select the highest-priority, oldest job that is
`queued` (or `blocked` with now-satisfied deps) and whose `capabilities` are a
subset of the factory's, then `tryClaimJob` does a `rev` CAS to flip it to
`assigned` and acquire/create its lease. Under contention exactly one factory wins
the CAS; losers get `conflict` and re-select. No double-assignment, ever.

**Leases + fencing.** Acquiring/reclaiming a lease increments `leaseEpoch`. Every
worker mutation (`patchJobFenced`, `renewLease`, `releaseLease`) carries its
`leaseEpoch`; a call whose epoch is `< job.leaseEpoch` is rejected (`fenced`) — a
stale/zombie worker can never overwrite a reassigned job.

**Heartbeat.** `heartbeat(factory)` upserts `lastHeartbeatAt` + health/load;
`isFactoryStale` detects a missed-heartbeat factory (stale after
`DEFAULT_STALE_FACTORY_MS = 90_000`). A factory's heartbeat cadence
(`AQ_FLEET_LEASE_RENEW_SEC` on the runner) MUST be well under 90s or a healthy
factory flaps to "stale" between beats — the fleet launcher uses 30s.

**Reaper.** `reapExpiredLeases(now)` scans `held` leases with `expiresAt < now`,
bumps `leaseEpoch` (fencing the dead holder), returns the job to `queued` (or
`blocked` if deps are now unmet) **preserving the `checkpoint` pointer** (resume
from WIP), and marks the lease `expired`. Idempotent — a reaped lease is no longer
`held`, so a second pass reaps nothing. Cosmos TTL cannot do this (it only deletes
the lease doc; it cannot requeue the job, bump the epoch, or keep the checkpoint),
so the reaper — not TTL — owns recovery.

## Cost/quality-aware routing (§2)

Two **additive, flag-gated** routing refinements layer on top of the §7 scoring
core. Both default OFF, so the deterministic claim path is byte-for-byte unchanged
unless explicitly enabled.

- **Soft engine preference (`FLEET_COST_ROUTING`).** `engine-stats.ts`
  (`computeEngineQuality`) aggregates each run's `insights.costUsd` + duration
  (`endedAt − startedAt`) into a per-engine `[0,1]` quality, anchored so the
  cheapest/fastest engine scores 1 and pricier/slower engines decay proportionally.
  The scheduler's `engineQuality` term (weight `0.4`, below affinity) gently biases
  routing toward the cheaper/faster engine among otherwise-comparable candidates.
  Engines with no history are **neutral (1)** — history can only demote a
  demonstrably costly engine, never penalise a new one. The term contributes `0`
  unless the lookup is injected, so default scoring is untouched.
- **Per-engine circuit breaker (`FLEET_ENGINE_BREAKER`).** `engine-breaker.ts`
  tracks failures per `(factoryId, engine)` (CLOSED → OPEN after N consecutive
  failures → HALF*OPEN probe after a reset window). `releaseLease` always **records**
  the outcome (so breaker state is observable in `/fleet/metrics → engineBreakers`
  even before enforcement); when the flag is on, an OPEN pair is removed from that
  factory's candidate set so a repeatedly-failing engine (e.g. `codex` erroring on a
  box) is routed \_around*. The breaker only ever **restricts** — it never forces a
  route. Jobs without a concrete `engine` resolve on the runner and are never gated.

## Budgets (§3)

Per-product cost ceilings (`FleetBudgetDoc`, pk `/productId`), enforced at claim
time when `FLEET_BUDGETS` is on:

- **Product ceiling (hard gate).** A claim is blocked when the budget is `paused`
  or `spentUsd >= ceilingUsd`. `accrueSpend` adds each shipped run's actual
  `insights.costUsd` (idempotent per `<jobId>:<leaseEpoch>`) and auto-pauses the
  product at the ceiling.
- **Per-engine sub-ceilings.** `engineCeilingsUsd` (e.g. `{ "codex": 50 }`) cap
  spend for a single engine; `spentByEngineUsd` tracks per-engine accrual. An engine
  at its sub-ceiling is **routed around** at claim time (composed into the same
  per-engine availability gate as the circuit breaker) — it never pauses the whole
  product, so other engines keep flowing.
- **Projected-overspend alerts.** `/fleet/metrics` surfaces a `budget` summary
  (ceiling, spend, status, per-engine breakdown, and the projected end-of-window
  spend) and derives alerts: `budget_overspend_projected` (warning, when the burn
  rate extrapolates past the ceiling — guarded so it won't fire in the first 10% of
  a window), `budget_exhausted` (critical), and `engine_budget_exhausted` (warning).
  These surface whenever a budget is configured, independent of the enforcement flag.

## Submit semantics (idempotency + deps)

- same `idempotencyKey` + identical `bodyMd` → returns the existing job (dedup).
- same key + different content while still `queued`/`blocked` → supersede in place.
- same key + different content once past `queued` → `409 Conflict`.
- a job with unmet `deps` is `blocked` (a dep is met at `shipped`, or `testing`
  when `depsMode: soft`); submit-time cycle detection rejects cyclic graphs.

## REST (under `/api`, auth + productId)

`POST /fleet/jobs` · `GET /fleet/jobs` · `GET /fleet/jobs/:id` ·
`PATCH /fleet/jobs/:id` (fenced) · `POST /fleet/claim` ·
`POST /fleet/jobs/:id/lease/renew` · `POST /fleet/jobs/:id/lease/release` ·
`POST /fleet/factories/heartbeat` · `GET /fleet/jobs/:id/runs` ·
`GET /fleet/jobs/:id/events` · `GET /fleet/metrics` ·
`GET /fleet/queue-state` (Phase-4 M0 RU gate).

## Files

`types.ts` (Zod schemas → inferred types) · `repository.ts` (per-container repos +
`revUpdate` CAS) · `coordinator.ts` (claim/lease/fence/heartbeat/reaper + submit) ·
`routes.ts` (REST) · `*.test.ts` (schema, repo, coordinator incl. the atomic-claim
race / fencing / reaper, and route inject tests).
