docs(agent-queue): draft P2-S3 factory-agent integration prompt (claim/heartbeat/report/fence behind AQ_FLEET)

2026-05-29 22:03:12 -07:00 · 2026-05-29 22:03:12 -07:00 · 9a073ef225
commit 9a073ef225
parent 7d6275f935
1 changed files with 156 additions and 0 deletions
--- a/agent-queue/docs/jobs/phase2-slice3.md
+++ b/agent-queue/docs/jobs/phase2-slice3.md
@ -0,0 +1,156 @@
+---
+engine: devin
+cwd: /Users/sd9235/code/mygh/learning_ai_devops_tools
+yolo: true
+lock: agent-queue
+timeout: 4h
+---
+
+ROLE: Senior bash + distributed-systems engineer. Implement PHASE 2 SLICE 3 —
+FACTORY-AGENT INTEGRATION: make the single-host `agent-queue.sh` runner act as a
+"factory" that registers / heartbeats / claims / reports against the already-merged
+`fleet` coordinator in platform-service, **behind a feature flag**, while keeping
+the existing offline git-queue path 100% intact when the flag is off.
+
+NON-NEGOTIABLE DESIGN RULE (prevents merge churn + regressions):
+- Put ALL coordinator-client logic in a NEW separate file `agent-queue/lib/fleet-client.sh`
+  that `agent-queue.sh` sources. Touch `agent-queue.sh` only at a few well-defined hook
+  points (claim source, stage-transition reporting, dispatch/help). The offline git-queue
+  code path MUST be byte-for-byte behaviorally unchanged when `AQ_FLEET` is unset/0.
+- Gate every coordinator interaction on `AQ_FLEET=1`. Default (unset) = today's offline
+  behavior. All 53 existing selftest checks MUST still pass unchanged.
+
+READ FIRST (verify the real contract — do not guess):
+- agent-queue/agent-queue.sh — the runner. Study: the manifest/lifecycle stages
+  (queued→assigned→building→review→testing→shipped + blocked/failed/dead_letter),
+  `run_worker`/`cmd_run`/`ship`/`promote`, the Slice-4 `tracker_api` curl wrapper +
+  `_api_call` + awk JSON helpers (REUSE these patterns — POSIX awk, curl-only, no jq),
+  and the Slice-4 auto-echo hooks. Mirror that style exactly.
+- agent-queue/selftest.sh — how stub-driven HTTP tests work (the tracker stub overrides
+  the curl wrapper). Build the fleet stub the same way.
+- THE COORDINATOR CONTRACT (read-only, in the sibling repo
+  ../learning_ai_common_plat/services/platform-service/src/modules/fleet/routes.ts):
+  all routes are registered under the `/api` prefix. Exact endpoints:
+    POST   /api/fleet/factories/heartbeat   {factoryId, capabilities[], health, load}
+    POST   /api/fleet/claim                 {factoryId, capabilities[]} -> job + leaseEpoch + lease expiry (or empty)
+    GET    /api/fleet/jobs/:id
+    PATCH  /api/fleet/jobs/:id               fenced stage transition: {stage, checkpoint?, leaseEpoch}
+    POST   /api/fleet/jobs/:id/lease/renew   {leaseEpoch}
+    POST   /api/fleet/jobs/:id/lease/release {leaseEpoch}
+    GET    /api/fleet/jobs/:id/runs
+    GET    /api/fleet/jobs/:id/events
+  Note: there is NO client-side "register factory" or "append event" endpoint — registration
+  is the heartbeat upsert, and `fleet_events` are written SERVER-SIDE by the coordinator on
+  each PATCH/claim. The coordinator owns `leaseEpoch` fencing: a PATCH/renew carrying a stale
+  epoch is rejected (409/conflict).
+- ../learning_ai_devops_tools/agent-queue/docs/GIGAFACTORY_ROADMAP.md §7 (claim loop),
+  §8 (factory/heartbeat/claim/report/drain), §9 (split-brain/offline-degrade), §18 (fencing).
+
+PREREQUISITE / BRANCHING:
+- Branch off CURRENT `main` (Phase 1 complete; foundation + hardening merged).
+  New branch: feat/gigafactory-p2-slice3. Commit in logical steps. Push + open a PR.
+  DO NOT merge.
+
+CONFIG BLOCK (env, in fleet-client.sh; document in README):
+- AQ_FLEET            (0/1, default 0 — master switch; 0 = pure offline git-queue)
+- AQ_FLEET_API        (default http://localhost:4003/api)
+- AQ_FLEET_TOKEN      (bearer; never hardcode)
+- AQ_PRODUCT_ID       (reuse the Slice-4 var; X-Product-Id header)
+- AQ_FACTORY_ID       (default: hostname + short rand; stable per process)
+- AQ_FLEET_LEASE_RENEW_SEC (default 300), AQ_FLEET_CAPS (auto-detected caps override)
+
+DELIVERABLES
+
+1. `agent-queue/lib/fleet-client.sh` (new) — a sourced library, curl-only + POSIX awk
+   (reuse Slice-4 helpers; do not add deps):
+   - `fleet_enabled` — returns true iff AQ_FLEET=1 (guard for every other fn).
+   - `fleet_api METHOD PATH [json]` — curl wrapper adding bearer + X-Product-Id; returns
+     body; captures HTTP code; non-2xx is logged and surfaced (never crashes the runner).
+   - `fleet_detect_caps` — reuse the runner's existing capability auto-detection (os, engines,
+     tools) to build the capabilities array.
+   - `fleet_heartbeat` — POST factories/heartbeat (registration == first heartbeat); call at
+     loop start + every AQ_FLEET_LEASE_RENEW_SEC during long runs.
+   - `fleet_claim` — POST /fleet/claim with caps; parse job id + bodyMd + leaseEpoch + lease
+     expiry; materialize a transient local job file (reuse the Slice-4 from-tracker
+     materialization) so the existing runner executes it unchanged. Store leaseEpoch in the
+     job meta.
+   - `fleet_report STAGE [checkpoint]` — PATCH /fleet/jobs/:id with {stage, checkpoint?,
+     leaseEpoch}. **Fencing-aware:** if the coordinator returns conflict/409 (stale epoch),
+     the worker MUST self-abort the job (stop work, do NOT ship/merge) and log a fenced-abort
+     event — a reclaimed/zombie worker can never corrupt coordinator state.
+   - `fleet_lease_renew` / `fleet_lease_release` — fenced; renew on a timer while building;
+     release on terminal stages.
+   - `fleet_checkpoint` — capture {wipBranch, wipCommit} and send via fleet_report so a
+     reclaim can resume (durability, §25).
+
+2. Wire `agent-queue.sh` at MINIMAL hook points (all guarded by `fleet_enabled`):
+   - source `lib/fleet-client.sh` near the top.
+   - claim: when AQ_FLEET=1 and the local inbox is empty, try `fleet_claim` before idling
+     (coordinator jobs interleave with local `.md` files; local files still work).
+   - stage transitions (building/review/testing/shipped/failed): call `fleet_report` +
+     checkpoint — REPLACE the meaning of the Slice-4 direct tracker echo when AQ_FLEET=1
+     (the coordinator records `fleet_events`, becoming the audit source of truth → "tracker
+     echo routed through fleet_events"); keep the direct tracker echo as the offline path.
+   - heartbeat timer in the run loop; lease renew while a fleet job is building; release on done.
+   - new subcommands: `aq fleet-status` (heartbeat + show claimable count) and surface
+     factoryId/leaseEpoch in `status`; add to dispatch + help.
+
+3. OFFLINE-DEGRADE + SPLIT-BRAIN (§9/§18): if the coordinator is unreachable mid-job, the
+   runner finishes the in-flight job locally and reconciles on the next reachable call; on
+   reconnect it presents its leaseEpoch — if the coordinator reports it stale (reclaimed),
+   the local result is quarantined (marked, NOT auto-shipped) and surfaced for human triage.
+
+TESTS — extend `agent-queue/selftest.sh` (stub the fleet API exactly like the tracker stub;
+tests are sacred, all 53 prior checks stay green):
+- flag off (default): AQ_FLEET unset → ZERO fleet API calls; existing offline flow identical
+  (re-assert a couple of the offline cases under flag-off).
+- heartbeat/register: AQ_FLEET=1 loop start → stub receives POST factories/heartbeat with caps.
+- claim: stub returns a job → runner materializes a local job (bodyMd + leaseEpoch in meta)
+  and executes it to review/.
+- report + checkpoint: building/review/testing → stub receives PATCH /fleet/jobs/:id with the
+  correct stage + leaseEpoch (+ checkpoint on building).
+- FENCING: stub returns conflict on PATCH (stale epoch) → worker self-aborts, job NOT shipped,
+  a fenced-abort is logged/surfaced.
+- lease renew: long-running stub → at least one renew call with current leaseEpoch.
+- offline-degrade: stub returns connection error mid-job → job still completes locally; on
+  next call presenting a now-stale epoch → result quarantined (not auto-shipped).
+- no-leak: assert the prompt/bodyMd + token are never sent in a report/comment payload they
+  shouldn't be (reuse the Slice-4 sentinel check).
+
+VERIFY GATE (must all pass):
+- bash agent-queue/selftest.sh   (all prior 53 + new fleet cases green; none weakened)
+- bash -n agent-queue/agent-queue.sh && bash -n agent-queue/lib/fleet-client.sh
+- node --check agent-queue/dashboard.mjs   (if present/unchanged)
+- shellcheck --severity=error agent-queue/agent-queue.sh agent-queue/lib/fleet-client.sh
+
+DOCS:
+- README: a "Fleet integration (Phase 2)" section — the AQ_FLEET flag, env table, the
+  claim/heartbeat/report/fence/renew protocol, offline-degrade + quarantine behavior, and a
+  one-paragraph "offline vs fleet mode" explainer.
+- Tick the relevant §8/§9/§14 Phase-2 boxes in GIGAFACTORY_ROADMAP.md with a P2-S3 slice note.
+
+CONSTRAINTS: bash + curl + POSIX awk only (no jq, no new deps); reuse Slice-4 helpers; never
+hardcode tokens/secrets; offline path unchanged when AQ_FLEET unset; conventional commits
+(feat(agent-queue): ...); never weaken a test; do not edit the sibling common-plat repo.
+
+FINAL OUTPUT — print the report in EXACTLY this format:
+
+## Implementation Report — Phase 2 Slice 3 (factory-agent integration)
+### Branch & commits / PR
+### Files changed
+- <path>: <summary>
+### What was implemented
+- fleet-client.sh: <functions + flag gating>
+- agent-queue.sh hook points: <the few places touched + why minimal>
+- fencing + offline-degrade + quarantine: <how>
+- tracker echo via fleet_events: <how>
+### Tests added
+- <name>: <assertion>   (esp. flag-off no-op, claim, fenced self-abort, offline quarantine)
+- selftest summary: <N checks = 53 prior + M new>
+### Verify gate results
+- selftest / bash -n / node --check / shellcheck: <results>
+### Deviations / assumptions
+- <claim/lease contract details, anything stubbed, how registration maps to heartbeat>
+### Suggested next slice
+- Phase 2 remaining: scheduler/router wiring, factory enrollment + scoped tokens, feature-flag
+  shadow/dual-run, and the two-factory parallel demo (Phase 2 exit criteria).