diff --git a/agent-queue/docs/jobs/phase2-slice3.md b/agent-queue/docs/jobs/phase2-slice3.md new file mode 100644 index 0000000..a5f74e0 --- /dev/null +++ b/agent-queue/docs/jobs/phase2-slice3.md @@ -0,0 +1,156 @@ +--- +engine: devin +cwd: /Users/sd9235/code/mygh/learning_ai_devops_tools +yolo: true +lock: agent-queue +timeout: 4h +--- + +ROLE: Senior bash + distributed-systems engineer. Implement PHASE 2 SLICE 3 — +FACTORY-AGENT INTEGRATION: make the single-host `agent-queue.sh` runner act as a +"factory" that registers / heartbeats / claims / reports against the already-merged +`fleet` coordinator in platform-service, **behind a feature flag**, while keeping +the existing offline git-queue path 100% intact when the flag is off. + +NON-NEGOTIABLE DESIGN RULE (prevents merge churn + regressions): +- Put ALL coordinator-client logic in a NEW separate file `agent-queue/lib/fleet-client.sh` + that `agent-queue.sh` sources. Touch `agent-queue.sh` only at a few well-defined hook + points (claim source, stage-transition reporting, dispatch/help). The offline git-queue + code path MUST be byte-for-byte behaviorally unchanged when `AQ_FLEET` is unset/0. +- Gate every coordinator interaction on `AQ_FLEET=1`. Default (unset) = today's offline + behavior. All 53 existing selftest checks MUST still pass unchanged. + +READ FIRST (verify the real contract — do not guess): +- agent-queue/agent-queue.sh — the runner. Study: the manifest/lifecycle stages + (queued→assigned→building→review→testing→shipped + blocked/failed/dead_letter), + `run_worker`/`cmd_run`/`ship`/`promote`, the Slice-4 `tracker_api` curl wrapper + + `_api_call` + awk JSON helpers (REUSE these patterns — POSIX awk, curl-only, no jq), + and the Slice-4 auto-echo hooks. Mirror that style exactly. +- agent-queue/selftest.sh — how stub-driven HTTP tests work (the tracker stub overrides + the curl wrapper). Build the fleet stub the same way. +- THE COORDINATOR CONTRACT (read-only, in the sibling repo + ../learning_ai_common_plat/services/platform-service/src/modules/fleet/routes.ts): + all routes are registered under the `/api` prefix. Exact endpoints: + POST /api/fleet/factories/heartbeat {factoryId, capabilities[], health, load} + POST /api/fleet/claim {factoryId, capabilities[]} -> job + leaseEpoch + lease expiry (or empty) + GET /api/fleet/jobs/:id + PATCH /api/fleet/jobs/:id fenced stage transition: {stage, checkpoint?, leaseEpoch} + POST /api/fleet/jobs/:id/lease/renew {leaseEpoch} + POST /api/fleet/jobs/:id/lease/release {leaseEpoch} + GET /api/fleet/jobs/:id/runs + GET /api/fleet/jobs/:id/events + Note: there is NO client-side "register factory" or "append event" endpoint — registration + is the heartbeat upsert, and `fleet_events` are written SERVER-SIDE by the coordinator on + each PATCH/claim. The coordinator owns `leaseEpoch` fencing: a PATCH/renew carrying a stale + epoch is rejected (409/conflict). +- ../learning_ai_devops_tools/agent-queue/docs/GIGAFACTORY_ROADMAP.md §7 (claim loop), + §8 (factory/heartbeat/claim/report/drain), §9 (split-brain/offline-degrade), §18 (fencing). + +PREREQUISITE / BRANCHING: +- Branch off CURRENT `main` (Phase 1 complete; foundation + hardening merged). + New branch: feat/gigafactory-p2-slice3. Commit in logical steps. Push + open a PR. + DO NOT merge. + +CONFIG BLOCK (env, in fleet-client.sh; document in README): +- AQ_FLEET (0/1, default 0 — master switch; 0 = pure offline git-queue) +- AQ_FLEET_API (default http://localhost:4003/api) +- AQ_FLEET_TOKEN (bearer; never hardcode) +- AQ_PRODUCT_ID (reuse the Slice-4 var; X-Product-Id header) +- AQ_FACTORY_ID (default: hostname + short rand; stable per process) +- AQ_FLEET_LEASE_RENEW_SEC (default 300), AQ_FLEET_CAPS (auto-detected caps override) + +DELIVERABLES + +1. `agent-queue/lib/fleet-client.sh` (new) — a sourced library, curl-only + POSIX awk + (reuse Slice-4 helpers; do not add deps): + - `fleet_enabled` — returns true iff AQ_FLEET=1 (guard for every other fn). + - `fleet_api METHOD PATH [json]` — curl wrapper adding bearer + X-Product-Id; returns + body; captures HTTP code; non-2xx is logged and surfaced (never crashes the runner). + - `fleet_detect_caps` — reuse the runner's existing capability auto-detection (os, engines, + tools) to build the capabilities array. + - `fleet_heartbeat` — POST factories/heartbeat (registration == first heartbeat); call at + loop start + every AQ_FLEET_LEASE_RENEW_SEC during long runs. + - `fleet_claim` — POST /fleet/claim with caps; parse job id + bodyMd + leaseEpoch + lease + expiry; materialize a transient local job file (reuse the Slice-4 from-tracker + materialization) so the existing runner executes it unchanged. Store leaseEpoch in the + job meta. + - `fleet_report STAGE [checkpoint]` — PATCH /fleet/jobs/:id with {stage, checkpoint?, + leaseEpoch}. **Fencing-aware:** if the coordinator returns conflict/409 (stale epoch), + the worker MUST self-abort the job (stop work, do NOT ship/merge) and log a fenced-abort + event — a reclaimed/zombie worker can never corrupt coordinator state. + - `fleet_lease_renew` / `fleet_lease_release` — fenced; renew on a timer while building; + release on terminal stages. + - `fleet_checkpoint` — capture {wipBranch, wipCommit} and send via fleet_report so a + reclaim can resume (durability, §25). + +2. Wire `agent-queue.sh` at MINIMAL hook points (all guarded by `fleet_enabled`): + - source `lib/fleet-client.sh` near the top. + - claim: when AQ_FLEET=1 and the local inbox is empty, try `fleet_claim` before idling + (coordinator jobs interleave with local `.md` files; local files still work). + - stage transitions (building/review/testing/shipped/failed): call `fleet_report` + + checkpoint — REPLACE the meaning of the Slice-4 direct tracker echo when AQ_FLEET=1 + (the coordinator records `fleet_events`, becoming the audit source of truth → "tracker + echo routed through fleet_events"); keep the direct tracker echo as the offline path. + - heartbeat timer in the run loop; lease renew while a fleet job is building; release on done. + - new subcommands: `aq fleet-status` (heartbeat + show claimable count) and surface + factoryId/leaseEpoch in `status`; add to dispatch + help. + +3. OFFLINE-DEGRADE + SPLIT-BRAIN (§9/§18): if the coordinator is unreachable mid-job, the + runner finishes the in-flight job locally and reconciles on the next reachable call; on + reconnect it presents its leaseEpoch — if the coordinator reports it stale (reclaimed), + the local result is quarantined (marked, NOT auto-shipped) and surfaced for human triage. + +TESTS — extend `agent-queue/selftest.sh` (stub the fleet API exactly like the tracker stub; +tests are sacred, all 53 prior checks stay green): +- flag off (default): AQ_FLEET unset → ZERO fleet API calls; existing offline flow identical + (re-assert a couple of the offline cases under flag-off). +- heartbeat/register: AQ_FLEET=1 loop start → stub receives POST factories/heartbeat with caps. +- claim: stub returns a job → runner materializes a local job (bodyMd + leaseEpoch in meta) + and executes it to review/. +- report + checkpoint: building/review/testing → stub receives PATCH /fleet/jobs/:id with the + correct stage + leaseEpoch (+ checkpoint on building). +- FENCING: stub returns conflict on PATCH (stale epoch) → worker self-aborts, job NOT shipped, + a fenced-abort is logged/surfaced. +- lease renew: long-running stub → at least one renew call with current leaseEpoch. +- offline-degrade: stub returns connection error mid-job → job still completes locally; on + next call presenting a now-stale epoch → result quarantined (not auto-shipped). +- no-leak: assert the prompt/bodyMd + token are never sent in a report/comment payload they + shouldn't be (reuse the Slice-4 sentinel check). + +VERIFY GATE (must all pass): +- bash agent-queue/selftest.sh (all prior 53 + new fleet cases green; none weakened) +- bash -n agent-queue/agent-queue.sh && bash -n agent-queue/lib/fleet-client.sh +- node --check agent-queue/dashboard.mjs (if present/unchanged) +- shellcheck --severity=error agent-queue/agent-queue.sh agent-queue/lib/fleet-client.sh + +DOCS: +- README: a "Fleet integration (Phase 2)" section — the AQ_FLEET flag, env table, the + claim/heartbeat/report/fence/renew protocol, offline-degrade + quarantine behavior, and a + one-paragraph "offline vs fleet mode" explainer. +- Tick the relevant §8/§9/§14 Phase-2 boxes in GIGAFACTORY_ROADMAP.md with a P2-S3 slice note. + +CONSTRAINTS: bash + curl + POSIX awk only (no jq, no new deps); reuse Slice-4 helpers; never +hardcode tokens/secrets; offline path unchanged when AQ_FLEET unset; conventional commits +(feat(agent-queue): ...); never weaken a test; do not edit the sibling common-plat repo. + +FINAL OUTPUT — print the report in EXACTLY this format: + +## Implementation Report — Phase 2 Slice 3 (factory-agent integration) +### Branch & commits / PR +### Files changed +- : +### What was implemented +- fleet-client.sh: +- agent-queue.sh hook points: +- fencing + offline-degrade + quarantine: +- tracker echo via fleet_events: +### Tests added +- : (esp. flag-off no-op, claim, fenced self-abort, offline quarantine) +- selftest summary: +### Verify gate results +- selftest / bash -n / node --check / shellcheck: +### Deviations / assumptions +- +### Suggested next slice +- Phase 2 remaining: scheduler/router wiring, factory enrollment + scoped tokens, feature-flag + shadow/dual-run, and the two-factory parallel demo (Phase 2 exit criteria).