diff --git a/agent-queue/docs/jobs/phase2-tracker-wiring.md b/agent-queue/docs/jobs/phase2-tracker-wiring.md new file mode 100644 index 0000000..2c71924 --- /dev/null +++ b/agent-queue/docs/jobs/phase2-tracker-wiring.md @@ -0,0 +1,120 @@ +--- +engine: devin +cwd: /Users/sd9235/code/mygh/learning_ai_common_plat +yolo: true +lock: common-plat-tracker +timeout: 4h +--- + +ROLE: Senior backend engineer. Implement the PHASE 2 DIRECT TRACKER -> MODULE WIRING +(§10) for the fleet coordinator: a service-side bridge that turns a tracker Item into +a fleet job (submitted through the coordinator, so it is routed by the §7 scheduler), +and echoes the job's lifecycle back onto the Item — the full task<->job ROUND-TRIP, +in-process, with no shell hop. This closes the §10 "direct tracker->module calls" box. + +PARALLEL-SAFETY: One other Devin is running in a DIFFERENT repo (learning_ai_devops_tools, +the two-factory demo). There is NO other Devin in this repo, so you may edit any fleet +file you need. Do NOT edit the agent-queue repo. + +READ FIRST (understand the contracts before writing): +- services/platform-service/src/modules/fleet/coordinator.ts + - submitJob(productId, SubmitJobInput) -> { job, outcome } : idempotent submit; the + job already has a `trackerItemId` field (types.ts) — reuse it, do NOT add a new one. + - claimNextJob(ctx) already routes candidates through the §7 scheduler (selectJob). + You do NOT change claim/scheduler — tracker jobs flow through the SAME path. + - patchJobFenced / stage transitions — the lifecycle you will mirror to the tracker. +- services/platform-service/src/modules/fleet/types.ts — FleetJobDoc.stage values, + SubmitJobSchema (trackerItemId, idempotencyKey, priority, capabilities, budget, kind). +- services/platform-service/src/modules/fleet/routes.ts — existing fleet route patterns + (auth, getRequestProductId(req), Zod parse, productId enforcement). Add new routes here + in the SAME style. +- services/platform-service/src/modules/items/{types,routes,repository}.ts — the Item + API contract you mirror to: Item fields (id, productId, title/description, status, + labels[]), the status vocabulary, and the comment/note mechanism. Call the items + repository DIRECTLY in-process (no HTTP/curl) — this is the whole point of "direct wiring". +- ../learning_ai_devops_tools/agent-queue/docs/GIGAFACTORY_ROADMAP.md §10 (tracker + integration), §24.5 (echo rule), §14 Phase-2 checklist (the §10 box you will tick). + +PREREQUISITE / BRANCHING: branch off CURRENT main -> feat/gigafactory-p2-tracker-wiring. +Push + open a PR. DO NOT merge. + +DELIVERABLES + +1. tracker-bridge.ts (NEW) — pure-ish service module (it may call the items + fleet + repositories, but no HTTP, no Fastify types inside it): + - `ingestItemAsJob(productId, itemId, opts?) -> { job, outcome }`: + * read the Item via the items repository (404 -> NotFoundError). + * map Item -> SubmitJobInput: title/description -> bodyMd (verbatim instruction); + labels carry manifest hints where present (engine-class:*, profile:*, + priority:*, cap:* -> capabilities[]); otherwise sane defaults. + * set trackerItemId = itemId and a STABLE idempotency-key (e.g. `tracker-`), + then call coordinator.submitJob — so re-ingest of the same Item dedupes (no + duplicate job) and the job is scheduled by the §7 router like any other. + - `echoJobToItem(productId, jobId) -> { echoed: status | null }`: + * load the job; if it has no trackerItemId -> no-op (return null). + * map stage -> Item status (FULL round-trip, both directions of the lifecycle): + queued/assigned/building/review/testing -> in_progress + shipped -> done + failed -> blocked (+ note) + * append a comment/note with metrics ONLY (attempts, duration, cost/tokens if + present) — NEVER the prompt body / secrets. + * IDEMPOTENT: persist the last-echoed status (on the job doc or a small bridge + record) and make a re-echo of an unchanged outcome a no-op. + - Echo is BEST-EFFORT and downstream: an items-write failure NEVER fails the job — + surface it as a logged error / a `{ echoed: null, error }` shape, never throw into + the job lifecycle. + +2. Wire echo into stage transitions (server-side, opt-in, additive): + - When the coordinator/route performs a stage transition for a job that has a + trackerItemId, call echoJobToItem (guarded by a config flag, default OFF, e.g. + FLEET_TRACKER_ECHO; OFF => behavior byte-for-byte unchanged). Do not block or fail + the transition on echo error. + +3. Routes (routes.ts, additive — match existing auth/productId style): + - POST /fleet/tracker/ingest { itemId } -> ingestItemAsJob + - POST /fleet/tracker/echo { jobId } -> echoJobToItem (manual echo) + - All productId-scoped via getRequestProductId(req); a foreign productId cannot ingest + or echo another product's Item/job. + +TESTS (tracker-bridge.test.ts + route additions — tests are sacred; use @bytelyst/testing ++ the in-memory providers; NO live HTTP): +- ingest creates exactly one job: Item -> job with trackerItemId set, bodyMd = description, + idempotency-key = tracker-; the job is claimable via the normal claimNextJob path. +- ingest label mapping: labels [engine-class:agentic-coder, priority:high, cap:os:mac] + -> job priority/capabilities reflect them. +- ingest idempotent: ingesting the same Item twice -> one job (dedupe), outcome reflects it. +- echo round-trip: a job advancing queued->building->shipped drives the Item + in_progress -> done, and a metrics-only comment is written (assert NO bodyMd/secret leaks). +- echo failed -> Item blocked (+ note). +- echo idempotent: re-echo of an unchanged stage -> no duplicate Item write. +- echo non-fatal: items-write throws -> echoJobToItem returns { echoed:null,error }, the + job state is untouched, the transition still succeeds. +- echo OFF (default flag): a stage transition performs ZERO items writes. +- productId isolation: ingest/echo for a foreign productId -> not found / rejected. +- REGRESSION: every existing fleet + items test stays green. + +VERIFY GATE: +- pnpm --filter @lysnrai/platform-service exec vitest run src/modules/fleet src/modules/items +- pnpm --filter @lysnrai/platform-service build +- pnpm build && pnpm test (no consumer regression) + +CONSTRAINTS: ESM `.js` import specifiers; no `any` (Zod inference / explicit types); no +console.log (use app.log / req.log); every Cosmos doc keeps `productId`; reuse the +existing `trackerItemId` field and items contract — do NOT fork a parallel schema; +do NOT change claimNextJob or the scheduler; conventional commits +(feat(platform-service): ...); do not edit the agent-queue repo. + +DOCS: tick §10 "direct tracker->module calls" in +../learning_ai_devops_tools/agent-queue/docs/GIGAFACTORY_ROADMAP.md §14 Phase-2 (note the +flag name + that it is the in-process round-trip; the agent-queue shell adapter remains the +single-host path). + +FINAL OUTPUT — report in EXACTLY this format: +## Implementation Report — Phase 2 Direct Tracker -> Module Wiring (§10) +### Branch & commits / PR +### Files changed +### What was implemented (ingest mapping, round-trip status map, echo idempotency + flag) +### Tests added (+ pnpm test summary) +### Verify gate results +### Deviations / assumptions (Item status vocabulary matched, flag name, where last-echoed is stored) +### Suggested next slice (Phase 3 tracker-web fleet control plane) diff --git a/agent-queue/docs/jobs/phase2-two-factory-demo.md b/agent-queue/docs/jobs/phase2-two-factory-demo.md new file mode 100644 index 0000000..3b7ca8f --- /dev/null +++ b/agent-queue/docs/jobs/phase2-two-factory-demo.md @@ -0,0 +1,91 @@ +--- +engine: devin +cwd: /Users/sd9235/code/mygh/learning_ai_devops_tools +yolo: true +lock: devops-tools-demo +timeout: 4h +--- + +ROLE: Senior engineer. Build the PHASE 2 TWO-FACTORY PARALLEL DEMO — the final +Phase-2 EXIT-CRITERIA box (§14): >=2 factories executing jobs in parallel via the +coordinator, proving conflict-free atomic claims, lease fencing, and reaper-reclaim +end-to-end. This is a DEMO HARNESS + DOCS, not new runtime behavior — agent-queue.sh +and lib/fleet-client.sh already implement everything; you orchestrate + observe them. + +PARALLEL-SAFETY: One other Devin is running in a DIFFERENT repo (learning_ai_common_plat, +the tracker-wiring slice) — no overlap. In THIS repo you OWN a NEW demo directory and the +additive selftest/docs only: +- You OWN (create/edit): agent-queue/demo/two-factory-demo.sh (NEW), + agent-queue/demo/README.md (NEW), additive checks in agent-queue/selftest.sh, + and the §14 Phase-2 demo/exit-criteria ticks in agent-queue/docs/GIGAFACTORY_ROADMAP.md. +- You MUST NOT change the behavior of agent-queue.sh or lib/fleet-client.sh. You may READ + them and CALL them; if a tiny additive hook is unavoidable, keep it flag-gated and prove + all 68 existing selftest checks still pass byte-for-byte. +- Leave the runtime agent-queue/queue/* working-tree artifacts ALONE (live-daemon state, + not yours) — never stage or commit them. + +READ FIRST: +- agent-queue/agent-queue.sh — the run loop, AQ_FLEET / AQ_FLEET_ROUTE flags, claim path, + fencing/quarantine, offline-degrade. +- agent-queue/lib/fleet-client.sh — fleet_register/heartbeat, claim, lease renew, fenced + PATCH, the coordinator HTTP wrappers and their env (AQ_FLEET_API, AQ_FLEET_TOKEN, factory id). +- agent-queue/selftest.sh — how the EXISTING fleet tests STUB the coordinator (the canned + responder pattern). Reuse that exact stub style so the demo's selftest needs NO live service. +- ../learning_ai_common_plat/services/platform-service/src/modules/fleet/coordinator.ts — + the claim/lease/fence/reaper contract you are demonstrating (read-only; do not edit). +- agent-queue/docs/GIGAFACTORY_ROADMAP.md §14 Phase-2 "Two-factory demo" + "Exit criteria". + +DELIVERABLES + +1. agent-queue/demo/two-factory-demo.sh — an orchestration script that: + - Starts >=2 factories (distinct factoryIds, e.g. mac-1 + ubuntu-1) against ONE + coordinator with AQ_FLEET=1 AQ_FLEET_ROUTE=1, each in its own working dir/queue so + they do not share local inbox state — they compete ONLY through the coordinator. + - Submits 3 jobs and lets the two factories drain them in parallel. + - DEMONSTRATES + ASSERTS the Phase-2 exit guarantees: + (a) no double-assign: each job is claimed/executed by exactly ONE factory. + (b) fencing: kill a factory MID-JOB -> the reaper returns the job -> the OTHER + factory reclaims and completes it AND the dead worker's late/zombie report is + FENCED (rejected, never shipped). + (c) parallelism: both factories make progress concurrently (not serialized). + - Prints a clear PASS/FAIL summary (per-job winner, reclaim event, fence event). + - DUAL MODE: works against a real coordinator when AQ_FLEET_API/AQ_FLEET_TOKEN are set; + otherwise drives the SAME selftest coordinator STUB so the demo is runnable + CI-safe + with zero external deps. Document both invocations. + - bash, mac+linux safe, curl-only, no new runtime deps; style consistent with the repo. + +2. agent-queue/demo/README.md — how to run the demo (stub mode + real-coordinator mode), + the env vars, what each asserted guarantee proves, and a short "what to watch" guide + (the kanban/log lines that show the reclaim + fence). + +3. selftest.sh — ADD a small number of checks (do NOT modify the existing 68) that run the + demo in STUB mode headlessly and assert: 3 jobs all reach a terminal state across the 2 + factories with no double-assignment; the kill -> reclaim -> fenced-zombie path fires; + exit 0. Keep them fast + deterministic (seeded, no real sleeps where avoidable). + +TESTS / VERIFY GATE: +- bash agent-queue/selftest.sh -> all prior 68 + the new demo checks green, exit 0. +- bash -n agent-queue/demo/two-factory-demo.sh && bash -n agent-queue/agent-queue.sh + && bash -n agent-queue/lib/fleet-client.sh -> OK. +- shellcheck --severity=error on the new script + the two core scripts -> clean. +- node --check agent-queue/dashboard.mjs -> OK (must remain unchanged). + +CONSTRAINTS: do NOT alter agent-queue.sh / fleet-client.sh runtime behavior; reuse the +existing coordinator stub pattern; never commit queue/* runtime artifacts; mac+linux safe; +no emojis; conventional commits (feat(agent-queue): ...); tests sacred (the 68 stay green). + +DOCS: tick the §14 Phase-2 "Two-factory demo" box and, once the demo asserts all three +guarantees, the Phase-2 "Exit criteria" line in GIGAFACTORY_ROADMAP.md — set §0 Phase 2 -> +complete (or note the exact remaining %). This is the box that closes Phase 2. + +FINAL OUTPUT — report in EXACTLY this format: +## Implementation Report — Phase 2 Two-Factory Parallel Demo (Exit Criteria) +### Branch & commits / PR +- branch / based-on: feat/gigafactory-p2-two-factory-demo off current main +### Files changed +### What was implemented (orchestration, the 3 asserted guarantees, stub vs real mode) +### Tests added (+ selftest PASS/FAIL summary: prior 68 + new) +### Verify gate results (selftest / bash -n / shellcheck / node --check) +### Deviations / assumptions (how factories are isolated, how kill/reclaim is simulated in stub) +### Phase 2 status (which §14 boxes now complete; exit criteria met Y/N; what (if anything) remains) +### Suggested next slice (Phase 3 — tracker-web fleet control plane + DAG + budgets)