From 7c4f5bc9b0dd95a664ccb41b89f70835151ae6d4 Mon Sep 17 00:00:00 2001 From: saravanakumardb1 Date: Fri, 29 May 2026 19:11:09 -0700 Subject: [PATCH] docs(agent-queue): draft Slice 4 (tracker adapter) + Phase 2 Slice 1 (fleet data model) --- agent-queue/docs/jobs/phase1-slice4.md | 125 +++++++++++++++++++++++++ agent-queue/docs/jobs/phase2-slice1.md | 125 +++++++++++++++++++++++++ 2 files changed, 250 insertions(+) create mode 100644 agent-queue/docs/jobs/phase1-slice4.md create mode 100644 agent-queue/docs/jobs/phase2-slice1.md diff --git a/agent-queue/docs/jobs/phase1-slice4.md b/agent-queue/docs/jobs/phase1-slice4.md new file mode 100644 index 0000000..1f05064 --- /dev/null +++ b/agent-queue/docs/jobs/phase1-slice4.md @@ -0,0 +1,125 @@ +--- +engine: devin +cwd: /Users/sd9235/code/mygh/learning_ai_devops_tools +yolo: true +lock: devops-tools +timeout: 3h +--- + +ROLE: Senior engineer. Implement Phase 1 — Slice 4: TRACKER ADAPTER (single host). +This CLOSES Phase 1: a task in the tracker can become a job, and job outcomes echo +back to the tracker — the task<->job round-trip (§10, the last Phase-1 §14 item). + +SOURCE OF TRUTH: agent-queue/docs/GIGAFACTORY_ROADMAP.md (read §10 tracker +integration, §5 manifest incl. tracker-item + idempotency-key, §24.5 one-way echo). + +PREREQUISITE / BRANCHING: +- Slice 1, Slice 3, AND Slice 2 (profiles/deps) are merged into `main`. Branch off + the CURRENT `main`. This slice MUST run AFTER Slice 2 is merged (it shares + agent-queue.sh) — do not start it until then. +- New branch: feat/gigafactory-p1-slice4. Push + open a PR. DO NOT merge. +- Keep ALL existing selftest checks green (regression). + +STRICT SCOPE: +- Edit ONLY under agent-queue/ (agent-queue.sh, selftest.sh, README.md, + docs/GIGAFACTORY_ROADMAP.md). No other repo is modified. +- You MAY READ (not edit) ../learning_ai_common_plat/services/platform-service/ + src/modules/items/{types,routes}.ts to match the real Item API contract + (paths, fields, auth header). Do not change that repo. +- bash, single host, mac+linux safe, zero new runtime deps (curl only). + +CONFIG (all via env; document in README; never hardcode URLs/tokens/secrets): +- AQ_TRACKER_API : base URL of the items API (default http://localhost:4003). +- AQ_TRACKER_TOKEN : bearer token for auth (required for real calls). +- AQ_PRODUCT_ID : productId to stamp/filter (every tracker Item has productId). +- A single `tracker_api [json]` wrapper does ALL HTTP via curl + (bearer header, content-type, base URL). It MUST be overridable for tests via + AQ_TRACKER_API_CMD (a stub script path) so selftest needs NO live service. + +DELIVERABLES + +1. `aq from-tracker ` — pull a tracker Item and materialize a job in inbox/: + - GET the item via tracker_api; map fields → job frontmatter: + title/description -> job body (the instruction markdown, verbatim) + item type/labels -> engine-class/profile/capabilities/priority where + labels carry them (e.g. label `engine-class:agentic-coder`, + `profile:backend-engineer`, `priority:high`, + `cap:os:mac`); otherwise sane defaults. + item id -> `tracker-item: ` and + `idempotency-key: tracker-` (stable). + - IDEMPOTENT: if a job for this tracker-item already exists in any stage + (reuse Slice 1 idempotency on the derived key) → no duplicate enqueue. + - On success print the created inbox filename; on missing item → clear error, nonzero. + +2. `aq to-tracker ` — push a job's CURRENT outcome to its tracker Item + (one-way echo, child -> tracker; §24.5). Only if the job meta has tracker-item. + - Map stage/result -> item status PATCH: + building/review/testing -> in_progress + shipped -> done + failed -> blocked (or the API's failure status) + note + - Post a comment/note with result, attempts, and insights summary + (duration, tokens/cost if present) — reuse Slice 3 metrics. Metrics only, + NEVER prompt content or secrets. + - IDEMPOTENT: re-running to-tracker for an unchanged outcome is a no-op + (track last-echoed state in meta, e.g. `tracker_echoed=`). + +3. Auto-echo hook (opt-in, default OFF): an env flag (e.g. AQ_TRACKER_AUTO=1) + makes the worker call `to-tracker` automatically on each stage transition it + already performs (enqueue→building→review/testing/failed/shipped). When OFF, + echo is manual via the command. Never block/fail a job because an echo failed — + log the echo error and continue (the tracker is downstream, not authoritative + for execution). + +4. `status` / `aq insights`: show the tracker-item id and last echoed status where + present (you already surface tracker-item in status from Slice 1 — extend it). + +TESTS (selftest.sh — only ADD; NO live service — use AQ_TRACKER_API_CMD stub that +returns canned JSON and records the calls it received): +- from-tracker creates an inbox job: stub returns an item JSON → + `aq from-tracker T-1` creates one inbox/*.md whose frontmatter has + tracker-item: T-1 and idempotency-key: tracker-T-1, body = item description. +- from-tracker label mapping: item with labels [engine-class:agentic-coder, + priority:high] → frontmatter reflects them. +- from-tracker idempotent: calling it twice for T-1 → exactly one job (dedupe). +- to-tracker status echo: a shipped job → stub receives a PATCH to status=done and + a comment with the insights summary; assert no prompt body is sent. +- to-tracker idempotent: second call with unchanged outcome → no duplicate + PATCH/comment (tracker_echoed honored). +- echo failure is non-fatal: stub returns HTTP 500 → `to-tracker` logs the error, + exits without corrupting job state; the job's stage is unchanged. +- REGRESSION: all existing checks (Slice 0/1/2/3) still green. + +DOCS: +- README: "Tracker integration" section — from-tracker/to-tracker, the env config, + label→manifest mapping table, the one-way-echo rule, AQ_TRACKER_AUTO, and a note + that real use needs platform-service running + a token. +- docs/GIGAFACTORY_ROADMAP.md: tick the §10 single-host items + the §14 Phase-1 + "tracker adapter" item; set §0 Phase 1 → complete (or note the exact remaining %). + +CONSTRAINTS: bash style consistent with the script; curl-only HTTP through the one +wrapper; mac+linux safe; no emojis; conventional commits; tests sacred. + +VERIFY GATE: bash agent-queue/selftest.sh fully green; bash -n agent-queue.sh; +node --check dashboard.mjs; shellcheck --severity=error clean. + +FINAL OUTPUT — print the report in EXACTLY this format: + +## Implementation Report — Phase 1 Slice 4 +### Branch & commits +- branch / based-on: +- commits: +- PR: +### Files changed +- : +### What was implemented (1-4) +- : +### Tests added +- : (+ selftest PASS/FAIL summary) +### Verify gate results +- selftest / bash -n / node --check / shellcheck: +### Deviations / assumptions +- +### Phase 1 status +- +### Suggested next slice +- Phase 2 Slice 1 (fleet data model + repositories in platform-service) diff --git a/agent-queue/docs/jobs/phase2-slice1.md b/agent-queue/docs/jobs/phase2-slice1.md new file mode 100644 index 0000000..36d8e91 --- /dev/null +++ b/agent-queue/docs/jobs/phase2-slice1.md @@ -0,0 +1,125 @@ +--- +engine: devin +cwd: /Users/sd9235/code/mygh/learning_ai_common_plat +yolo: true +lock: common-plat +timeout: 4h +--- + +ROLE: Senior backend/distributed-systems engineer. Implement Phase 2 — Slice 1: +the FLEET DATA MODEL + REPOSITORIES as a new platform-service module. This is the +durable backbone (§13) that supersedes the single-host stand-ins. NO atomic +claim/lease/fencing logic yet — that is Phase 2 Slice 2. This slice is schemas, +repositories, container registration, basic guarded CRUD, and tests. + +NOTE: This runs in a DIFFERENT repo (learning_ai_common_plat), so it does NOT +conflict with the agent-queue (devops-tools) slices and can run independently. + +READ FIRST (this is NOT the platform-service you may assume — verify conventions): +- services/platform-service/src/modules/items/{types,repository,routes}.ts — copy + this module pattern EXACTLY (types.ts -> repository.ts -> routes.ts, Zod schemas, + the cloud-agnostic datastore, productId on every doc, req.log/app.log). +- packages/cosmos (container registry) + how existing modules register containers. +- The fleet container spec in the roadmap: agent-queue/docs/GIGAFACTORY_ROADMAP.md + §13 lives in the devops-tools repo at ../learning_ai_devops_tools — read it for + the field lists (fleet_jobs incl. bodyMd + checkpoint; fleet_runs incl. token/ + cost/tool/diff insights; fleet_leases incl. leaseEpoch; fleet_factories; + fleet_profiles; fleet_events; fleet_artifacts) and §25/§26. + +PREREQUISITE / BRANCHING: +- Branch off CURRENT `main` of learning_ai_common_plat. +- New branch: feat/gigafactory-p2-slice1. Push + open a PR. DO NOT merge. + +STRICT SCOPE: +- Add a NEW module: services/platform-service/src/modules/fleet/ (+ its tests). +- Register the new Cosmos containers via the existing registration path. +- Do NOT modify unrelated modules. Do NOT hand-edit shared infra (.npmrc, + docker-prep.sh, tsconfig.base, pnpm-workspace) — those are template-managed. +- ESM everywhere ("type": "module", .js import suffixes). No `any` (Zod inference + or explicit types). No console.log (use req.log/app.log). Every Cosmos doc has + productId. Tests are sacred. + +DELIVERABLES + +1. types.ts — Zod schemas + inferred types for each container, each with productId: + - FleetJobDoc (pk /productId): manifestSnapshot, bodyMd (verbatim instructions), + stage, idempotencyKey, trackerItemId?, parentId?, kind ('leaf'|'composite', + default 'leaf'), checkpoint? { wipBranch, wipBase, wipCommit }, priority, + capabilities[], engineClass?, profile?, deps[], depsMode?, timestamps. + - FleetRunDoc (pk /jobId): jobId, attempt, factoryId?, engine, profileSnapshot?, + startedAt, endedAt?, exit?, verifyResult?, result?, and insights: model?, + tokensIn?, tokensOut?, tokensCached?, costUsd?, estimated?, turns?, toolCalls?, + filesChanged?, linesAdded?, linesDeleted?. + - FleetLeaseDoc (pk /jobId): jobId, holderFactoryId?, expiresAt?, leaseEpoch + (number, default 0), renewals, status. (Fields only — reclaim/claim logic is S2.) + - FleetFactoryDoc (pk /productId): factoryId, descriptor, capabilities[], health, + load, lastHeartbeatAt, seatLimit. + - FleetProfileDoc (pk /productId): name, version, immutable snapshot (persona, + defaults). FleetEventDoc (pk /jobId): append-only event { type, at, data }. + FleetArtifactDoc (pk /jobId): pointers to blob-stored artifacts (no inline logs). + - Define enums for stage and result that MATCH the agent-queue lifecycle. + +2. repository.ts — one repository per container using the existing datastore + abstraction (so DB_PROVIDER=memory works in tests, cosmos in prod): + - CRUD: create, getById, list (by productId; jobs also by stage), update + (optimistic via _etag where the datastore supports it — expose the etag, + even though the ATOMIC claim flow is S2), delete where sensible. + - appendEvent(jobId, event) for the append-only fleet_events stream. + - All queries partition-aware; no cross-partition fan-out in hot paths. + +3. container registration — register all fleet_* containers with correct partition + keys via the existing cosmos container registry; memory provider auto-handles. + +4. routes.ts — minimal guarded REST under the existing auth + productId middleware: + - POST /fleet/jobs (create), GET /fleet/jobs (list by stage/productId), + GET /fleet/jobs/:id, PATCH /fleet/jobs/:id (stage/fields), and read endpoints + for runs (GET /fleet/jobs/:id/runs) + events. Keep it thin — claim/lease + endpoints are S2. Validate all bodies with the Zod schemas. + - Register the route module in the platform-service app the same way items does. + +TESTS (Vitest — write alongside; memory provider; tests sacred): +- schema validation: valid docs pass; missing productId / bad enum fail with + precise errors; at least one invalid case per container. +- repository CRUD round-trip per container (create→get→list→update→delete) on the + memory provider; list filters by productId and by stage (jobs). +- appendEvent produces an ordered, append-only stream for a jobId. +- routes: create+get+list+patch a job via fastify inject (use the shared testing + helpers); auth/productId enforced; invalid body rejected. +- _etag surfaced on update (lost-update guard groundwork) — assert the etag flows. + +VERIFY GATE (must pass): +- pnpm --filter @lysnrai/platform-service typecheck +- pnpm --filter @lysnrai/platform-service test (new tests green; none weakened) +- pnpm --filter @lysnrai/platform-service build + +DOCS: +- Short module README or header docblock describing the containers + that + claim/lease/fencing is Phase 2 Slice 2. +- In ../learning_ai_devops_tools roadmap you may NOT edit (different repo) — instead + note in your report which §13 items are now satisfied so I can tick them. + +CONSTRAINTS: follow the items-module conventions precisely; ESM .js imports; no any; +no console.log; productId everywhere; conventional commits (feat(platform-service): +...); do not touch template-managed infra files. + +FINAL OUTPUT — print the report in EXACTLY this format: + +## Implementation Report — Phase 2 Slice 1 +### Branch & commits +- branch / based-on / PR +- commits: +### Files changed +- : +### What was implemented (1-4) +- containers + schemas + repos + routes; partition keys; etag handling +### Tests added +- : (+ pnpm test summary: N passed) +### Verify gate results +- typecheck / test / build: +### §13 items now satisfied +- +### Deviations / assumptions +- +### Suggested next slice +- Phase 2 Slice 2: atomic claim (_etag/If-Match) + lease renew/release + heartbeat + + reaper + fencing (leaseEpoch) — the concurrency core.