diff --git a/agent-queue/README.md b/agent-queue/README.md index 516cc76..9c3a453 100644 --- a/agent-queue/README.md +++ b/agent-queue/README.md @@ -317,6 +317,67 @@ agent-queue.sh insights # recent-jobs table + per-engine rollup > numbers are never fabricated. The per-engine rollup marks totals that include any > estimated value with `*`. +## Tracker integration (§10) + +Closes the task ↔ job round-trip against the platform-service **items API**: a +tracker Item can become a job, and a job's outcome echoes back to the Item. + +```bash +agent-queue.sh from-tracker # pull an Item -> materialize a job in inbox/ +agent-queue.sh to-tracker # echo the job's current outcome to its Item +``` + +All HTTP goes through one curl wrapper (`tracker_api`); there are no other network +calls. Real use needs **platform-service running and a bearer token**. + +### Config (env) + +| Var | Default | Meaning | +| --- | ------- | ------- | +| `AQ_TRACKER_API` | `http://localhost:4003` | base URL of the items API (routes live under `/api`) | +| `AQ_TRACKER_TOKEN` | _(none)_ | bearer token — **required** for real calls; never hardcode | +| `AQ_PRODUCT_ID` | _(none)_ | productId (sent as `X-Product-Id`; every Item has one) | +| `AQ_TRACKER_CWD` | `$PWD` | cwd a tracker-derived job runs in (Items carry no cwd) | +| `AQ_TRACKER_AUTO` | `0` | `1` = auto-echo on each transition (default OFF — echo is manual) | +| `AQ_TRACKER_STATUS_INPROGRESS` / `_DONE` / `_FAILED` | `in_progress` / `done` / `wont_fix` | Item status per bucket (the API has no blocked/failed status) | +| `AQ_TRACKER_API_CMD` | _(none)_ | test seam: a stub that replaces the curl HTTP entirely (selftest uses it) | + +### `from-tracker` — Item → job + +`GET /api/items/`, then maps fields to job frontmatter: + +| Item | Job | +| ---- | --- | +| `title` + `description` | job body (verbatim instruction markdown) | +| `id` | `tracker-item: ` and `idempotency-key: tracker-` (stable) | +| `priority` | `priority:` (label overrides; else Item priority; else `medium`) | +| label `engine-class:` | `engine-class: ` | +| label `profile:` | `profile: ` | +| label `priority:` | `priority: ` | +| label `cap:` | a `capabilities: [...]` entry | + +Idempotent on the derived `idempotency-key` (Slice 1 dedupe) — pulling the same +Item twice never enqueues a duplicate. + +### `to-tracker` — job → Item (one-way echo, §24.5) + +Only if the job's meta has a `tracker-item`. Maps the job's stage/result to an Item +status and `PATCH /api/items//status`, then `POST /api/items//comments` +with a **metrics-only** summary (result, attempts, duration, tokens/cost, +/- lines — +**never prompt content or secrets**): + +| job result/stage | Item status | +| ---------------- | ----------- | +| building / review / testing / recovered | `in_progress` | +| shipped | `done` | +| failed / timeout / verify_failed / retries_exhausted / capability_mismatch / no_engine / rejected | `wont_fix` (override via `AQ_TRACKER_STATUS_FAILED`) | + +Idempotent via `tracker_echoed` in the meta (re-echoing an unchanged outcome is a +no-op). The echo is **one-way** (child → tracker) and **never authoritative for +execution**: an echo failure is logged and the job continues unchanged. With +`AQ_TRACKER_AUTO=1` the worker echoes automatically on each transition; otherwise +echo is manual. `status` / `insights` surface the `tracker-item` and last echoed status. + ## Config (env overrides) | Var | Default | Meaning | diff --git a/agent-queue/docs/GIGAFACTORY_ROADMAP.md b/agent-queue/docs/GIGAFACTORY_ROADMAP.md index e50fa0a..1d64068 100644 --- a/agent-queue/docs/GIGAFACTORY_ROADMAP.md +++ b/agent-queue/docs/GIGAFACTORY_ROADMAP.md @@ -11,7 +11,7 @@ | Phase | Theme | Status | % | Gate | | ----- | ----- | ------ | - | ---- | | **0** | Baseline (today) | ✅ shipped | 100% | `selftest.sh` green | -| **1** | Manifest + profiles + capabilities + tracker adapter (single host) | ◐ in progress | 80% | adapter e2e + selftest | +| **1** | Manifest + profiles + capabilities + tracker adapter (single host) | ◐ in progress | 95% | adapter e2e + selftest | | **2** | Coordinator as platform-service module + Cosmos + multi-factory leasing | ☐ not started | 0% | fleet e2e + module tests | | **3** | Fleet control plane in tracker-web + DAG deps + budgets + scoring router | ☐ not started | 0% | web e2e + router tests | | **4** | Message bus + autoscaling + cross-OS capability marketplace | ☐ not started | 0% | load/chaos suite | @@ -253,10 +253,10 @@ Three transports were evaluated. **Decision: platform-service-native coordinator **Layering:** tracker = *WHAT/WHY* (plan, intake, prioritize, roadmap, votes) · gigafactory = *HOW* (execute) · platform-service = shared brain · agent-queue runner = offline edge. Grounded in the real `tracker-service` model (`Item`: `type` bug/feature/**task**, `status` open/in_progress/done/closed/wont_fix, priority, labels, assignee, `source` incl. **auto_detected**, votes, comments, public roadmap) and the `tracker-web` `/api/tracker/[...path]` proxy pattern. ### Phase 1 — Adapter (no new infra) -- [ ] **task → job**: a tracker `Item` of `type: task` (e.g. `assignee: @agent` or label `agent:run`) is exported to a job `.md` (manifest mapped: title/description → body, priority → priority, labels → capabilities/profile hints). -- [ ] **job → tracker**: lifecycle events post back as **status updates + comments** — `building` → status `in_progress` + comment "started on factory X"; `shipped` → `done` + comment with commit SHAs / PR link / verify results; `failed` → comment with reason (status stays `in_progress` for human triage). -- [ ] Idempotency: re-running the adapter for the same item doesn't create duplicate jobs (idempotency-key = item id + content hash). -- [ ] Adapter is a thin script/CLI (`aq from-tracker ITEM-789`) + optional poller. +- [x] **task → job**: a tracker `Item` of `type: task` (e.g. `assignee: @agent` or label `agent:run`) is exported to a job `.md` (manifest mapped: title/description → body, priority → priority, labels → capabilities/profile hints). *(P1-S4: `aq from-tracker`; labels `engine-class:`/`profile:`/`priority:`/`cap:` → frontmatter.)* +- [x] **job → tracker**: lifecycle events post back as **status updates + comments** — `building` → status `in_progress` + comment "started on factory X"; `shipped` → `done` + comment with commit SHAs / PR link / verify results; `failed` → comment with reason (status stays `in_progress` for human triage). *(P1-S4: `aq to-tracker` PATCHes status + posts a metrics-only comment; one-way echo §24.5; never fatal. The items API has no blocked/failed status, so failures map to `wont_fix` by default — override via `AQ_TRACKER_STATUS_FAILED`.)* +- [x] Idempotency: re-running the adapter for the same item doesn't create duplicate jobs (idempotency-key = item id + content hash). *(P1-S4: derived `idempotency-key: tracker-` reuses Slice 1 dedupe; `to-tracker` is idempotent via `tracker_echoed`.)* +- [x] Adapter is a thin script/CLI (`aq from-tracker ITEM-789`) + optional poller. *(P1-S4: `from-tracker`/`to-tracker` + opt-in `AQ_TRACKER_AUTO` auto-echo; a standalone poller is deferred.)* - **Acceptance:** filing a tracker task, marking it `agent:run`, results in a queued job; on ship, the item flips to `done` with a SHA comment. - **Verify gate:** adapter e2e against a tracker-service test instance (or mock); round-trip assertion. @@ -347,7 +347,9 @@ Each phase: **Goal → checklist → Exit criteria**. Don't start a phase until > > **Slice progress — P1-S3 (resilience & insights, single host):** crash recovery (`recover_orphans` + `aq recover`), git WIP checkpoint/resume (`aq/wip/`), functional `retry` policy (backoff + `retries_exhausted`), and execution insights (`parse_usage`, per-run metrics in meta, `aq insights`, `status`/`dash` insights) are **done** — see §11/§25/§26. > -> **Slice progress — P1-S2 (profiles + deps/DAG, single host):** the `profiles/` catalog + resolution (`fm_eff` inheritance with job>profile>default precedence, persona injection), the warn-only `allowed-scope` guardrail (`scope_check`/`path_in_scope`), and single-host `deps` (block-with-reason in selection, `status` surfacing, submit-time cycle detection) are **done** — see §5/§6. The tracker adapter and `budget.wall` remain **for later slices**. +> **Slice progress — P1-S2 (profiles + deps/DAG, single host):** the `profiles/` catalog + resolution (`fm_eff` inheritance with job>profile>default precedence, persona injection), the warn-only `allowed-scope` guardrail (`scope_check`/`path_in_scope`), and single-host `deps` (block-with-reason in selection, `status` surfacing, submit-time cycle detection) are **done** — see §5/§6. +> +> **Slice progress — P1-S4 (tracker adapter, single host):** the task ↔ job round-trip is **done** (§10) — `aq from-tracker` materializes a job from a tracker Item (idempotent on `tracker-`, label→manifest mapping), `aq to-tracker` echoes status + a metrics-only comment one-way (idempotent via `tracker_echoed`, never fatal), and opt-in `AQ_TRACKER_AUTO` auto-echoes on transitions. All HTTP is curl-only through one wrapper (test seam `AQ_TRACKER_API_CMD`). **This closes the Phase-1 §14 tracker-adapter item.** Remaining P1 extras: `budget.wall` (P1-S3 left it) and Node-`dash` surfacing of the new fields. - [x] Extend `agent-queue.sh` frontmatter parsing for all new manifest fields (§5), defaulted + backward-compatible. *(P1-S1)* - [x] Add `profiles/` directory + profile resolution (persona injection, default verify/caps/scope) (§6). *(P1-S2)* @@ -356,9 +358,9 @@ Each phase: **Goal → checklist → Exit criteria**. Don't start a phase until - [x] `deps` (DAG) blocking on a single host; `idempotency-key` dedupe on `add`. *(P1-S1 idempotency dedupe + P1-S2 `deps` blocking/cycle detection.)* - [ ] `retry` with backoff into `failed`/requeue; `budget.wall` enforced (extends `timeout`). *(P1-S3: `retry` with backoff + `retries_exhausted` DONE; `budget.wall` still pending.)* - [x] `allowed-scope` guardrail (warn-only this phase) + post-run diff report. *(P1-S2: `scope_check` WARN-only + `scope_warning=`.)* -- [ ] **Tracker adapter** `aq from-tracker ` + `aq to-tracker` event poster (§10 P1). -- [ ] Dashboard shows profile + priority + capability tags + tracker-item link. *(P1-S1: `status` shows priority/profile/caps/tracker-item; Node `dash` surfacing pending.)* -- [ ] Update `selftest.sh` with: manifest parse fixtures, profile resolution, priority order, dep-block, idempotency, adapter round-trip (mock). *(P1-S1 manifest/priority/idempotency + P1-S2 profile resolution/persona/scope/dep-block/cycle + P1-S3 resilience/insights; tracker adapter round-trip still pending.)* +- [x] **Tracker adapter** `aq from-tracker ` + `aq to-tracker` event poster (§10 P1). *(P1-S4: curl-only `tracker_api`; from-tracker materializes a job (idempotent), to-tracker echoes status+metrics one-way; opt-in `AQ_TRACKER_AUTO`. A standalone background poller is deferred to P2.)* +- [ ] Dashboard shows profile + priority + capability tags + tracker-item link. *(P1-S1: `status` shows priority/profile/caps/tracker-item; P1-S4: status/insights also show last echoed tracker status; Node `dash` surfacing pending.)* +- [x] Update `selftest.sh` with: manifest parse fixtures, profile resolution, priority order, dep-block, idempotency, adapter round-trip (mock). *(P1-S1 manifest/priority/idempotency + P1-S2 profile/persona/scope/dep-block/cycle + P1-S3 resilience/insights + P1-S4 tracker from/to round-trip via stub.)* - [x] Update README + this doc's progress table. *(P1-S1)* - **Exit criteria:** all boxes ✅; `selftest.sh` green; a tracker task → executed → tracker `done` with SHA comment, fully on one host; no regression to Phase-0 `.md` files.