diff --git a/agent-queue/README.md b/agent-queue/README.md index 46c333e..516cc76 100644 --- a/agent-queue/README.md +++ b/agent-queue/README.md @@ -109,10 +109,10 @@ are otherwise **no-ops until a later phase** (they do not yet affect execution). | `prefers-engine` | **active** | _(none)_ | optional order hint for `engine-class` resolution, e.g. `[claude, devin]` | | `capabilities` | **active** | _(none)_ | hard host requirements, e.g. `[os:any, node>=20, has:git]`. If the host can't satisfy them the job is sent to `failed/` with `result=capability_mismatch` **and the agent is never launched** (grammar below) | | `idempotency-key` | **active** | _(none)_ | dedupe on `add` (semantics below) | -| `profile` | RESERVED | _(none)_ | role/persona + caps (profiles land in a later slice) | +| `profile` | **active** | _(none)_ | inherit persona + verify/caps/engine-class/prefers-engine/allowed-scope/review-policy from `profiles/.md` (job fields override — see **Profiles**) | | `prefers` | RESERVED | _(none)_ | soft routing/affinity hints (e.g. `[factory:mac-2]`) | | `budget` | RESERVED | _(none)_ | `{ usd, tokens, wall }` ceilings (`wall` enforcement is a later slice) | -| `deps` / `deps-mode` | RESERVED | _(none)_ | DAG dependencies (single-host blocking is a later slice) | +| `deps` / `deps-mode` | **active** | _(none)_ | block until each referenced `idempotency-key` is in `shipped/` (or `testing/` when `deps-mode: soft`). Submit-time cycle detection (see **Profiles & deps**) | | `retry` | **active** | _(none)_ | `{ max: N, backoff: 5m, on: [timeout, verify_failed, crash] }` — requeue failures with backoff up to `max`, then `retries_exhausted` (see **Resilience**) | | `review-policy` | RESERVED | _(none)_ | `auto\|manual\|reviewers:[…]` | | `artifacts` | RESERVED | _(none)_ | extra outputs to capture (coverage, screenshots) | @@ -232,6 +232,42 @@ queue/ (transient: requeued for another attempt), `recovered` (transient: an orphan was reclaimed to `inbox/`). +## Profiles & deps + +### Profiles (roadmap §6) + +A **profile** is a reusable role preset in `profiles/.md`. A job opts in with +`profile: ` and inherits any of these fields it does **not** set itself: +`verify` (from the profile's `default-verify`), `capabilities`, `engine-class`, +`prefers-engine`, `allowed-scope`, `review-policy`. The profile's `persona` block is +**prepended** to the body sent to the engine (the job `.md` on disk is unchanged; +secrets are never logged). Resolution runs **before** the capability gate and engine +resolution, so inherited caps / engine-class take effect. + +**Precedence:** `job field > profile field > built-in default`. Set `AGENT_QUEUE_PROFILES` +to point at a different catalog directory (defaults to `./profiles`). + +Starter catalog: `developer`, `backend-engineer`, `frontend-engineer`, `ux-designer`, +`ui-designer`, `qa`, `reviewer`, `docs-writer`, and a reserved `planner`. Each presets +`name`, `persona`, `capabilities`, `default-verify`, `engine-class`, `prefers-engine`, +`allowed-scope`, and `review-policy`. + +**allowed-scope (warn-only this phase).** After a run on a git `cwd`, changed paths +outside the profile/job `allowed-scope` globs (`dir/**` matches the whole subtree) are +logged as a `WARNING` and recorded as `scope_warning=` in the meta — **non-blocking** +(the job is not failed). `path_in_scope` is exposed as a unit-testable function. + +### deps / DAG, single host (roadmap §5) + +`deps: [keyA, keyB]` references other jobs by their author-controlled +`idempotency-key`. A dep is **satisfied** when a job with that key is in `shipped/` +(default), or in `shipped/` **or** `testing/` when the dependent job sets +`deps-mode: soft`. A job with unmet deps is **blocked**: it is skipped in inbox +selection (never launched, never failed) and surfaced in `status` as +`blocked (waiting on: )`, then re-evaluated every loop until its deps are met. +`add` performs **submit-time cycle detection** over the inbox + active-stage dep graph +and rejects (nonzero exit) a job that would create a cycle. Cross-machine deps are P2. + ## Resilience (crash recovery & work preservation) Single-host implementations of the durability model (roadmap §25): diff --git a/agent-queue/docs/GIGAFACTORY_ROADMAP.md b/agent-queue/docs/GIGAFACTORY_ROADMAP.md index f202805..d49dd0a 100644 --- a/agent-queue/docs/GIGAFACTORY_ROADMAP.md +++ b/agent-queue/docs/GIGAFACTORY_ROADMAP.md @@ -11,7 +11,7 @@ | Phase | Theme | Status | % | Gate | | ----- | ----- | ------ | - | ---- | | **0** | Baseline (today) | ✅ shipped | 100% | `selftest.sh` green | -| **1** | Manifest + profiles + capabilities + tracker adapter (single host) | ◐ in progress | 55% | adapter e2e + selftest | +| **1** | Manifest + profiles + capabilities + tracker adapter (single host) | ◐ in progress | 80% | adapter e2e + selftest | | **2** | Coordinator as platform-service module + Cosmos + multi-factory leasing | ☐ not started | 0% | fleet e2e + module tests | | **3** | Fleet control plane in tracker-web + DAG deps + budgets + scoring router | ☐ not started | 0% | web e2e + router tests | | **4** | Message bus + autoscaling + cross-OS capability marketplace | ☐ not started | 0% | load/chaos suite | @@ -142,7 +142,7 @@ tracker-item: ITEM-789 # link back to the originating tracker task - [x] **Capability grammar** defined: tokens are `key` (presence, e.g. `has:xcode`), `key:value` (e.g. `os:mac`, `engine:devin`), or `keyversion` with `op ∈ {>=,>,=,<=,<}` (e.g. `node>=20`). `os:any` is a wildcard that matches every factory. A job matches a factory iff every required token is satisfied by the factory descriptor. *(P1-S1: `caps_match`/`detect_capabilities` in `agent-queue.sh`.)* - [x] **`engine-class` taxonomy** defined as an enum (`agentic-coder`, `chat-coder`, `review-only`) with a documented engine→class map (`devin,claude,codex → agentic-coder`; `copilot → chat-coder`). If `engine` is set it wins; else the scheduler picks any free engine in the class honoring `prefers-engine`. *(P1-S1: `resolve_engine`; `review-only` mapping reserved.)* - [x] **`idempotency-key` semantics:** `key + content-hash` identical ⇒ no-op (returns existing job). Same `key`, **different** content ⇒ **rejected with 409** unless the prior job is still `queued`/`blocked` (then it is superseded). A re-`run`/`retry` of an existing job is **not** a new submit and never trips dedupe. *(P1-S1: add-time dedupe; bash maps "409" → clear error, `queued` → still in `inbox/` ⇒ superseded.)* -- [ ] **`deps` semantics:** a dep is satisfied when it reaches `shipped` (default) or `testing` if `deps-mode: soft`. Submit-time **cycle detection** rejects cyclic graphs; unmet deps put the job in `blocked` (not `queued`). Cross-factory deps require the coordinator (P2); single-host deps work in P1. +- [x] **`deps` semantics:** a dep is satisfied when it reaches `shipped` (default) or `testing` if `deps-mode: soft`. Submit-time **cycle detection** rejects cyclic graphs; unmet deps put the job in `blocked` (not `queued`). Cross-factory deps require the coordinator (P2); single-host deps work in P1. *(P1-S2: `deps_unmet` skip-with-reason in selection + `status` surfacing; `deps_would_cycle` on `add`. Cross-machine deps remain P2.)* - **Acceptance:** a manifest fixture suite parses/validates; invalid manifests fail with precise errors; capability-grammar + dep-cycle + idempotency-conflict cases covered. - **Verify gate:** schema unit tests (≥ 1 per field incl. defaults + 5 invalid cases + grammar/cycle/409 cases). @@ -167,11 +167,11 @@ review-policy: manual --- ``` -- [ ] Author starter catalog: `developer`, `backend-engineer`, `frontend-engineer`, `ux-designer`, `ui-designer`, `qa`, `reviewer`, `docs-writer`. -- [ ] Persona overlay is **prepended** to the job body before the agent runs; secrets are never written to logs or the event stream (redaction at the source). -- [ ] Profile supplies default `verify`, `capabilities`, `engine-class`, `allowed-scope` when the job omits them. -- [ ] Profile versioning: changing a profile doesn't mutate in-flight jobs (snapshot at assign time). -- [ ] `allowed-scope` enforced as a guardrail (warn in P1, enforce/deny in P2 via pre-flight diff check). +- [x] Author starter catalog: `developer`, `backend-engineer`, `frontend-engineer`, `ux-designer`, `ui-designer`, `qa`, `reviewer`, `docs-writer`. *(P1-S2: `profiles/*.md` + a reserved `planner`.)* +- [x] Persona overlay is **prepended** to the job body before the agent runs; secrets are never written to logs or the event stream (redaction at the source). *(P1-S2: `profile_persona` prepended to the stripped body file.)* +- [x] Profile supplies default `verify`, `capabilities`, `engine-class`, `allowed-scope` when the job omits them. *(P1-S2: `fm_eff` — also `prefers-engine` + `review-policy`; job fields always override.)* +- [ ] Profile versioning: changing a profile doesn't mutate in-flight jobs (snapshot at assign time). *(P2 — needs Cosmos snapshot at assign time.)* +- [x] `allowed-scope` enforced as a guardrail (warn in P1, enforce/deny in P2 via pre-flight diff check). *(P1-S2: `scope_check` post-run WARN-only + `scope_warning=` in meta; `path_in_scope` unit-testable.)* - **Acceptance:** a job with `profile: backend-engineer` and no `verify` inherits the profile's verify + persona. - **Verify gate:** profile-resolution unit tests; persona-injection golden test. @@ -342,18 +342,20 @@ Each phase: **Goal → checklist → Exit criteria**. Don't start a phase until > **Slice progress — P1-S1:** manifest parsing (all §5 fields, defaulted + backward-compatible), `priority` ordering, capability detection+match gate, `engine-class` resolution, and `idempotency-key` dedupe are **done** on the bash runner. > -> **Slice progress — P1-S3 (resilience & insights, single host):** crash recovery (`recover_orphans` + `aq recover`), git WIP checkpoint/resume (`aq/wip/`), functional `retry` policy (backoff + `retries_exhausted`), and execution insights (`parse_usage`, per-run metrics in meta, `aq insights`, `status`/`dash` insights) are **done** — see §11/§25/§26. Profiles, `deps` DAG, `budget.wall`, `allowed-scope`, and the tracker adapter remain **for later slices**. +> **Slice progress — P1-S3 (resilience & insights, single host):** crash recovery (`recover_orphans` + `aq recover`), git WIP checkpoint/resume (`aq/wip/`), functional `retry` policy (backoff + `retries_exhausted`), and execution insights (`parse_usage`, per-run metrics in meta, `aq insights`, `status`/`dash` insights) are **done** — see §11/§25/§26. +> +> **Slice progress — P1-S2 (profiles + deps/DAG, single host):** the `profiles/` catalog + resolution (`fm_eff` inheritance with job>profile>default precedence, persona injection), the warn-only `allowed-scope` guardrail (`scope_check`/`path_in_scope`), and single-host `deps` (block-with-reason in selection, `status` surfacing, submit-time cycle detection) are **done** — see §5/§6. The tracker adapter and `budget.wall` remain **for later slices**. - [x] Extend `agent-queue.sh` frontmatter parsing for all new manifest fields (§5), defaulted + backward-compatible. *(P1-S1)* -- [ ] Add `profiles/` directory + profile resolution (persona injection, default verify/caps/scope) (§6). +- [x] Add `profiles/` directory + profile resolution (persona injection, default verify/caps/scope) (§6). *(P1-S2)* - [x] Local capability detection + a job/factory capability match check before launch (§8 subset). *(P1-S1: `detect_capabilities` + `caps_match`; mismatch ⇒ `failed/` `result=capability_mismatch`, agent never launched.)* - [x] `priority` ordering in the inbox pick (replace pure FIFO with priority-then-age). *(P1-S1: `inbox_sorted`; per-lock serialization preserved.)* -- [ ] `deps` (DAG) blocking on a single host; `idempotency-key` dedupe on `add`. *(P1-S1: `idempotency-key` dedupe DONE; `deps` DAG blocking still pending.)* +- [x] `deps` (DAG) blocking on a single host; `idempotency-key` dedupe on `add`. *(P1-S1 idempotency dedupe + P1-S2 `deps` blocking/cycle detection.)* - [ ] `retry` with backoff into `failed`/requeue; `budget.wall` enforced (extends `timeout`). *(P1-S3: `retry` with backoff + `retries_exhausted` DONE; `budget.wall` still pending.)* -- [ ] `allowed-scope` guardrail (warn-only this phase) + post-run diff report. +- [x] `allowed-scope` guardrail (warn-only this phase) + post-run diff report. *(P1-S2: `scope_check` WARN-only + `scope_warning=`.)* - [ ] **Tracker adapter** `aq from-tracker ` + `aq to-tracker` event poster (§10 P1). - [ ] Dashboard shows profile + priority + capability tags + tracker-item link. *(P1-S1: `status` shows priority/profile/caps/tracker-item; Node `dash` surfacing pending.)* -- [ ] Update `selftest.sh` with: manifest parse fixtures, profile resolution, priority order, dep-block, idempotency, adapter round-trip (mock). *(P1-S1: added backward-compat, priority, capability-mismatch, engine-class, idempotency cases; profile/dep-block/adapter pending.)* +- [ ] Update `selftest.sh` with: manifest parse fixtures, profile resolution, priority order, dep-block, idempotency, adapter round-trip (mock). *(P1-S1 manifest/priority/idempotency + P1-S2 profile resolution/persona/scope/dep-block/cycle + P1-S3 resilience/insights; tracker adapter round-trip still pending.)* - [x] Update README + this doc's progress table. *(P1-S1)* - **Exit criteria:** all boxes ✅; `selftest.sh` green; a tracker task → executed → tracker `done` with SHA comment, fully on one host; no regression to Phase-0 `.md` files.