diff --git a/agent-queue/docs/jobs/phase3-overnight.md b/agent-queue/docs/jobs/phase3-overnight.md new file mode 100644 index 0000000..5494d69 --- /dev/null +++ b/agent-queue/docs/jobs/phase3-overnight.md @@ -0,0 +1,162 @@ +--- +engine: devin +cwd: /Users/sd9235/code/mygh/learning_ai_common_plat +yolo: true +lock: common-plat-phase3 +timeout: 10h +--- + +ROLE: Senior full-stack engineer. Implement PHASE 3 of the Agent Gigafactory END-TO-END +in `learning_ai_common_plat`, SEQUENTIALLY, over a long unattended run: smart routing +(tunable weights + preemption), DAG job decomposition, per-product budgets, and the +tracker-web fleet control plane. Work SLICE BY SLICE; each slice is self-contained, +fully tested, and pushed before the next begins. This is an overnight run — favor +correctness, small verifiable steps, and never leaving main/PR in a broken state. + +================================================================================ +PREREQUISITE (the operator guarantees this before starting): Phase 2 is COMPLETE and +merged to origin/main — fleet foundation, atomic claim, scheduler/router core, artifacts, +enrollment+tokens, feature-flags/shadow, the in-process tracker->module wiring, and the +two-factory demo are ALL on main. You branch off CURRENT origin/main. +================================================================================ + +GLOBAL GUARDRAILS (unattended danger mode — obey strictly): +- Branch: feat/gigafactory-phase3 off CURRENT origin/main. ONE long-lived branch; ONE + commit per slice (conventional commits). Push after EVERY slice. Open ONE PR after + Slice 1 and keep pushing to it. DO NOT MERGE anything. DO NOT touch origin/main. +- Tests are SACRED: never delete, weaken, skip, or `.skip`/`.only` a test to go green. + If you cannot make a slice pass honestly, see the FAILURE PROTOCOL below. +- A slice is "done" only when its VERIFY GATE is fully green. Never start slice N+1 with + slice N red. +- Reserved / DO NOT TOUCH: the agent-queue repo (different repo), unrelated services + (cowork-service, extraction-service), packages/* internals (consume, don't edit), + and any backup/* or dependabot/* branches. Stay in services/platform-service + + dashboards/tracker-web. +- Conventions: ESM `.js` import specifiers; no `any`; no console.log (use app.log/req.log, + and the tracker-web logger/telemetry pattern); every Cosmos doc carries `productId`; + reuse @bytelyst/* packages and existing module patterns (types.ts -> repository.ts -> + routes.ts). Do NOT hardcode colors/URLs/secrets. +- CHECKPOINTING: maintain docs/gigafactory-phase3-progress.md on the branch. After each + slice, record: slice name, status (DONE/WIP/FAILED), commit sha, verify-gate result, + and any follow-ups. Commit it WITH the slice. If you resume after an interruption, read + it first and continue from the first not-DONE slice. + +FAILURE PROTOCOL (per slice): attempt the verify gate up to 3 times, fixing the ROOT +cause each time (not the test). If still red after 3 honest attempts: commit the WIP with +message `wip(): — BLOCKED: `, mark it FAILED in +progress.md with the exact failing output, and MOVE ON to the next slice that does NOT +depend on it (dependencies noted per slice). Never thrash; never fake green. + +READ FIRST: +- ../learning_ai_devops_tools/agent-queue/docs/GIGAFACTORY_ROADMAP.md — §7 (scoring; + Phase-3 = tunable weights + preemption), §5/§6 (DAG/deps), §11/§13 (budgets), §14 + Phase-3 checklist + Exit criteria, §16 Definition-of-Done. +- services/platform-service/src/modules/fleet/{scheduler,coordinator,repository,routes, + types}.ts — the engine you extend (read the existing claim/lease/fence/selectJob). +- dashboards/tracker-web/ — match its App-Router structure (src/app, src/app/api), + data-fetching/auth pattern, @bytelyst/ui + design-tokens usage, vitest + Playwright + (e2e/, playwright.config.ts) setup. The existing fleet HTTP API you consume: + POST/GET /fleet/jobs, GET /fleet/jobs/:id, PATCH /fleet/jobs/:id, POST /fleet/claim, + lease renew/release, POST /fleet/factories/heartbeat|enroll, token rotate/revoke, + GET /fleet/jobs/:id/runs, GET /fleet/jobs/:id/events, artifacts upload/list/get/delete. + +-------------------------------------------------------------------------------- +SLICE 1 — Tunable scoring weights + preemption (backend; depends on: nothing) +-------------------------------------------------------------------------------- +Extend the PURE scheduler (scheduler.ts) without breaking §7 Phase-2 behavior: +- Weights become configurable per-product/per-request (passed in; fixed defaults preserved + so existing tests stay green). Add a small typed FleetWeightConfig resolver (defaults -> + optional product override). NO env reads inside the pure module. +- Preemption: a `selectWithPreemption(candidates, runningJobs, factory, ctx, weights?)` + that, when a CRITICAL job cannot be placed and only lower-priority jobs are running, + returns a preemption decision { evict: jobId, reason, breakdown } — PURE, no I/O. +- Wire preemption into the coordinator behind a flag (FLEET_PREEMPTION, default OFF; OFF = + byte-for-byte current behavior). Eviction must checkpoint + requeue the victim via the + EXISTING fenced-requeue path (bump leaseEpoch; the zombie's late report is fenced). +TESTS: weight override changes ranking; defaults reproduce all prior picks; preemption +evicts only a strictly-lower-priority running job, never an equal/higher; victim is +requeued with checkpoint + bumped epoch and its stale report is fenced; flag OFF = no +preemption. VERIFY GATE: pnpm --filter @lysnrai/platform-service exec vitest run +src/modules/fleet && build && (pnpm build && pnpm test). + +-------------------------------------------------------------------------------- +SLICE 2 — DAG job decomposition (backend; depends on: nothing; independent of S1) +-------------------------------------------------------------------------------- +Parent/child jobs with dependency-gated execution (§5/§6): +- types: a job may declare children (subtasks) and dependsOn[] (sibling/child ids). Reuse + existing kind ('leaf'|...) + parentId; add child submission + a DAG edge model. Cycle + detection at submit (extend the existing submit-time cycle check). +- coordinator: a parent is not claimable until its children reach a terminal state (or its + declared deps are satisfied); completing the last child unblocks the parent. claimNextJob + only returns deps-satisfied jobs (extend the existing predicate). Fan-out: submitting a + parent with children atomically creates the children. +- routes (additive): POST /fleet/jobs/:id/children (submit children), GET /fleet/jobs/:id/dag + (return the subtree + per-node stage). productId-scoped. +TESTS: parent blocked until children done; last child completion unblocks parent; cycle at +submit -> rejected; capability/priority still respected per node; DAG endpoint returns the +correct subtree; all prior fleet tests green. VERIFY GATE as in Slice 1 (+ items unaffected). + +-------------------------------------------------------------------------------- +SLICE 3 — Per-product budgets + pause/resume (backend; depends on: nothing) +-------------------------------------------------------------------------------- +Cost ceilings that pause routing (§11/§13): +- A FleetBudgetDoc per productId (ceilingUsd, window, spentUsd, status active|paused). + Spend accrues from job run cost (reuse run/insights cost if present; else estimate from + budget.usd at completion). Container partitioned by /productId. +- Enforcement in claimNextJob: if the product's budget is paused or the next job would + exceed the ceiling, that product's jobs are NOT claimed (other products unaffected). + Behind FLEET_BUDGETS (default OFF = unchanged). +- routes (additive): GET/PUT /fleet/budgets/:productId, POST /fleet/budgets/:productId/pause, + POST /fleet/budgets/:productId/resume. +TESTS: under ceiling -> claims proceed; crossing ceiling -> that product pauses, others +still claim; manual pause blocks claims; resume restores; flag OFF = no enforcement; +spend accounting is monotonic + idempotent per run. VERIFY GATE as above. + +-------------------------------------------------------------------------------- +SLICE 4 — tracker-web Fleet Control Plane UI (frontend; depends on: S1-S3 endpoints, + but build defensively — feature-detect/degrade if an endpoint is absent) +-------------------------------------------------------------------------------- +A new `/fleet` section in dashboards/tracker-web (App Router), matching existing patterns: +- Typed fleet API client (src/lib or src/app/api proxy as the repo does it) wrapping the + fleet endpoints with auth token injection (reuse the existing auth/client pattern). +- Pages/components (use @bytelyst/ui + --*-tokens; every interactive element has an + aria-label or visible label): + * Fleet map: factories (id, caps, health, load, lease state) as live cards. + * Job table: filter by product/stage/priority; submit-job modal; row -> job detail. + * Job detail: stage timeline from /events, runs from /runs, artifacts list, a SHIP + action (PATCH stage), and the DAG subtree (from /dag) when present. + * Budget panel: per-product ceiling + spent + pause/resume controls. +- Live updates via polling (simple, robust) unless an SSE/stream endpoint exists. +TESTS: vitest component/unit tests for the client + key components (render, actions call +the right endpoint, error/empty/degraded states); Playwright e2e for the core flow +(see fleet map -> open a job -> ship; pause a budget -> resume). VERIFY GATE: +the tracker-web `verify` script (typecheck + lint + test + e2e) green — run exactly what +its package.json defines (e.g. pnpm --filter run verify, or the documented +equivalent). Do not weaken its lint/e2e config. + +-------------------------------------------------------------------------------- +SLICE 5 — Docs + roadmap + Phase-3 exit criteria (depends on: S1-S4 outcomes) +-------------------------------------------------------------------------------- +- Update ../learning_ai_devops_tools/agent-queue/docs/GIGAFACTORY_ROADMAP.md §14 Phase-3 + checkboxes for every box you actually completed, with a one-line note + the flag names + (FLEET_PREEMPTION/FLEET_BUDGETS) and which are default-OFF. Tick the Phase-3 Exit-criteria + line ONLY if its conditions are genuinely met; otherwise note the exact remaining %. + (This is a docs edit in the OTHER repo — make it as a separate small commit/PR in + learning_ai_devops_tools, OR include the roadmap delta as a patch file under + docs/ in THIS branch and note it for the operator — do NOT entangle the two repos' + git history. Prefer the patch-file note if a clean cross-repo PR isn't trivial.) +- Update dashboards/tracker-web/README + a short docs/FLEET_CONTROL_PLANE.md (how to use + the new UI, the flags, the endpoints consumed). +- Finalize docs/gigafactory-phase3-progress.md with the end-state of every slice. + +FINAL OUTPUT — print ONE consolidated report in EXACTLY this format: +## Implementation Report — Phase 3 (overnight) +### Branch & PR +### Per-slice results +| slice | status (DONE/WIP/FAILED) | commit | verify gate | notes | +### What was implemented (per slice: key files, flags, endpoints, UI surfaces) +### Tests added (counts per area + the final verify-gate output per slice) +### Deviations / assumptions (weight defaults, budget accounting source, polling vs SSE, any degraded UI paths) +### Phase 3 status (which §14 boxes now complete; exit criteria met Y/N; remaining %) +### Anything that needs a human decision (esp. risky majors, cross-repo roadmap tick) +### Suggested next phase (Phase 4 — message bus + autoscaling + capability marketplace)