# Agent Runtime & Orchestration — Detailed Roadmap (2026-04-17)

> **Supersedes:** `platform_AGENT_RUNTIME_ORCHESTRATION_ROADMAP.md` (scaffold).
> **Status:** Planned — enables platform-hosted multi-step agent runs reusable across products.
> **Primary surfaces:** `services/platform-service/src/modules/{agent-runtime,agents,runs}/`, `services/mcp-server/src/modules/a2a/`, `services/cowork-service/`.
> **Estimated effort:** 3–5 weeks (1 engineer) / 2–3 weeks (2 engineers).
> **Priority:** **P1** — closes the gap between cowork (desktop agent) and platform-hosted agents invoked by product backends or admins.

---

## 1. Current State (as of 2026-04-17)

| Surface                                  | State                                                                                              | Evidence                                             |
| ---------------------------------------- | -------------------------------------------------------------------------------------------------- | ---------------------------------------------------- |
| `platform-service/modules/agent-runtime` | Routes only; no persistent state model                                                             | `routes.ts`, `routes.test.ts`                        |
| `platform-service/modules/agents`        | Fuller: `repository`, `routes`, `executor`, `tool-registry`, `types` — but in-memory execution     | 8 files in `agents/`                                 |
| `platform-service/modules/runs`          | Run tracker (`tracker.ts`), repository, routes — already models run lifecycle but not queue-backed | 7 files                                              |
| `platform-service/modules/ai-budgets`    | Exists — enforces spend per product                                                                | `ai-budgets/`                                        |
| `platform-service/modules/agent-evals`   | Exists — eval harness skeleton                                                                     | `agent-evals/`                                       |
| `mcp-server/modules/a2a`                 | A2A pipelines as composed code paths; no durable step state                                        | `services/mcp-server/src/modules/a2a/`               |
| `cowork-service/modules/agent-runtime`   | IPC bridge to cowork-orchestrator `submit_task`                                                    | `services/cowork-service/src/modules/agent-runtime/` |
| `@bytelyst/queue`                        | `QueueWorker` + memory/file stores. No cross-process yet                                           | `packages/queue/`                                    |

**Net:** the run data model is partially there; what's missing is the **durable queue-backed execution** plus **unified orchestration** so cowork, product-backend agents, and admin-initiated runs share one run API.

---

## 2. Success Criteria (V1)

1. An `POST /api/agent-runs` request creates a queued run; executor worker picks it up and progresses through steps with full persistence.
2. Kill the executor mid-step → run resumes from the last completed step on restart without duplicated side effects (idempotency on step outputs).
3. A run can enter `waiting_for_review` (see Approval Queue roadmap) and resume when a reviewer decides.
4. Cowork sessions created via `ipc-bridge` register as platform agent runs and appear in `/ops/runs`.
5. Each run carries `aiBudgetId` and writes per-step token usage; over-budget runs pause, not crash.
6. 300+ concurrent runs sustained with per-run latency < 2× single-run baseline.

---

## 3. Non-Goals (V1)

- Full Temporal adoption (contract is Temporal-compatible for future migration; no Temporal infra yet)
- User-authored workflow DSL
- Multi-region run migration
- Streaming step outputs to clients (add post-V1 via SSE + existing `@bytelyst/fastify-sse`)

---

## 4. Dependencies

- **Hard:** Durable Event Bus & Worker Runtime (C1)
- **Hard:** `@bytelyst/queue` Redis adapter (Phase 2 of C1)
- **Soft:** Human Review Approval Queue (C4) for `waiting_for_review` semantics
- **Soft:** Agent Registry & Prompt Versioning (existing scaffolded doc) — for deterministic `agentVersion` binding

---

## 5. Canonical Run Model

New/expanded Cosmos containers (PK in **bold**):

| Container          | PK            | Purpose                                                                                                    |
| ------------------ | ------------- | ---------------------------------------------------------------------------------------------------------- |
| `agent-runs`       | **productId** | `AgentRunDoc`: state, workflowId, agentId, agentVersion, triggerSource, parentRunId, aiBudgetId, createdBy |
| `agent-run-steps`  | **runId**     | Step inputs/outputs/timings/errors; idempotency key; `state`                                               |
| `agent-run-events` | **runId**     | Durable event log (mirror into event store) for replay/operator view                                       |
| `agent-run-leases` | **runId**     | Worker lease with expiry — enables safe failover                                                           |

Run states: `queued → running → waiting_for_input | waiting_for_review | paused → running → (succeeded | failed | cancelled)`.

Step states: `pending → running → (succeeded | failed | retrying | skipped)`.

Every doc includes `productId` + `workspaceId` + `correlationId`.

---

## 6. Phase Plan

### Phase 1 — Durable Run Model (week 1)

- [ ] **1.1** Extend `agent-runtime/types.ts` with `AgentRunDoc`, `AgentRunStepDoc`, `AgentRunEventDoc`, `AgentRunLeaseDoc` (zod schemas)
- [ ] **1.2** Repository methods: `createRun`, `getRun`, `listRuns`, `appendStep`, `updateStepState`, `acquireLease`, `renewLease`, `releaseLease`
- [ ] **1.3** Migrate existing `runs/` module → rename to legacy alias; new agent-runs module becomes source of truth
- [ ] **1.4** REST surface:
  - [ ] `POST /api/agent-runs` → creates run, publishes `agent.run.execute` event
  - [ ] `GET /api/agent-runs/:id`
  - [ ] `GET /api/agent-runs/:id/events` (paginated + optional SSE)
  - [ ] `POST /api/agent-runs/:id/cancel` → sets `cancelRequested` flag read by executor
  - [ ] `POST /api/agent-runs/:id/pause` / `/resume`
  - [ ] `POST /api/agent-runs/:id/signal` — named signal (for approval, input, tool-result)
- [ ] **1.5** 25+ route tests using `@bytelyst/testing` fastify inject helpers

**Exit:** run lifecycle transitions persist across fastify restart; all state queryable.

---

### Phase 2 — Queue-Backed Executor (week 2)

- [ ] **2.1** Add `agent.run.execute` queue type on top of Durable Event Bus + `@bytelyst/queue`
- [ ] **2.2** `AgentRunExecutor` class in `modules/agent-runtime/executor.ts`:
  - Acquires lease (30s heartbeat); holds until step completes
  - For each step: resolves agent spec → calls tool (`agents/tool-registry.ts`) → records output with idempotency key
  - On failure: exponential backoff, max retries per step from `agentVersion.retryPolicy`
  - DLQ on exhausted retries; run → `failed`
- [ ] **2.3** Idempotency:
  - Caller can pass `idempotencyKey` on `POST /agent-runs`; repeat within 24h returns existing run
  - Each step output stored by `{runId, stepIndex, attempt}`; external tool calls opt-in to idempotency via handler contract
- [ ] **2.4** Budget integration: pre-step check against `ai-budgets`; over-budget → run → `waiting_for_input` with reason `budget_exceeded`
- [ ] **2.5** Cancellation: `cancelRequested` polled between steps; in-flight step finishes cleanly, run → `cancelled`

**Exit:** chaos test — kill executor process during a 5-step run. Run resumes from last checkpoint. Idempotent tool not double-invoked.

---

### Phase 3 — A2A + Cowork Integration (week 3)

- [ ] **3.1** `mcp-server/modules/a2a` refactor: pipeline steps become agent-run steps. Pipeline definition → stored as workflow template, not code path.
- [ ] **3.2** `cowork-service/modules/agent-runtime` updates: when cowork submits a task via IPC, also register as platform agent run (dual-write with `triggerSource=cowork`)
- [ ] **3.3** Cross-system correlation: cowork's `taskId` stored as external ref on agent-run; admin `/ops/runs` surfaces both sides
- [ ] **3.4** Agent Registry integration (depends on separately scaffolded roadmap): `agentId + agentVersion` required on run creation; registry prompt snapshot frozen at run start

**Exit:** every cowork session visible under `/ops/runs`; every A2A pipeline surfaces step-by-step.

---

### Phase 4 — Operator Experience (week 4)

- [ ] **4.1** Admin UI `dashboards/admin-web/app/ops/runs/`:
  - List with filters (product, workspace, agent, state, age, reviewer)
  - Timeline view per run
  - Step diff view (prompt/tool-call/output)
  - Cancel / retry / replay controls
- [ ] **4.2** SLO dashboard: success rate, mean duration, retry rate, DLQ count per agent
- [ ] **4.3** `@bytelyst/platform-client` additions: `client.agentRuns.create/get/cancel/signal` for product backends
- [ ] **4.4** Migration guide: product backends that roll their own agent loops → 1-call swap

**Exit:** one product backend (e.g., JarvisJr) dispatches an agent via platform runs instead of in-proc call.

---

### Phase 5 — Scale & Hardening (week 5)

- [ ] **5.1** Horizontal scale: 2+ executor replicas via lease-based safety
- [ ] **5.2** Load test: 300 concurrent runs, 5 steps each, measure P50/P95/P99
- [ ] **5.3** Backpressure: when queue depth > N, new runs enter `queued` but `POST` returns immediately; admin can see queue saturation
- [ ] **5.4** Circuit breakers: per-agent failure-rate trips new run acceptance for 60s
- [ ] **5.5** Observability: OpenTelemetry spans around every step; trace ID propagates to external tool calls

**Exit:** load test meets latency budget; operator dashboard usable under load.

---

## 7. Tech Stack Decision

| Option                                                  | V1?                           | Rationale                                         |
| ------------------------------------------------------- | ----------------------------- | ------------------------------------------------- |
| **`@bytelyst/queue` + Redis adapter + Cosmos run docs** | ✅ Yes                        | Reuses Durable Event Bus; no new major infra      |
| Temporal                                                | Deferred                      | Right answer for V2; too much infra for V1        |
| BullMQ direct                                           | Replaced by `@bytelyst/queue` | Same class; we'd just be skipping our abstraction |
| SQS / EventBridge                                       | Skip                          | Cloud-vendor coupling                             |

---

## 8. Risks & Mitigations

| Risk                                    | Mitigation                                                                                                                                      |
| --------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| "Weak in-house Temporal" drift          | Contract borrows Temporal terminology (workflowId, runId, signals, child runs) so future migration is mechanical                                |
| Step idempotency misuse                 | Executor requires handlers declare `idempotent: true`; non-idempotent defaults to single-attempt + failover-safe retry only on transient errors |
| Lease contention at scale               | Leases include random jitter; stuck runs auto-released after 3× heartbeat period                                                                |
| Product teams bypass to run local loops | Deprecation + budget-module enforcement: unregistered runs get 0 budget                                                                         |

---

## 9. Acceptance Checklist

- [ ] Phase 1 — run model persistent; REST complete
- [ ] Phase 2 — executor survives restart; idempotency proven
- [ ] Phase 3 — A2A + cowork both integrated
- [ ] Phase 4 — admin UI shipped; one product backend migrated
- [ ] Phase 5 — 300-concurrent load test passes
- [ ] `platform-client` published with new methods
- [ ] `docs/ECOSYSTEM_ARCHITECTURE.md` + AGENTS.md updated
- [ ] Roadmap moved to `docs/roadmaps/completed/`
