# Agent Runtime & Orchestration Roadmap

> **Purpose:** Build a durable execution layer for agent runs, step transitions,
> cancellations, retries, resumability, and operator-visible history.
>
> **Primary Surfaces:** `services/platform-service/`, `services/mcp-server/`,
> `packages/queue/`
>
> **Status:** Planned
>
> **Estimated Effort:** 3-5 weeks

---

## Why This Is Missing

The repo now has durable queue primitives, but agent execution is still not a
first-class platform service. A2A pipelines in `services/mcp-server/src/modules/a2a/`
are composed code paths rather than durable runs with persistent step state.

That is enough for prototypes. It is not enough for:

- long-running multi-step agents
- retries after process restarts
- human escalation in the middle of a run
- cancellation and pause/resume
- auditable run history

---

## Recommended Stack

### Best Long-Term Industry Standard

- **Temporal** for workflow orchestration
- **PostgreSQL** for run metadata and operator queries
- **Redis** for short-lived coordination and cache

### Best Repo-Fit Option

- `@bytelyst/queue` for durable job dispatch
- `platform-service` run records in Cosmos or datastore abstraction
- `mcp-server` as orchestration client and tool executor

### Recommendation

Start with the repo-fit option to get durable runs quickly, but design the run model
so a later move to Temporal is possible without rewriting every agent contract.

### Default Decision For This Repo

- Build v1 in `platform-service`
- Use `@bytelyst/queue` as the execution primitive
- Use Redis later for leases and distributed coordination
- Keep `mcp-server` as a client/executor, not the run system of record

---

## Dependencies

- Durable Event Bus & Worker Runtime
- Human Review & Approval Queue
- Agent Registry & Prompt Versioning

---

## V1 Non-Goals

- Full Temporal adoption
- Visual workflow builders
- Arbitrary user-authored workflow DSLs
- Multi-region active-active workflow execution

---

## Phase 1 - Canonical Run Model

- [ ] Create `services/platform-service/src/modules/agent-runs/`
- [ ] Define `AgentRunDoc`, `AgentRunStepDoc`, `AgentRunEventDoc`
- [ ] Support states: `queued`, `running`, `waiting_for_input`, `paused`, `succeeded`, `failed`, `cancelled`
- [ ] Add `parentRunId`, `workflowId`, `agentId`, `agentVersion`, `triggerSource`
- [ ] Persist step inputs, outputs, timings, error summaries, and correlation IDs
- [ ] Add APIs:
  - [ ] `POST /agent-runs`
  - [ ] `GET /agent-runs/:id`
  - [ ] `GET /agent-runs/:id/events`
  - [ ] `POST /agent-runs/:id/cancel`
  - [ ] `POST /agent-runs/:id/pause`
  - [ ] `POST /agent-runs/:id/resume`

**Acceptance Criteria**

- Every agent run has durable metadata and step history
- A run can be fetched after service restart
- Cancellation and pause are explicit states, not implicit errors

---

## Phase 2 - Queue-Backed Execution

- [ ] Add `agent.run.execute` queue type on top of `@bytelyst/queue`
- [ ] Add per-step retries with backoff
- [ ] Add lease heartbeat for long-running steps
- [ ] Add idempotency keys for replays
- [ ] Add dead-letter handling and operator inspection
- [ ] Record structured run events for step started/completed/failed/retried

**Acceptance Criteria**

- In-flight runs survive worker restart
- Retried steps do not duplicate side effects when idempotency is configured
- Dead-lettered runs are queryable and replayable

---

## Phase 3 - A2A Integration

- [ ] Replace direct A2A pipeline progression in `mcp-server` with run orchestration APIs
- [ ] Make every pipeline step emit durable run events
- [ ] Support handoff to human review queue
- [ ] Support child runs for delegated agent tasks
- [ ] Add run-level audit links to diagnostics, telemetry, and support systems

**Acceptance Criteria**

- `mcp-server` no longer owns the durable run state itself
- A2A pipelines are observable step-by-step
- Human review can pause and later resume a run

---

## Phase 4 - Operator Experience

- [ ] Admin UI for runs, filters, and replay
- [ ] Timeline view per run
- [ ] Step diff view for prompt/tool transitions
- [ ] Cancel/retry/replay controls
- [ ] SLOs: success rate, mean run duration, retry rate, dead-letter count

---

## Tech Stack Options

| Option                       | Pros                                              | Cons                              | Fit                     |
| ---------------------------- | ------------------------------------------------- | --------------------------------- | ----------------------- |
| Temporal                     | Best workflow semantics, retries, signals, timers | New infra, steeper learning curve | Best long-term          |
| BullMQ + Redis + run DB      | Simple, common in Node                            | Workflow semantics are custom     | Strong practical option |
| `@bytelyst/queue` + run docs | Lowest disruption to repo                         | More framework logic to build     | Best immediate path     |

---

## Risks

- Custom orchestration can become a weak in-house Temporal clone if not scoped tightly
- If step contracts are not versioned, replay becomes unsafe
- If all state remains in logs, operator tooling will never be reliable

---

## Recommendation

Implement the v1 run system inside `platform-service` using `@bytelyst/queue`, but
borrow Temporal-style concepts from day one:

- workflow ID
- run ID
- signals
- child runs
- durable timers
- explicit waiting states
