# Durable Event Bus & Worker Runtime — Detailed Roadmap (2026-04-17)

> **Supersedes:** `platform_DURABLE_EVENT_BUS_AND_WORKER_RUNTIME_ROADMAP.md` (scaffold).
> **Status:** Planned — unblocks Agent Runtime, Approval Queue, Knowledge ingestion.
> **Primary surfaces:** `packages/events/`, `packages/queue/`, `packages/event-store/`, `services/platform-service/`.
> **Estimated effort:** 3–4 weeks (1 engineer) / 2 weeks (2 engineers).
> **Priority:** **P0** — foundation for Categories C3/C4 and for reliable cross-service side effects.

---

## 1. Current State (as of 2026-04-17)

| Surface                 | State                                                                                                                                | Evidence                                   |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------ |
| `@bytelyst/events`      | `EventBus` (memory) + `DurableEventBus` interface + typed schemas (AgentRun, AgentTask, AgentDispatch, AgentCheckpoint, Artifact, …) | `packages/events/src/index.ts`             |
| `@bytelyst/event-store` | Cosmos-backed append-only event store (exists, underused)                                                                            | `packages/event-store/src/`                |
| `@bytelyst/queue`       | `QueueWorker` + `MemoryQueueStore` + `FileQueueStore`. No Redis, no cross-process delivery                                           | `packages/queue/src/index.ts`              |
| Platform subscribers    | `delivery`, `notifications`, `audit`, `runs/tracker.ts` emit on the in-proc memory bus; cross-service = ad-hoc HTTP                  | `services/platform-service/src/modules/*/` |
| Worker processes        | None standardized. Scheduled jobs embed inside fastify via setInterval / per-module timers                                           | n/a                                        |

**Net:** we have durable-queue and durable-event-store primitives but no **standardized durable bus or worker runtime** consuming them. Production flows still ride the in-memory bus.

---

## 2. Success Criteria (V1)

1. One event emitted in `platform-service` is reliably consumed by `cowork-service`, `extraction-service`, or `mcp-server` within 1s P95, surviving restart on either side.
2. Every durable subscriber records ack / retry / DLQ counts; operators can query/replay from Cosmos.
3. A single `workers/` bootstrap pattern is documented and used by ≥3 side-effect consumers.
4. No new module added after Phase 3 relies on the in-memory bus for production-critical flows.
5. `@bytelyst/events` memory adapter remains the default in **test environments only** (`NODE_ENV=test`).

---

## 3. Non-Goals (V1)

- Kafka-scale throughput, exactly-once delivery, cross-region replication
- Full Temporal semantics (that's the Agent Runtime roadmap)
- Replacing the HTTP REST layer between services

---

## 4. Dependencies

- None (this is the foundation). Enables: Agent Runtime Orchestration, Human Review Approval Queue, Knowledge RAG ingestion.

---

## 5. Phase Plan

### Phase 1 — Pluggable Event Bus Backends (week 1)

- [ ] **1.1** Formalize `DurableEventAdapter` contract in `@bytelyst/events` (publish, subscribe, ack, nack, dlq, replay)
- [ ] **1.2** Keep `MemoryAdapter` as the default for tests
- [ ] **1.3** Implement `CosmosEventStoreAdapter` on top of existing `@bytelyst/event-store` (append + tail via change-feed)
  - [ ] Partition key: `productId` + day-bucket
  - [ ] Consumer groups via lease-collection (reuse `packages/cosmos/src/lease-manager` if present; else add lightweight doc-based lease)
- [ ] **1.4** Add `correlationId` + `causationId` to every `PlatformEventSchemas` entry (schema-breaking; ship behind zod `.catchall()` fallback for unknown fields)
- [ ] **1.5** Add `packages/events/src/durable.test.ts` — adapter conformance suite (publish/subscribe/ack/retry/DLQ)

**Exit:** `pnpm --filter @bytelyst/events test` covers the Cosmos adapter using the Cosmos emulator.

---

### Phase 2 — Standardized Worker Runtime (week 2)

- [ ] **2.1** New surface: `services/platform-service/src/workers/` with a single `createWorker(opts)` bootstrap
  - [ ] Concurrency control (default: `AGENT_WORKER_CONCURRENCY=4`)
  - [ ] Lease heartbeat (default 30s, configurable per handler)
  - [ ] Graceful shutdown on SIGINT/SIGTERM
  - [ ] Pino child logger with `workerName`, `eventId`, `correlationId` bindings
- [ ] **2.2** Expose `/ops/workers/*` admin routes: list workers, inspect in-flight leases, force-replay DLQ item, pause/resume
- [ ] **2.3** Health: each worker reports into `GET /api/health/dependencies` under a `workers` key
- [ ] **2.4** Operator command: `pnpm --filter @lysnrai/platform-service worker:run <name>` for running a single worker out-of-process (k8s-ready)
- [ ] **2.5** Dead-letter inspection: Cosmos container `events-dlq` with 30-day TTL; admin UI can re-enqueue

**Exit:** `reviews` module's notification side-effects migrated to durable worker. Kill the Fastify process mid-emit; event still delivered after restart.

---

### Phase 3 — Migrate Production Side Effects (week 3)

- [ ] **3.1** Audit in-memory `EventBus.emit()` call sites (expect ≤20) and categorize:
  - Critical (must durably deliver): auth-events, run-state-changes, billing-events, approval-events
  - Best-effort (OK to lose): dev-time dashboards, local UI refresh hints
- [ ] **3.2** Migrate critical callers to `DurableEventBus` one module at a time:
  - [ ] `auth` side-effects (login audit, MFA trigger)
  - [ ] `runs/tracker.ts` state transitions
  - [ ] `billing-checkout` / `dunning` / `subscriptions` webhooks
  - [ ] `notifications` fan-out
  - [ ] `delivery` outbox
- [ ] **3.3** Replace per-module `setInterval` schedulers with registered durable workers (e.g., `retention/`, `exports/`, `predictive-analytics/`)
- [ ] **3.4** Telemetry: emit `event_published_total`, `event_consumed_total`, `event_dlq_total`, `worker_lease_expired_total` via `@bytelyst/backend-telemetry`
- [ ] **3.5** Grafana dashboard: event-lag P95 per subscriber, DLQ size, worker concurrency saturation

**Exit:** post-migration, no production code path emits to `new EventBus()` in `NODE_ENV=production`. Enforced by an ESLint rule in `eslint.config.js` (`no-restricted-imports` scoped to `@bytelyst/events/memory`).

---

### Phase 4 — Cross-Service Subscriptions (week 3–4)

- [ ] **4.1** `cowork-service` subscribes to `agent.run.*` events and mirrors state into its own tracker (currently polls `platform-service/runs`)
- [ ] **4.2** `extraction-service` subscribes to `knowledge.ingest.requested` (see RAG roadmap) instead of exposing REST
- [ ] **4.3** `mcp-server` emits `a2a.pipeline.step.*` events that become the audit trail source of truth
- [ ] **4.4** Compose: add a Redis container to `docker-compose.yml` as an **optional** fast-path adapter (later phase)

**Exit:** `pnpm prototype:self-test` exercises one end-to-end cross-service event.

---

## 6. Tech Stack Decision

| Option                         | V1 default?              | Rationale                                                                |
| ------------------------------ | ------------------------ | ------------------------------------------------------------------------ |
| **Cosmos change-feed adapter** | ✅ Yes                   | Reuses existing emulator, persistence story, Azure runtime; no new infra |
| Redis Streams                  | Later (Phase 4 optional) | Lower latency; add after Cosmos proves semantics                         |
| NATS JetStream                 | Deferred                 | Best long-term; reassess at Phase 5+                                     |
| Kafka                          | Never (overkill)         | —                                                                        |

**Default decision:** Cosmos-backed adapter first → add Redis optional adapter at Phase 4 → revisit NATS once event volume > 10 events/sec sustained.

---

## 7. Risks & Mitigations

| Risk                               | Mitigation                                                                                                 |
| ---------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| Cosmos change-feed lag under burst | Per-consumer lease + configurable `maxItemCount`; alert on lag > 5s                                        |
| Schema drift breaks replay         | Every event carries `schemaVersion`; adapters refuse unknown major versions; replay path calls migrator    |
| Workers multiply process count     | `services/platform-service` runs workers in-process by default; only split out when concurrency demands it |
| In-memory bus regressions          | Lint rule + CI grep blocks `new EventBus()` outside `*/test/*` paths                                       |

---

## 8. Acceptance Checklist

- [ ] Phase 1 complete — adapter conformance suite passes on memory + Cosmos
- [ ] Phase 2 complete — `/ops/workers` + DLQ admin routes shipped
- [ ] Phase 3 complete — ≥5 modules migrated; ESLint guard active
- [ ] Phase 4 complete — one cross-service flow demonstrated end-to-end
- [ ] Grafana dashboard `platform-events` live
- [ ] AGENTS.md + `docs/ECOSYSTEM_ARCHITECTURE.md` updated
- [ ] Roadmap moved to `docs/roadmaps/completed/`
