# Knowledge & RAG Service — Detailed Roadmap (2026-04-17)

> **Supersedes:** `platform_KNOWLEDGE_RAG_SERVICE_ROADMAP.md` (scaffold).
> **Status:** Planned — highest user-visible leverage once Event Bus lands.
> **Primary surfaces:** `services/platform-service/src/modules/knowledge/`, `services/extraction-service/`, `packages/palace/`, new `packages/rag-client/`.
> **Estimated effort:** 4–6 weeks (1 engineer) / 3 weeks (2 engineers).
> **Priority:** **P1** — converges cowork institutional knowledge + MindLyst MemPalace + NoteLett palace + JarvisJr context.

---

## 1. Current State (as of 2026-04-17)

| Surface                              | State                                                                                                  | Evidence                                                                       |
| ------------------------------------ | ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------ |
| `platform-service/modules/knowledge` | Basic CRUD over knowledge docs — no chunking, no embeddings, no retrieval API                          | `services/platform-service/src/modules/knowledge/{routes,repository,types}.ts` |
| `@bytelyst/palace`                   | MindLyst-style memory package with embeddings scaffolding (19 files)                                   | `packages/palace/`                                                             |
| `extraction-service`                 | PDF/xlsx/docx text extraction via Python sidecar — no embedding step                                   | `services/extraction-service/`                                                 |
| `packages/llm-router`                | Deterministic router (provider/model selection); no embeddings client yet                              | `packages/llm-router/`                                                         |
| Cowork `InstitutionalKnowledge`      | Rust-side per-project `.md`/`.txt` loader (≤100KB scan)                                                | `crates/cowork-orchestrator/src/admin.rs`                                      |
| Products using RAG                   | MindLyst (custom Palace), NoteLett (custom Palace), LysnrAI (none), JarvisJr (none), ChronoMind (none) | grep of product backends                                                       |

**Net:** every product wants RAG; nothing shared; extraction exists but stops at "text + metadata". No citation surface, no ACL.

---

## 2. Success Criteria (V1)

1. One `POST /api/knowledge/documents` call ingests a PDF → extraction → chunking → embedding → indexed and queryable within 60s P95.
2. `POST /api/knowledge/search` returns chunks with citations (`documentId`, `pageNumber`, `quoteStart`, `quoteEnd`, `score`) scoped by `productId` + `workspaceId`.
3. MindLyst + NoteLett both swap their custom palace indexers for `@bytelyst/rag-client` without UX regression.
4. Cowork's `InstitutionalKnowledge` can optionally call `@bytelyst/rag-client` instead of local file scan when the flag `knowledge_rag_enabled` is on.
5. Tenant isolation verified by integration test: product A query cannot retrieve product B docs even when bucket names overlap.

---

## 3. Non-Goals (V1)

- Multi-hop retrieval / graph-RAG
- Hybrid reranking with a dedicated cross-encoder model
- User-facing document editors / rich collaboration
- Non-text modalities (image/audio embeddings) — deferred to V2

---

## 4. Dependencies

- **Hard:** Durable Event Bus (C1) for ingestion pipeline reliability
- **Soft:** Org & Workspace RBAC (existing `@bytelyst/org-client` is sufficient for V1; deeper ACL comes with SCIM roadmap)

---

## 5. Data Model

New Cosmos containers (partition key in **bold**):

| Container               | PK             | Purpose                                                                        |
| ----------------------- | -------------- | ------------------------------------------------------------------------------ |
| `knowledge-documents`   | **productId**  | Doc metadata (title, source URI, mime, sha256, workspaceId, createdBy, status) |
| `knowledge-chunks`      | **documentId** | Text + embedding ref + page/line spans + tokens                                |
| `knowledge-embeddings`  | **documentId** | Vector payload; separate from chunks to support vector-store migration         |
| `knowledge-ingest-jobs` | **productId**  | Job lifecycle (queued/extracting/embedding/indexed/failed)                     |

Every doc carries `productId` (house rule) + `workspaceId` + `sourceHash` for dedup.

---

## 6. Phase Plan

### Phase 1 — Document Ingestion Pipeline (week 1–2)

- [ ] **1.1** Extend `knowledge/types.ts` with `KnowledgeDocumentDoc`, `KnowledgeChunkDoc`, `KnowledgeIngestJobDoc`
- [ ] **1.2** `POST /api/knowledge/documents` creates document row → emits `knowledge.ingest.requested` event
- [ ] **1.3** Worker (via Durable Event Bus): pulls event → fetches source (blob URL or inline bytes) → calls `extraction-service`
- [ ] **1.4** Extraction worker returns normalized Markdown + per-chunk anchors (page, line range)
- [ ] **1.5** Chunker (new `packages/rag-chunker`):
  - sliding-window 800 tokens, 120 overlap (configurable)
  - preserves heading hierarchy for "section" metadata
  - 25+ unit tests: markdown tables, code fences, multi-page PDFs, RTL text
- [ ] **1.6** Dedup: if `sourceHash` exists for `productId`+`workspaceId`, skip re-embedding; update references only

**Exit:** Upload `AGENTS.md` → within 60s, 10–30 chunks stored with anchors.

---

### Phase 2 — Embedding + Retrieval (week 2–3)

- [ ] **2.1** New `@bytelyst/llm-router` subpath: `llmRouter.embedBatch({ texts, model })` — default `text-embedding-3-small`
- [ ] **2.2** Embedding worker consumes `knowledge.chunks.ready` events
  - Batching: up to 256 chunks / request
  - Retry + DLQ via Durable Event Bus
  - Cost accounting wired to `ai-budgets` module
- [ ] **2.3** Vector store — **primary** pgvector via new `packages/vector-store/` adapter
  - `docker-compose.yml` adds Postgres 16 + pgvector extension
  - AKV binding: `AZURE_POSTGRES_CONNECTION_STRING`
  - Fallback: Cosmos native vector search (when Azure emulator supports it) as secondary adapter
- [ ] **2.4** `POST /api/knowledge/search { query, topK, filters: { workspaceId, tags, dateRange } }`:
  - Embeds query (cache by sha256 of normalized query, 5-min TTL)
  - pgvector k-NN with metadata filter pushdown
  - Returns `{ chunkId, documentId, score, text, citation: { page, lineStart, lineEnd } }`
- [ ] **2.5** `POST /api/knowledge/assemble { query, maxTokens, productId }` — composes a context block under `maxTokens` with inline citations (`[^1]` syntax)

**Exit:** integration test: upload a 50-page PDF → query → answer includes citations pointing to real page numbers.

---

### Phase 3 — Client SDK + Access Control (week 4)

- [ ] **3.1** New package `@bytelyst/rag-client` (mirrors existing client package patterns — see `subscription-client`, `billing-client`)
  - `createRagClient({ baseUrl, token, productId })`
  - Methods: `uploadDocument`, `search`, `assemble`, `getDocument`, `deleteDocument`, `listDocuments`
  - TypeScript types generated from platform zod schemas
- [ ] **3.2** RBAC: every request checks `workspaceId` membership via `@bytelyst/org-client`
- [ ] **3.3** Tenant isolation test suite (`knowledge-acl.test.ts`) — 10+ cases including cross-product attack scenarios
- [ ] **3.4** Field-encrypt sensitive docs at rest (via `@bytelyst/field-encrypt`) — encrypt `text` column; chunk vectors remain plain (they're derived and low-info)

**Exit:** NoteLett integrates `@bytelyst/rag-client` in a branch; local E2E still passes.

---

### Phase 4 — Connectors + Operator UX (week 5–6)

- [ ] **4.1** Connector: file upload via blob (reuse `@bytelyst/blob`)
- [ ] **4.2** Connector: URL crawl (single-page) using existing Python sidecar
- [ ] **4.3** Connector: Gitea repo ingest (`.md` + `AGENTS.md` across a list of repos) — feeds cowork's knowledge loop
- [ ] **4.4** Admin UI in `dashboards/admin-web/app/ops/knowledge/` — list, reindex, delete, see ingestion DLQ
- [ ] **4.5** Metrics: `knowledge_ingest_latency_seconds`, `knowledge_search_latency_ms`, `knowledge_embedding_tokens_total`, surfaced in Grafana

**Exit:** 20 real `AGENTS.md` files across the workspace ingested; cowork query returns citations.

---

## 7. Tech Stack Decision

| Vector Backend              | V1?               | Rationale                                                   |
| --------------------------- | ----------------- | ----------------------------------------------------------- |
| **pgvector on Postgres 16** | ✅ Yes            | Unified metadata+vector, SQL filtering, ACID for ACL, cheap |
| Qdrant                      | Deferred          | Better pure-vector perf; adds an extra system               |
| Cosmos vector search        | Secondary adapter | Useful in Azure-only envs; still evolving                   |
| Azure AI Search             | No                | Vendor lock-in; less control                                |

**Default:** pgvector primary, Cosmos secondary. Adapter interface allows swap without touching callers.

---

## 8. Risks & Mitigations

| Risk                                | Mitigation                                                                        |
| ----------------------------------- | --------------------------------------------------------------------------------- |
| Retrieval leaks cross-tenant data   | ACL filter enforced in adapter layer, not app layer; adversarial test suite       |
| Embedding cost balloons             | `ai-budgets` enforcement at dispatch + dedup by sourceHash + query cache          |
| Stale embeddings after model change | `embeddingModel` recorded per chunk; reindex job triggered by model registry bump |
| Product teams bypass the service    | Deprecation path: mark palace methods `@deprecated`, CI warning, 90-day sunset    |

---

## 9. Acceptance Checklist

- [ ] Phase 1 — ingestion pipeline live; DLQ visible in admin
- [ ] Phase 2 — pgvector in compose; search returns citations
- [ ] Phase 3 — `@bytelyst/rag-client` published; NoteLett migration branch green
- [ ] Phase 4 — admin UI + Gitea connector + Grafana dashboard
- [ ] Cowork `InstitutionalKnowledge` opt-in migration path documented
- [ ] `docs/ECOSYSTEM_ARCHITECTURE.md` updated with the RAG substrate
- [ ] Roadmap moved to `docs/roadmaps/completed/`
