# Knowledge & RAG Service Roadmap

> **Purpose:** Build a shared knowledge platform for ingestion, chunking,
> embeddings, retrieval, citations, and access-controlled context assembly.
>
> **Primary Surfaces:** `services/platform-service/`, `services/extraction-service/`
>
> **Status:** Planned
>
> **Estimated Effort:** 4-6 weeks

---

## Why This Is Missing

The repo already has extraction and some vector-based diagnostics work, but there is
no reusable platform service for general retrieval-augmented generation across
products and agents.

Every serious agent company eventually needs:

- managed document ingestion
- chunking and metadata
- embeddings
- retrieval APIs
- citations and provenance
- workspace-aware access control

---

## Recommended Stack

### Best Long-Term Industry Standard

- **PostgreSQL + pgvector** for integrated metadata + vector search
- **Qdrant** if vector-first performance becomes dominant
- **Blob storage** for source documents

### Cloud-Native Alternative

- **Azure AI Search** for retrieval
- Cosmos or Postgres for metadata

### Recommendation

Use PostgreSQL + pgvector if you want the strongest balance of flexibility,
ownership, and industry-standard retrieval patterns. Azure AI Search is a valid
alternative if deep Azure integration matters more than datastore simplicity.

### Default Decision For This Repo

- Keep ingestion orchestration in `platform-service`
- Reuse `extraction-service` for parsing/chunk preparation where it makes sense
- Do not use Cosmos as the long-term primary RAG engine

---

## Dependencies

- Org, Workspace & RBAC
- AI Budget & Cost Governance

---

## V1 Non-Goals

- General web-scale crawling
- Training or fine-tuning custom foundation models
- Cross-tenant shared retrieval by default

---

## Phase 1 - Knowledge Objects

- [ ] Create modules:
  - [ ] `knowledge-sources`
  - [ ] `knowledge-documents`
  - [ ] `knowledge-chunks`
  - [ ] `knowledge-indexes`
- [ ] Add ingestion states:
  - [ ] uploaded
  - [ ] parsed
  - [ ] chunked
  - [ ] embedded
  - [ ] indexed
  - [ ] failed
- [ ] Add source provenance:
  - [ ] filename
  - [ ] URL
  - [ ] connector type
  - [ ] page or section references

---

## Phase 2 - Retrieval Pipeline

- [ ] Add chunking service with configurable strategies
- [ ] Add embedding generation pipeline
- [ ] Add hybrid search:
  - [ ] lexical
  - [ ] vector
  - [ ] metadata filters
- [ ] Add citation builder and quote bounds
- [ ] Add workspace and org scoping

**Acceptance Criteria**

- Retrieval returns chunks with citations and permission-safe metadata
- Different products can share the same retrieval API

---

## Phase 3 - Connectors

- [ ] File upload
- [ ] Web page ingestion
- [ ] Notes/workspace connector
- [ ] Blob-backed ingestion
- [ ] Optional Slack/Confluence/Google Drive connectors

---

## Tech Stack Options

| Option                   | Pros                                       | Cons                         | Fit                              |
| ------------------------ | ------------------------------------------ | ---------------------------- | -------------------------------- |
| Postgres + pgvector      | Strong standard, unified metadata + vector | Requires new datastore       | Best overall                     |
| Qdrant + metadata DB     | Great vector performance                   | Two systems to operate       | Good at scale                    |
| Azure AI Search          | Strong managed search                      | Vendor-tighter coupling      | Best Azure-managed option        |
| Cosmos vector workaround | Least disruption                           | Not ideal as main RAG engine | Avoid as primary long-term stack |

---

## Risks

- Retrieval without access control causes data leakage between tenants
- Retrieval without citations causes trust and compliance issues
- Embeddings without source lifecycle management become stale quickly
