65 lines
3.1 KiB
Markdown
65 lines
3.1 KiB
Markdown
# Backend LLM Router Decision
|
|
|
|
Status: active decision
|
|
Date: May 5, 2026
|
|
Decision: keep `@bytelyst/llm` as the NoteLett release-1 backend LLM abstraction; defer `@bytelyst/llm-router` adoption.
|
|
|
|
## Current NoteLett Usage
|
|
|
|
NoteLett backend uses `@bytelyst/llm` through `backend/src/lib/llm.ts` for:
|
|
|
|
- copilot transforms and title suggestions
|
|
- Smart Action prompt runs, including text and image inputs
|
|
- prompt schedules and webhooks
|
|
- MCP tag suggestions and note operations
|
|
- Palace memory extraction
|
|
- embeddings through `provider.embed()` for duplicate/related note and Palace search behavior
|
|
- mock provider support for local development and tests
|
|
- OpenAI/Azure provider selection through existing `LLM_PROVIDER`, `LLM_DEFAULT_MODEL`, `LLM_VISION_MODEL`, and `LLM_EMBEDDING_MODEL` environment settings
|
|
|
|
## Router Evaluation
|
|
|
|
`@bytelyst/llm-router` provides useful provider/model governance:
|
|
|
|
- deterministic prompt classification
|
|
- provider/model selection
|
|
- round-robin routing
|
|
- health tracking
|
|
- fallback across configured providers
|
|
- telemetry hooks
|
|
|
|
However, the current router API is centered on OpenAI-compatible chat completions with string-only messages and API-key filtered providers. It does not yet replace the NoteLett release-1 needs listed above:
|
|
|
|
- no embedding API equivalent to `provider.embed()`
|
|
- no first-class mock provider for test/local parity
|
|
- no Azure provider compatibility path matching current production env
|
|
- no multipart image-message compatibility with `buildVisionMessage()` from `@bytelyst/llm`
|
|
- no streaming or provider interface parity with the existing `LLMProvider` contract
|
|
- default provider set targets free-tier routing, while NoteLett release-1 config is already expressed through `@bytelyst/llm`
|
|
|
|
## Decision
|
|
|
|
Do not adopt `@bytelyst/llm-router` in the production-readiness handoff.
|
|
|
|
Continue using `@bytelyst/llm` for release 1, while keeping the router as a future governance layer once it can sit behind or implement the same `LLMProvider` capabilities NoteLett already depends on.
|
|
|
|
## Required Adoption Criteria
|
|
|
|
Adopt the router later when it supports or cleanly bridges:
|
|
|
|
- `chatCompletion()` parity with the current `LLMProvider` response shape
|
|
- `embed()` for NoteLett embeddings and Palace semantic search
|
|
- mock provider behavior for deterministic tests
|
|
- Azure/OpenAI production settings without changing release env names
|
|
- multipart text+image prompts or an adapter for `buildVisionMessage()`
|
|
- telemetry mapping into NoteLett/platform telemetry
|
|
- fallback behavior that preserves current timeout/retry semantics
|
|
- tests for copilot, Smart Actions, prompt schedules, MCP suggestions, Palace extraction, and embeddings
|
|
|
|
## Current Guardrails
|
|
|
|
- Keep `backend/src/lib/llm.ts` as the single backend LLM entrypoint.
|
|
- Do not call both `@bytelyst/llm` and `@bytelyst/llm-router` from product routes.
|
|
- Treat `@bytelyst/llm-router` as a future provider-governance improvement after release-1 LLM and embedding behavior is stable.
|
|
- Continue tracking current `@bytelyst/llm`/package build drift through the production-readiness baseline and backend hardening phases.
|