learning_ai_notes/docs/BACKEND_LLM_ROUTER_DECISION.md

Backend LLM Router Decision

Status: active decision Date: May 5, 2026 Decision: keep @bytelyst/llm as the NoteLett release-1 backend LLM abstraction; defer @bytelyst/llm-router adoption.

Current NoteLett Usage

NoteLett backend uses @bytelyst/llm through backend/src/lib/llm.ts for:

• copilot transforms and title suggestions
• Smart Action prompt runs, including text and image inputs
• prompt schedules and webhooks
• MCP tag suggestions and note operations
• Palace memory extraction
• embeddings through provider.embed() for duplicate/related note and Palace search behavior
• mock provider support for local development and tests
• OpenAI/Azure provider selection through existing LLM_PROVIDER, LLM_DEFAULT_MODEL, LLM_VISION_MODEL, and LLM_EMBEDDING_MODEL environment settings

Router Evaluation

@bytelyst/llm-router provides useful provider/model governance:

• deterministic prompt classification
• provider/model selection
• round-robin routing
• health tracking
• fallback across configured providers
• telemetry hooks

However, the current router API is centered on OpenAI-compatible chat completions with string-only messages and API-key filtered providers. It does not yet replace the NoteLett release-1 needs listed above:

• no embedding API equivalent to provider.embed()
• no first-class mock provider for test/local parity
• no Azure provider compatibility path matching current production env
• no multipart image-message compatibility with buildVisionMessage() from @bytelyst/llm
• no streaming or provider interface parity with the existing LLMProvider contract
• default provider set targets free-tier routing, while NoteLett release-1 config is already expressed through @bytelyst/llm

Decision

Do not adopt @bytelyst/llm-router in the production-readiness handoff.

Continue using @bytelyst/llm for release 1, while keeping the router as a future governance layer once it can sit behind or implement the same LLMProvider capabilities NoteLett already depends on.

Required Adoption Criteria

Adopt the router later when it supports or cleanly bridges:

• chatCompletion() parity with the current LLMProvider response shape
• embed() for NoteLett embeddings and Palace semantic search
• mock provider behavior for deterministic tests
• Azure/OpenAI production settings without changing release env names
• multipart text+image prompts or an adapter for buildVisionMessage()
• telemetry mapping into NoteLett/platform telemetry
• fallback behavior that preserves current timeout/retry semantics
• tests for copilot, Smart Actions, prompt schedules, MCP suggestions, Palace extraction, and embeddings

Current Guardrails

• Keep backend/src/lib/llm.ts as the single backend LLM entrypoint.
• Do not call both @bytelyst/llm and @bytelyst/llm-router from product routes.
• Treat @bytelyst/llm-router as a future provider-governance improvement after release-1 LLM and embedding behavior is stable.
• Continue tracking current @bytelyst/llm/package build drift through the production-readiness baseline and backend hardening phases.