learning_ai_common_plat/packages/llm-router

@bytelyst/llm-router

Pure-code LLM router for free-tier API providers. No LLM-in-the-loop — deterministic routing with automatic fallback, health tracking, and round-robin load distribution.

Features

• 4 free providers out of the box: Groq, OpenRouter, Together AI, Cerebras
• Prompt classification — regex-based detection of code/math/reasoning/creative prompts
• Smart selection — routes to the best model for each prompt category
• Round-robin — distributes load across providers to maximize free-tier usage
• Auto-fallback — retries on 429/5xx with next-best provider
• Health tracking — sliding-window stats (latency, error rate, rate-limit rate)
• Telemetry hook — log every routing decision for analysis
• OpenAI-compatible — same request/response format as OpenAI chat completions
• Zero dependencies — pure TypeScript, uses native fetch

Quick Start

# Set at least one API key
export GROQ_API_KEY=gsk_...
export OPENROUTER_API_KEY=sk-or-...
export TOGETHER_API_KEY=...
export CEREBRAS_API_KEY=...

import { LlmRouter } from '@bytelyst/llm-router';

const router = new LlmRouter();

// Automatic routing — classifier picks best provider+model
const result = await router.chat({
  messages: [{ role: 'user', content: 'Write a quicksort in TypeScript' }],
});

console.log(result.response.choices[0].message.content);
console.log(`Served by: ${result.provider}/${result.model} in ${result.totalLatencyMs}ms`);

Explicit Provider Routing

// Force a specific provider:model
const result = await router.chat({
  messages: [{ role: 'user', content: 'Hello' }],
  model: 'groq:llama-3.3-70b-versatile',
});

Telemetry

const router = new LlmRouter({
  onTelemetry: entry => {
    // entry: { event, provider, model, attempt, latencyMs, category, tokens?, error? }
    console.log(`[${entry.event}] ${entry.provider}/${entry.model} — ${entry.latencyMs}ms`);
  },
});

Health Monitoring

const snapshots = router.getHealth();
// Returns: HealthSnapshot[] with per-provider stats
// { provider, model, totalRequests, successes, rateLimits, errors, avgLatencyMs, p95LatencyMs, healthy }

Configuration

const router = new LlmRouter({
  // Override default providers
  providers: [...],
  // Health window (default: 60s)
  healthWindowMs: 120_000,
  // Error rate to mark unhealthy (default: 50%)
  errorThreshold: 0.4,
  // Rate-limit rate to mark unhealthy (default: 30%)
  rateLimitThreshold: 0.2,
  // Request timeout (default: 30s)
  timeoutMs: 15_000,
  // Max retry attempts (default: 3)
  maxRetries: 4,
});

Provider Selection Logic

1. Classify prompt → code, math, reasoning, creative, or general
2. Score each available model based on category match, speed tier, context window, and model size
3. Filter unhealthy models (based on sliding-window error/rate-limit rates)
4. Round-robin across top-scoring providers to spread rate-limit load
5. Fallback on 429/5xx → exclude failed model, pick next best

Default Provider Registry

Provider	Models	Speed	Strengths
Groq	Llama 3.3 70B, Llama 3.1 8B, Gemma 2 9B	⚡ Fastest	General, reasoning, code
OpenRouter	DeepSeek R1, Llama 3.3 70B, Gemma 2 9B	Medium	Reasoning, code, math
Together	Llama 3.3 70B Turbo, DeepSeek R1 Distill	Medium	General, reasoning, code
Cerebras	Llama 3.3 70B	⚡ Fastest	General, reasoning, code

Adding Custom Providers

Any OpenAI-compatible endpoint works:

import { LlmRouter, DEFAULT_PROVIDERS } from '@bytelyst/llm-router';

const router = new LlmRouter({
  providers: [
    ...DEFAULT_PROVIDERS,
    {
      name: 'my-provider',
      baseUrl: 'https://my-api.example.com/v1',
      apiKeyEnv: 'MY_PROVIDER_KEY',
      rpmLimit: 60,
      tpmLimit: 100_000,
      models: [
        {
          id: 'my-model',
          label: 'My Model',
          contextWindow: 32_000,
          strengths: ['general', 'code'],
          speedTier: 2,
        },
      ],
    },
  ],
});