From a769f671e208715b56d85d6573b8cfff26b1e988 Mon Sep 17 00:00:00 2001 From: saravanakumardb1 Date: Fri, 20 Mar 2026 07:04:46 -0700 Subject: [PATCH] docs: cross-repo DRY audit + end-to-end migration roadmap - CROSS_REPO_DRY_AUDIT.md: identified 14 duplication patterns across 9 product backends + web clients (~4,200 LOC duplicated) - CROSS_REPO_DRY_MIGRATION_ROADMAP.md: 5-phase execution plan with concrete file lists, migration steps, verification checkpoints, rollback strategy, and success metrics - Phase 0: quick wins (delete errors.ts re-exports, standardize product-config) - Phase 1: @bytelyst/fastify-auth (auth + request-context for 9 repos) - Phase 2: @bytelyst/backend-config + datastore migration for 3 older repos - Phase 3: backend-flags, backend-telemetry, domain-events (FlowMonk + ActionTrail) - Phase 4: web client DRY (telemetry, diagnostics, product-config conveniences) - Estimated: 11 days, 5 new packages, ~3,260 LOC eliminated --- docs/CROSS_REPO_DRY_AUDIT.md | 277 ++++++ .../CROSS_REPO_DRY_MIGRATION_ROADMAP.md | 816 ++++++++++++++++++ 2 files changed, 1093 insertions(+) create mode 100644 docs/CROSS_REPO_DRY_AUDIT.md create mode 100644 docs/roadmaps/CROSS_REPO_DRY_MIGRATION_ROADMAP.md diff --git a/docs/CROSS_REPO_DRY_AUDIT.md b/docs/CROSS_REPO_DRY_AUDIT.md new file mode 100644 index 00000000..8641daff --- /dev/null +++ b/docs/CROSS_REPO_DRY_AUDIT.md @@ -0,0 +1,277 @@ +# Cross-Repo DRY Audit — Migration & Integration Candidates + +> **Date:** 2026-03-20 +> **Scope:** 9 product backends + web dashboards vs. 53 `@bytelyst/*` shared packages +> **Goal:** Identify duplicated patterns across product repos that should be consolidated into common platform packages. + +--- + +## Executive Summary + +Audited all 9 product repos with Fastify backends. Found **6 high-priority** and **4 medium-priority** duplication patterns across backends and web clients. The most impactful candidates involve backend `lib/` files that are **copy-pasted identically** (or near-identically) across all repos — differing only by product ID, port number, or service name. + +**Estimated LOC savings:** ~2,400 lines backend + ~1,800 lines web = **~4,200 lines** of duplicated code that could be replaced by shared packages. + +--- + +## HIGH Priority — Backend `lib/` Duplication (9 repos × 6 files) + +### 1. `auth.ts` — JWT verification + JWKS + role checking + +**Status:** Identical across all 9 repos (60-90 lines each = ~700 LOC total) +**Pattern:** RS256 JWKS verification → HS256 fallback → `extractAuth()` + `requireRole()` +**Only difference:** imports `config` from local `./config.js` + +**Recommendation:** Create `@bytelyst/fastify-auth` package: + +```ts +// New package: packages/fastify-auth/ +export function createAuthPlugin(opts: { jwtSecret: string; jwksUrl?: string }) { ... } +export function extractAuth(req: FastifyRequest): Promise { ... } +export function requireRole(req: FastifyRequest, ...roles: string[]): Promise { ... } +``` + +**Affected repos:** LysnrAI, MindLyst, ChronoMind, JarvisJr, NomGap, PeakPulse, FlowMonk, NoteLett, ActionTrail + +--- + +### 2. `request-context.ts` — productId validation + getUserId() + +**Status:** Nearly identical across all 9 repos (~30-50 lines each = ~350 LOC total) +**Only difference:** `const PRODUCT_ID = ''` string literal + +**Recommendation:** Add to `@bytelyst/fastify-auth` or new `@bytelyst/request-context`: + +```ts +export function createRequestContext(productId: string) { + return { + getRequestProductId(req: FastifyRequest): string { ... }, + getUserId(req: FastifyRequest): string { ... }, + }; +} +``` + +**Affected repos:** All 9 backends + +--- + +### 3. `errors.ts` — Re-export of `@bytelyst/errors` + +**Status:** Identical in 8/9 repos (FlowMonk imports directly) +**Pattern:** 7-12 line file that just re-exports from `@bytelyst/errors` + +**Recommendation:** **DELETE these files.** Product repos should import `@bytelyst/errors` directly in their modules — the re-export layer adds no value. + +**Affected repos:** LysnrAI, MindLyst, ChronoMind, JarvisJr, NomGap, PeakPulse, NoteLett, ActionTrail + +--- + +### 4. `config.ts` — Zod env schema + +**Status:** 80% identical across all 9 repos (~15-30 lines each = ~200 LOC total) +**Common fields:** PORT, HOST, NODE*ENV, CORS_ORIGIN, SERVICE_NAME, DB_PROVIDER, COSMOS*\*, JWT_SECRET, PLATFORM_JWKS_URL +**Product-specific fields:** PLATFORM_SERVICE_URL, EXTRACTION_SERVICE_URL, WEBHOOK_SECRET, LLM_API_KEY, etc. + +**Recommendation:** Create `@bytelyst/backend-config` with a composable schema: + +```ts +import { createBackendConfig } from '@bytelyst/backend-config'; + +export const config = createBackendConfig({ + serviceName: 'nomgap-backend', + defaultPort: 4013, + extraSchema: z.object({ + EXTRACTION_SERVICE_URL: z.string().default('http://localhost:4005'), + }), +}); +``` + +**Affected repos:** All 9 backends + +--- + +### 5. `datastore.ts` — DB_PROVIDER switch (Cosmos vs Memory) + +**Status:** Identical in 6/9 repos; 3 older repos use direct Cosmos without the provider abstraction +**Pattern:** Import `@bytelyst/datastore`, configure based on `DB_PROVIDER` env var + +**Recommendation:** Already have `@bytelyst/datastore` package. The 3 older repos (LysnrAI, MindLyst, ChronoMind) should be **migrated** to use `@bytelyst/datastore` with `DB_PROVIDER` support instead of direct `@azure/cosmos` calls. + +**Migration needed:** LysnrAI, MindLyst, ChronoMind backends + +--- + +### 6. `product-config.ts` — Backend product identity + +**Status:** Present in 4/9 repos (FlowMonk, ActionTrail, NoteLett, NomGap), hardcoded in others +**Pattern:** Read `shared/product.json`, export `PRODUCT_ID`, `productConfig` + +**Recommendation:** Already solved by `@bytelyst/config` (loadProductIdentity). The 5 repos without `product-config.ts` should adopt `@bytelyst/config/product-identity` instead of hardcoding product IDs. + +**Migration needed:** LysnrAI, MindLyst, ChronoMind, JarvisJr, PeakPulse + +--- + +## MEDIUM Priority — Backend Patterns (2-3 repos) + +### 7. `events.ts` — Domain event bus + SSE + webhook dispatch + +**Status:** FlowMonk and ActionTrail have near-identical event bus infrastructure (~150 lines each) +**Pattern:** `@bytelyst/event-store` + `@bytelyst/fastify-sse` + `@bytelyst/webhook-dispatch` wired together with typed domain events + SSE hub + webhook targets + +**Recommendation:** Create `@bytelyst/domain-events` factory: + +```ts +export function createDomainEventBus(opts: { + productId: string; + eventTypes: TEvent['type'][]; + webhookSecret?: string; +}) { ... } +``` + +Product repos would only define their event type interfaces and call `createDomainEventBus()`. + +**Affected repos:** FlowMonk, ActionTrail (and future products that need real-time events) + +--- + +### 8. `feature-flags.ts` — Backend in-memory flag registry + +**Status:** FlowMonk and ActionTrail have **identical** implementations (25 lines each) +**Pattern:** `Map` with `isFeatureEnabled()`, `getAllFlags()`, `setFlag()` + +**Recommendation:** Create `@bytelyst/backend-flags`: + +```ts +export function createFlagRegistry(defaults: Record) { ... } +``` + +**Affected repos:** FlowMonk, ActionTrail (expandable to all backends) + +--- + +### 9. `telemetry.ts` — Backend telemetry buffer + +**Status:** FlowMonk and ActionTrail have **byte-for-byte identical** implementations (33 lines each) +**Pattern:** In-memory buffer with `trackEvent()`, `getBufferedEvents()`, `flushEvents()` + +**Recommendation:** Add to existing `@bytelyst/events` or new `@bytelyst/backend-telemetry`: + +```ts +export function createTelemetryBuffer(opts: { enabled: boolean }) { ... } +``` + +**Affected repos:** FlowMonk, ActionTrail + +--- + +### 10. `scheduler.ts` — In-process job scheduler + +**Status:** ActionTrail has a full scheduler; FlowMonk has a scheduling engine +**Pattern:** Cron parsing, job registry, runner with diagnostics + +**Recommendation:** ActionTrail's scheduler could generalize to `@bytelyst/cron-scheduler` for any product needing periodic jobs. Currently `@bytelyst/queue` exists but is a different pattern (durable job queue). + +**Affected repos:** ActionTrail (FlowMonk's scheduler is domain-specific — planning, not cron) + +--- + +## HIGH Priority — Web `lib/` Duplication (5+ repos) + +### 11. `telemetry.ts` — Web telemetry init + +**Status:** NomGap, NoteLett, ChronoMind, JarvisJr have near-identical wrappers (~35 lines each) +**Only difference:** `channel` name string + +**Recommendation:** The `@bytelyst/telemetry-client` package already provides `createTelemetryClient()`. Products should call it directly in their `providers.tsx` instead of maintaining a wrapper file. Alternatively, add a `createWebTelemetry(productId, channel)` convenience to the package. + +**Affected repos:** NomGap, NoteLett, ChronoMind, JarvisJr web apps + +--- + +### 12. `diagnostics.ts` — Web diagnostics init + +**Status:** NomGap, NoteLett, ChronoMind, JarvisJr have near-identical wrappers (~40 lines each) +**Only difference:** `channel` name, auth token retrieval method + +**Recommendation:** Add `createWebDiagnostics(opts)` convenience to `@bytelyst/diagnostics-client` that handles install ID generation, localStorage auth token, and default config. + +**Affected repos:** NomGap, NoteLett, ChronoMind, JarvisJr web apps + +--- + +### 13. `feature-flags.ts` + `kill-switch.ts` — Web client wrappers + +**Status:** NomGap and NoteLett have identical wrappers; ChronoMind has similar + +**Recommendation:** These are thin wrappers around `@bytelyst/feature-flag-client` and `@bytelyst/kill-switch-client`. Consider adding React hook convenience exports directly to the packages. + +--- + +### 14. `product-config.ts` — Web product identity + +**Status:** 5 different patterns across repos: + +- NomGap/NoteLett: import `shared/product.json` + env var overrides +- FlowMonk/ActionTrail: inline object literal +- JarvisJr/ChronoMind: custom exports + +**Recommendation:** Standardize on `shared/product.json` import pattern. Could add `@bytelyst/web-config` convenience that reads product.json and provides typed exports with NEXT*PUBLIC* env var overrides. + +--- + +## Migration Priority Matrix + +| # | Candidate | LOC Saved | Repos | Effort | Priority | +| --- | -------------------------------------------------------- | --------- | ----- | -------- | -------- | +| 1 | `auth.ts` → `@bytelyst/fastify-auth` | ~700 | 9 | 2 days | **P0** | +| 2 | `request-context.ts` → merge into fastify-auth | ~350 | 9 | 1 day | **P0** | +| 3 | `errors.ts` → delete (import directly) | ~100 | 8 | 0.5 day | **P0** | +| 4 | `config.ts` → `@bytelyst/backend-config` | ~200 | 9 | 2 days | **P1** | +| 5 | Older repos → `@bytelyst/datastore` | ~300 | 3 | 1 day | **P1** | +| 6 | Hardcoded productId → `@bytelyst/config` | ~50 | 5 | 0.5 day | **P1** | +| 7 | `events.ts` → `@bytelyst/domain-events` | ~300 | 2 | 1.5 days | **P2** | +| 8 | `feature-flags.ts` → `@bytelyst/backend-flags` | ~50 | 2 | 0.5 day | **P2** | +| 9 | `telemetry.ts` → `@bytelyst/backend-telemetry` | ~70 | 2 | 0.5 day | **P2** | +| 10 | Web telemetry/diagnostics wrappers → package convenience | ~400 | 4 | 1 day | **P2** | +| 11 | Web `product-config.ts` → standardize pattern | ~200 | 5+ | 1 day | **P2** | + +**Total estimated effort: ~11 days for full DRY migration** +**Total LOC eliminated: ~2,700+ duplicated lines** + +--- + +## Repos NOT Yet Using Common Platform + +### `learning_ai_local_memory_gpt` + +- Standalone Express + SQLite app — no Cosmos, no auth, no platform-service +- **No integration needed** — this is intentionally a standalone local tool + +### `learning_ai_productivity_web` + +- No backend directory found +- Needs investigation to determine if it should adopt platform patterns + +--- + +## Recommended Execution Order + +### Sprint 1 (P0 — 3.5 days) + +1. Create `@bytelyst/fastify-auth` package with `extractAuth()`, `requireRole()`, `createRequestContext()` +2. Migrate all 9 backends to use it (delete local `auth.ts` + `request-context.ts`) +3. Delete all `errors.ts` re-export files, update imports to `@bytelyst/errors` directly + +### Sprint 2 (P1 — 3.5 days) + +4. Create `@bytelyst/backend-config` with composable Zod schema factory +5. Migrate LysnrAI, MindLyst, ChronoMind backends to `@bytelyst/datastore` +6. Migrate 5 repos to `@bytelyst/config/product-identity` for productId + +### Sprint 3 (P2 — 4 days) + +7. Create `@bytelyst/domain-events` factory (from FlowMonk/ActionTrail pattern) +8. Create `@bytelyst/backend-flags` and `@bytelyst/backend-telemetry` +9. Add convenience functions to web-side shared packages +10. Standardize web `product-config.ts` pattern diff --git a/docs/roadmaps/CROSS_REPO_DRY_MIGRATION_ROADMAP.md b/docs/roadmaps/CROSS_REPO_DRY_MIGRATION_ROADMAP.md new file mode 100644 index 00000000..fa65ae91 --- /dev/null +++ b/docs/roadmaps/CROSS_REPO_DRY_MIGRATION_ROADMAP.md @@ -0,0 +1,816 @@ +# Cross-Repo DRY Migration Roadmap + +> **Status:** Planning +> **Created:** 2026-03-20 +> **Audit:** See [`../CROSS_REPO_DRY_AUDIT.md`](../CROSS_REPO_DRY_AUDIT.md) +> **Total effort:** ~11 days across 5 phases +> **Total LOC eliminated:** ~4,200 duplicated lines across 9 product repos + +--- + +## Phase Overview + +| Phase | Name | New Packages | Repos Touched | Est. Effort | Status | +| ----- | --------------------------------------------------- | ------------ | ------------- | ----------- | ------ | +| **0** | Quick Wins (no new packages) | 0 | 8 | 1 day | ⬜ | +| **1** | `@bytelyst/fastify-auth` | 1 | 10 | 3 days | ⬜ | +| **2** | `@bytelyst/backend-config` + datastore migration | 1 | 12 | 3 days | ⬜ | +| **3** | Backend utilities (flags, telemetry, domain events) | 3 | 3 | 2 days | ⬜ | +| **4** | Web client DRY (telemetry, diagnostics, config) | 0 | 6 | 2 days | ⬜ | + +**Repos involved (9 product backends):** + +| Short Name | Repo | Backend Port | Backend Tests | +| ----------- | ----------------------------------- | ------------ | ------------- | +| LysnrAI | `learning_voice_ai_agent` | 4015 | 67 | +| MindLyst | `learning_multimodal_memory_agents` | 4014 | 63 | +| ChronoMind | `learning_ai_clock` | 4011 | 176 | +| JarvisJr | `learning_ai_jarvis_jr` | 4012 | 203 | +| NomGap | `learning_ai_fastgap` | 4013 | 203 | +| PeakPulse | `learning_ai_peakpulse` | 4010 | 59 | +| FlowMonk | `learning_ai_flowmonk` | 4017 | 181 | +| NoteLett | `learning_ai_notes` | 4016 | 80 | +| ActionTrail | `learning_ai_trails` | 4018 | 185 | + +--- + +## Phase 0 — Quick Wins (No New Packages) + +> **Goal:** Delete dead re-export files, standardize product identity loading. +> **Effort:** ~1 day +> **Risk:** Low — mechanical changes only. + +### 0.1 Delete `errors.ts` re-export files + +These files exist in 8 repos and contain only re-exports from `@bytelyst/errors`. Every module that imports from `./errors.js` should import from `@bytelyst/errors` directly (FlowMonk already does this). + +**Files to delete:** + +| Repo | File | +| ----------- | --------------------------- | +| LysnrAI | `backend/src/lib/errors.ts` | +| MindLyst | `backend/src/lib/errors.ts` | +| ChronoMind | `backend/src/lib/errors.ts` | +| JarvisJr | `backend/src/lib/errors.ts` | +| NomGap | `backend/src/lib/errors.ts` | +| PeakPulse | `backend/src/lib/errors.ts` | +| NoteLett | `backend/src/lib/errors.ts` | +| ActionTrail | `backend/src/lib/errors.ts` | + +**Steps per repo:** + +1. `grep -rn "from './errors.js'" backend/src/` → list all importing files +2. Replace `from './errors.js'` with `from '@bytelyst/errors'` in each file +3. Also replace `from '../lib/errors.js'` and `from '../../lib/errors.js'` patterns +4. Delete `backend/src/lib/errors.ts` +5. Run: `cd backend && npm run typecheck && npm test` + +**Commit:** `refactor(backend): remove errors.ts re-export, import @bytelyst/errors directly` + +### 0.2 Add `product-config.ts` to repos that hardcode product IDs + +Five repos hardcode `const PRODUCT_ID = ''` in `request-context.ts` instead of reading from `shared/product.json`. + +**Repos needing `product-config.ts`:** + +| Repo | Hardcoded PRODUCT_ID | Port | +| ---------- | -------------------------------------- | ---- | +| LysnrAI | `'lysnrai'` in `request-context.ts` | 4015 | +| MindLyst | `'mindlyst'` in `request-context.ts` | 4014 | +| ChronoMind | `'chronomind'` in `request-context.ts` | 4011 | +| JarvisJr | `'jarvisjr'` in `request-context.ts` | 4012 | +| PeakPulse | `'peakpulse'` in `request-context.ts` | 4010 | + +**Steps per repo:** + +1. Verify `shared/product.json` exists with correct `productId` +2. Create `backend/src/lib/product-config.ts`: + + ```ts + import { readFileSync } from 'node:fs'; + import { resolve, dirname } from 'node:path'; + import { fileURLToPath } from 'node:url'; + + const __dirname = dirname(fileURLToPath(import.meta.url)); + const raw = JSON.parse( + readFileSync(resolve(__dirname, '../../../../shared/product.json'), 'utf8') + ); + + export const PRODUCT_ID: string = raw.productId; + export const productConfig = raw as { + productId: string; + displayName: string; + domain: string; + backendPort: number; + [key: string]: unknown; + }; + ``` + +3. Update `request-context.ts`: replace `const PRODUCT_ID = ''` with `import { PRODUCT_ID } from './product-config.js'` +4. Update `config.ts` if it hardcodes the port: use `productConfig.backendPort` +5. Run: `cd backend && npm run typecheck && npm test` + +**Commit:** `refactor(backend): load product identity from shared/product.json` + +### 0.3 Verification checkpoint + +```bash +# Run ALL backend tests across all 9 repos +for repo in learning_voice_ai_agent learning_multimodal_memory_agents learning_ai_clock \ + learning_ai_jarvis_jr learning_ai_fastgap learning_ai_peakpulse \ + learning_ai_flowmonk learning_ai_notes learning_ai_trails; do + echo "=== $repo ===" && cd /Users/sd9235/code/mygh/$repo/backend && npm test 2>&1 | tail -3 && cd - +done +``` + +**Expected:** All 1,217 backend tests pass (unchanged count). + +--- + +## Phase 1 — `@bytelyst/fastify-auth` Package + +> **Goal:** Extract the duplicated JWT auth + request context into a single shared package. Delete `auth.ts` and `request-context.ts` from all 9 product repos. +> **Effort:** ~3 days +> **Risk:** Medium — auth is critical path. Must verify every repo's tests pass. + +### 1.1 Create `packages/fastify-auth/` + +**Location:** `learning_ai_common_plat/packages/fastify-auth/` + +**Package API:** + +```ts +// packages/fastify-auth/src/index.ts + +export interface FastifyAuthConfig { + /** HS256 JWT secret (required) */ + jwtSecret: string; + /** RS256 JWKS URL from platform-service (optional, tried first) */ + jwksUrl?: string; +} + +export interface RequestContextConfig { + /** Product ID to enforce (e.g., 'nomgap') */ + productId: string; + /** If true, returns 'demo-user' in non-production when no JWT present */ + allowDemoUser?: boolean; +} + +// Re-export AuthPayload type from @bytelyst/auth +export type { AuthPayload } from '@bytelyst/auth'; + +/** + * Extract and verify JWT from Authorization: Bearer header. + * Tries RS256 via JWKS first, falls back to HS256. + */ +export function createExtractAuth( + config: FastifyAuthConfig +): (req: { headers: { authorization?: string } }) => Promise; + +/** + * Require specific roles. Calls extractAuth first, then checks role. + */ +export function createRequireRole( + config: FastifyAuthConfig +): (req: { headers: { authorization?: string } }, ...roles: string[]) => Promise; + +/** + * Create product-scoped request context helpers. + */ +export function createRequestContext(config: RequestContextConfig): { + getUserId(req: FastifyRequest): string; + getRequestProductId(req: FastifyRequest): string; +}; +``` + +**Files to create:** + +| File | Purpose | +| ------------------------------------------------------------- | --------------------------------------------------- | +| `packages/fastify-auth/package.json` | Package manifest (`@bytelyst/fastify-auth`) | +| `packages/fastify-auth/tsconfig.json` | Extends `../../tsconfig.base.json` | +| `packages/fastify-auth/src/index.ts` | Main exports | +| `packages/fastify-auth/src/extract-auth.ts` | JWT verification (RS256 + HS256 fallback) | +| `packages/fastify-auth/src/request-context.ts` | Product-scoped getUserId + getRequestProductId | +| `packages/fastify-auth/src/types.ts` | Fastify module augmentation (JwtPayload on request) | +| `packages/fastify-auth/src/__tests__/extract-auth.test.ts` | Unit tests | +| `packages/fastify-auth/src/__tests__/request-context.test.ts` | Unit tests | + +**Dependencies:** + +```json +{ + "dependencies": { + "@bytelyst/errors": "workspace:*" + }, + "peerDependencies": { + "jose": ">=5.0.0", + "fastify": ">=5.0.0", + "@bytelyst/auth": "workspace:*" + } +} +``` + +**Steps:** + +1. Create package files (scaffold from existing `@bytelyst/auth` package structure) +2. Extract the canonical `extractAuth()` from any repo (they're identical) +3. Parameterize: accept `{ jwtSecret, jwksUrl }` instead of importing `config` +4. Extract `createRequestContext()` — parameterize with `{ productId, allowDemoUser }` +5. Add Fastify module augmentation for `req.jwtPayload` and `req.apiKeyUserId` +6. Write tests (port existing `auth.test.ts` from any repo) +7. Build: `pnpm --filter @bytelyst/fastify-auth build` +8. Run tests: `pnpm --filter @bytelyst/fastify-auth test` + +**Commit:** `feat(fastify-auth): new package — extractAuth, requireRole, createRequestContext` + +### 1.2 Migrate all 9 product backends + +**For each repo, the migration is identical:** + +**Files to modify:** + +| Action | File | Change | +| ----------- | ------------------------------------ | -------------------------------------------------------------------------------------------- | +| **Add dep** | `backend/package.json` | Add `"@bytelyst/fastify-auth": "file:../../learning_ai_common_plat/packages/fastify-auth"` | +| **Rewrite** | `backend/src/lib/auth.ts` | Replace 60-90 lines with ~10-line wrapper that calls `createExtractAuth()` with local config | +| **Rewrite** | `backend/src/lib/request-context.ts` | Replace 30-50 lines with ~5-line wrapper that calls `createRequestContext()` | +| **Verify** | — | `cd backend && npm install && npm run typecheck && npm test` | + +**New `auth.ts` (per repo, ~10 lines):** + +```ts +import { createExtractAuth, createRequireRole } from '@bytelyst/fastify-auth'; +import { config } from './config.js'; +export type { AuthPayload } from '@bytelyst/fastify-auth'; + +export const extractAuth = createExtractAuth({ + jwtSecret: config.JWT_SECRET, + jwksUrl: config.PLATFORM_JWKS_URL, +}); + +export const requireRole = createRequireRole({ + jwtSecret: config.JWT_SECRET, + jwksUrl: config.PLATFORM_JWKS_URL, +}); +``` + +**New `request-context.ts` (per repo, ~10 lines):** + +```ts +import { createRequestContext } from '@bytelyst/fastify-auth'; +import { PRODUCT_ID } from './product-config.js'; +export type { JwtPayload } from '@bytelyst/fastify-auth'; + +const ctx = createRequestContext({ productId: PRODUCT_ID, allowDemoUser: true }); +export const getUserId = ctx.getUserId; +export const getRequestProductId = ctx.getRequestProductId; +``` + +**Migration order (by test count — smallest first for safe iteration):** + +| Order | Repo | Tests | Notes | +| ----- | ----------- | ----- | ------------------------------------------------------------ | +| 1 | PeakPulse | 59 | Simplest, good canary | +| 2 | MindLyst | 63 | | +| 3 | LysnrAI | 67 | Has `audit.ts` extra import | +| 4 | NoteLett | 80 | | +| 5 | ChronoMind | 176 | | +| 6 | FlowMonk | 181 | Already imports `@bytelyst/errors` directly | +| 7 | ActionTrail | 185 | Has `apiKeyUserId` on request — include in type augmentation | +| 8 | JarvisJr | 203 | | +| 9 | NomGap | 203 | | + +**Per-repo commit:** `refactor(backend): migrate auth to @bytelyst/fastify-auth` + +### 1.3 Delete `auth.test.ts` from repos that test shared logic + +After migration, the `auth.test.ts` files in each repo test the same JWT logic that's now in the shared package. Keep only product-specific auth behavior tests; delete the ones testing `extractAuth()` and `requireRole()` directly (those now live in `packages/fastify-auth`). + +**Files to evaluate (keep if product-specific, delete if duplicate):** + +| Repo | File | Expected Action | +| ----- | ------------------------------ | ----------------------------------------------------------- | +| All 9 | `backend/src/lib/auth.test.ts` | Review — likely delete (logic now tested in shared package) | + +### 1.4 Verification checkpoint + +```bash +# Build shared package +cd /Users/sd9235/code/mygh/learning_ai_common_plat && pnpm --filter @bytelyst/fastify-auth build && pnpm --filter @bytelyst/fastify-auth test + +# Run ALL backend tests +for repo in learning_voice_ai_agent learning_multimodal_memory_agents learning_ai_clock \ + learning_ai_jarvis_jr learning_ai_fastgap learning_ai_peakpulse \ + learning_ai_flowmonk learning_ai_notes learning_ai_trails; do + echo "=== $repo ===" && cd /Users/sd9235/code/mygh/$repo/backend && npm install && npm run typecheck && npm test 2>&1 | tail -3 && cd - +done +``` + +**Expected:** All 1,217+ backend tests pass. Shared package has its own test suite (~15-20 tests). + +--- + +## Phase 2 — `@bytelyst/backend-config` + Datastore Migration + +> **Goal:** Extract the duplicated Zod env config schema into a composable factory. Migrate 3 older repos to `@bytelyst/datastore`. +> **Effort:** ~3 days +> **Risk:** Medium — config parsing is startup-critical. + +### 2.1 Create `packages/backend-config/` + +**Location:** `learning_ai_common_plat/packages/backend-config/` + +**Package API:** + +```ts +// packages/backend-config/src/index.ts +import { z } from 'zod'; + +/** + * Base schema shared across ALL product backends. + * Includes PORT, HOST, NODE_ENV, CORS_ORIGIN, SERVICE_NAME, + * DB_PROVIDER, COSMOS_*, JWT_SECRET, PLATFORM_JWKS_URL. + */ +export const baseBackendSchema = z.object({ + PORT: z.coerce.number(), + HOST: z.string().default('0.0.0.0'), + NODE_ENV: z.enum(['development', 'production', 'test']).default('development'), + CORS_ORIGIN: z.string().optional(), + SERVICE_NAME: z.string(), + DB_PROVIDER: z.enum(['cosmos', 'memory']).default('cosmos'), + COSMOS_ENDPOINT: z.string().default(''), + COSMOS_KEY: z.string().default(''), + COSMOS_DATABASE: z.string().default('lysnrai'), + JWT_SECRET: z.string().default('dev-secret-do-not-use-in-prod'), + PLATFORM_JWKS_URL: z.string().url().optional(), + PLATFORM_SERVICE_URL: z.string().default('http://localhost:4003'), +}); + +export type BaseBackendConfig = z.infer; + +/** + * Create a product-specific config by extending the base schema. + */ +export function createBackendConfig(opts: { + serviceName: string; + defaultPort: number; + defaultDatabase?: string; + extraSchema?: z.ZodObject; +}): BaseBackendConfig & z.infer>; +``` + +**Files to create:** + +| File | Purpose | +| ------------------------------------------------------ | --------------------------------------------- | +| `packages/backend-config/package.json` | Package manifest | +| `packages/backend-config/tsconfig.json` | TS config | +| `packages/backend-config/src/index.ts` | Base schema + `createBackendConfig()` factory | +| `packages/backend-config/src/__tests__/config.test.ts` | Tests | + +**Steps:** + +1. Create package scaffold +2. Extract common schema fields from the 9 existing `config.ts` files +3. Build the `createBackendConfig()` factory that merges base + extra schemas +4. Handle the Cosmos credential validation (required when `DB_PROVIDER=cosmos`) +5. Write tests +6. Build + test + +**Commit:** `feat(backend-config): new package — composable Zod env schema factory` + +### 2.2 Migrate all 9 product backends to `@bytelyst/backend-config` + +**New `config.ts` per repo (example: NomGap):** + +```ts +import { z } from 'zod'; +import { createBackendConfig } from '@bytelyst/backend-config'; + +export const config = createBackendConfig({ + serviceName: 'nomgap-backend', + defaultPort: 4013, + extraSchema: z.object({ + CORS_ORIGIN: z.string().default('http://localhost:3040'), + }), +}); +``` + +**Migration order:** Same as Phase 1 (smallest test count first). + +**Per-repo steps:** + +1. Add `"@bytelyst/backend-config": "file:../../learning_ai_common_plat/packages/backend-config"` to `package.json` +2. Rewrite `config.ts` to use `createBackendConfig()` +3. Ensure all modules still find the config fields they need +4. `cd backend && npm install && npm run typecheck && npm test` + +**Per-repo commit:** `refactor(backend): migrate config to @bytelyst/backend-config` + +### 2.3 Migrate LysnrAI, MindLyst, ChronoMind to `@bytelyst/datastore` + +These 3 repos use direct `@azure/cosmos` calls with custom `cosmos-init.ts` files instead of the `@bytelyst/datastore` abstraction that the 6 newer repos use. + +**Repos to migrate:** + +| Repo | Current Pattern | Target | +| ---------- | ----------------------------------------- | ------------------------------------- | +| LysnrAI | Direct `@azure/cosmos` + `cosmos-init.ts` | `@bytelyst/datastore` + `DB_PROVIDER` | +| MindLyst | Direct `@azure/cosmos` + `cosmos-init.ts` | `@bytelyst/datastore` + `DB_PROVIDER` | +| ChronoMind | Direct `@azure/cosmos` + `cosmos-init.ts` | `@bytelyst/datastore` + `DB_PROVIDER` | + +**Steps per repo:** + +1. Add `"@bytelyst/datastore": "file:../../learning_ai_common_plat/packages/datastore"` to `package.json` +2. Add `DB_PROVIDER` to config schema (default: `'cosmos'`) +3. Create/update `backend/src/lib/datastore.ts` matching the 6 newer repos' pattern: + + ```ts + import { + CosmosDatastoreProvider, + MemoryDatastoreProvider, + type DatastoreProvider, + } from '@bytelyst/datastore'; + import { config } from './config.js'; + + let provider: DatastoreProvider; + + export function getDatastoreProvider(): DatastoreProvider { + if (!provider) { + provider = + config.DB_PROVIDER === 'memory' + ? new MemoryDatastoreProvider() + : new CosmosDatastoreProvider({ + endpoint: config.COSMOS_ENDPOINT, + key: config.COSMOS_KEY, + database: config.COSMOS_DATABASE, + }); + } + return provider; + } + + export function getCollection(name: string) { + return getDatastoreProvider().getCollection(name); + } + ``` + +4. Update each repository file to use `getCollection()` instead of direct Cosmos container calls +5. Delete `cosmos-init.ts` if fully replaced +6. Add `DB_PROVIDER=memory` to test env for zero-dependency test runs +7. `cd backend && npm install && npm run typecheck && npm test` + +**Per-repo commit:** `refactor(backend): migrate to @bytelyst/datastore with DB_PROVIDER` + +### 2.4 Verification checkpoint + +```bash +# Build shared packages +cd /Users/sd9235/code/mygh/learning_ai_common_plat +pnpm --filter @bytelyst/backend-config build && pnpm --filter @bytelyst/backend-config test + +# Run ALL backend tests +for repo in learning_voice_ai_agent learning_multimodal_memory_agents learning_ai_clock \ + learning_ai_jarvis_jr learning_ai_fastgap learning_ai_peakpulse \ + learning_ai_flowmonk learning_ai_notes learning_ai_trails; do + echo "=== $repo ===" && cd /Users/sd9235/code/mygh/$repo/backend && npm install && npm run typecheck && npm test 2>&1 | tail -3 && cd - +done +``` + +**Expected:** All 1,217+ backend tests pass. 3 repos now support `DB_PROVIDER=memory` for offline testing. + +--- + +## Phase 3 — Backend Utilities (Flags, Telemetry, Domain Events) + +> **Goal:** Extract the 3 remaining duplicated backend patterns from FlowMonk + ActionTrail. +> **Effort:** ~2 days +> **Risk:** Low — only 2 repos affected, patterns are small. + +### 3.1 Create `packages/backend-flags/` + +**Source:** Identical code in FlowMonk + ActionTrail `backend/src/lib/feature-flags.ts` + +**Package API:** + +```ts +export interface FlagRegistry { + isFeatureEnabled(flag: string, userId?: string): boolean; + getAllFlags(): Record; + setFlag(flag: string, value: boolean): void; +} + +export function createFlagRegistry(defaults: Record): FlagRegistry; +``` + +**Files to create:** `package.json`, `tsconfig.json`, `src/index.ts`, `src/__tests__/flags.test.ts` + +**Migration:** + +- FlowMonk: rewrite `feature-flags.ts` to `import { createFlagRegistry } from '@bytelyst/backend-flags'` +- ActionTrail: same + +**Commit:** `feat(backend-flags): new package — in-memory feature flag registry` + +### 3.2 Create `packages/backend-telemetry/` + +**Source:** Byte-for-byte identical code in FlowMonk + ActionTrail `backend/src/lib/telemetry.ts` + +**Package API:** + +```ts +export interface TelemetryBuffer { + trackEvent(event: string, userId?: string, properties?: Record): void; + getBufferedEvents(): TelemetryEvent[]; + flushEvents(): TelemetryEvent[]; +} + +export function createTelemetryBuffer(opts: { enabled: boolean }): TelemetryBuffer; +``` + +**Files to create:** `package.json`, `tsconfig.json`, `src/index.ts`, `src/__tests__/telemetry.test.ts` + +**Migration:** + +- FlowMonk: rewrite `telemetry.ts` to import from `@bytelyst/backend-telemetry` +- ActionTrail: same + +**Commit:** `feat(backend-telemetry): new package — server-side telemetry event buffer` + +### 3.3 Create `packages/domain-events/` + +**Source:** Near-identical event bus infrastructure in FlowMonk + ActionTrail (~150 lines each) + +**Package API:** + +```ts +export interface DomainEventBusConfig { + productId: string; + webhookSecret?: string; +} + +export interface DomainEventBus { + emit(event: TEvent): void; + on(type: TEvent['type'], handler: (event: TEvent) => void): void; + getSSEHub(): SSEHub; + getEventStore(): EventStore; + registerWebhookTargets(targets: WebhookTarget[]): void; +} + +export function createDomainEventBus( + config: DomainEventBusConfig +): DomainEventBus; +``` + +**Product repos would only define their event type interfaces and call `createDomainEventBus()` — eliminating ~120 lines of wiring boilerplate per repo.** + +**Files to create:** `package.json`, `tsconfig.json`, `src/index.ts`, `src/bus.ts`, `src/__tests__/bus.test.ts` + +**Dependencies:** + +```json +{ + "dependencies": { + "@bytelyst/event-store": "workspace:*", + "@bytelyst/fastify-sse": "workspace:*", + "@bytelyst/webhook-dispatch": "workspace:*" + } +} +``` + +**Migration:** + +- FlowMonk: replace 150-line `events.ts` with ~30-line type definitions + `createDomainEventBus()` +- ActionTrail: same + +**Commit:** `feat(domain-events): new package — typed event bus + SSE + webhook wiring` + +### 3.4 Verification checkpoint + +```bash +cd /Users/sd9235/code/mygh/learning_ai_common_plat +pnpm --filter @bytelyst/backend-flags build && pnpm --filter @bytelyst/backend-flags test +pnpm --filter @bytelyst/backend-telemetry build && pnpm --filter @bytelyst/backend-telemetry test +pnpm --filter @bytelyst/domain-events build && pnpm --filter @bytelyst/domain-events test + +# Test affected repos +for repo in learning_ai_flowmonk learning_ai_trails; do + echo "=== $repo ===" && cd /Users/sd9235/code/mygh/$repo/backend && npm install && npm run typecheck && npm test 2>&1 | tail -3 && cd - +done +``` + +**Expected:** FlowMonk (181 tests) + ActionTrail (185 tests) still pass. 3 new packages with their own test suites. + +--- + +## Phase 4 — Web Client DRY + +> **Goal:** Reduce duplication in web app `lib/` directories. Add convenience functions to existing packages rather than creating new ones. +> **Effort:** ~2 days +> **Risk:** Low — web wrappers are thin, changes are cosmetic. + +### 4.1 Add `createWebTelemetry()` to `@bytelyst/telemetry-client` + +**Current state:** NomGap, NoteLett, ChronoMind, JarvisJr each have ~35-line `telemetry.ts` that wraps `createTelemetryClient()` with identical boilerplate. + +**Add to `packages/telemetry-client/src/index.ts`:** + +```ts +export function createWebTelemetry(opts: { + productId: string; + channel: string; + baseUrl?: string; + transport?: 'fetch' | 'beacon'; +}) { + let initialized = false; + const client = createTelemetryClient({ + productId: opts.productId, + baseUrl: opts.baseUrl ?? 'http://localhost:4003/api', + endpoint: '/telemetry/events', + platform: 'web', + channel: opts.channel, + transport: opts.transport ?? 'fetch', + appVersion: '0.1.0', + buildNumber: '1', + releaseChannel: 'dev', + osFamily: 'other', + }); + + return { + client, + init() { + if (initialized) return client; + client.init(); + client.trackEvent('info', 'app_shell', 'web_app_initialized'); + initialized = true; + return client; + }, + trackPageView(page: string) { + client.trackEvent('info', 'navigation', 'page_view', { feature: page }); + }, + }; +} +``` + +**Migration per web app:** + +- Replace ~35-line `telemetry.ts` with ~5-line wrapper: + ```ts + 'use client'; + import { createWebTelemetry } from '@bytelyst/telemetry-client'; + const { client, init, trackPageView } = createWebTelemetry({ + productId: 'nomgap', + channel: 'nomgap_web', + }); + export { client as telemetryClient, init as initTelemetry, trackPageView }; + ``` + +**Repos:** NomGap, NoteLett, ChronoMind, JarvisJr + +**Commit:** `feat(telemetry-client): add createWebTelemetry() convenience` + +### 4.2 Add `createWebDiagnostics()` to `@bytelyst/diagnostics-client` + +**Current state:** NomGap, NoteLett, ChronoMind, JarvisJr each have ~40-line `diagnostics.ts` with identical DiagnosticsClient init + install ID generation. + +**Add to `packages/diagnostics-client/`:** + +```ts +export function createWebDiagnostics(opts: { + productId: string; + channel: string; + serverUrl: string; + getAuthToken: () => string; +}): { init(): void; stop(): void }; +``` + +**Handles:** install ID generation, localStorage, default config, error swallowing. + +**Migration per web app:** Replace ~40-line file with ~8-line wrapper. + +**Repos:** NomGap, NoteLett, ChronoMind, JarvisJr + +**Commit:** `feat(diagnostics-client): add createWebDiagnostics() convenience` + +### 4.3 Standardize web `product-config.ts` pattern + +**Current state:** 5 different patterns across repos. + +**Target pattern (already used by NomGap + NoteLett):** + +```ts +import productIdentity from '../../../shared/product.json'; + +export const PRODUCT_NAME = process.env.NEXT_PUBLIC_PRODUCT_NAME ?? productIdentity.displayName; +export const PRODUCT_ID = process.env.NEXT_PUBLIC_PRODUCT_ID ?? productIdentity.productId; +export const PLATFORM_SERVICE_URL = + process.env.NEXT_PUBLIC_PLATFORM_SERVICE_URL ?? 'http://localhost:4003/api'; +export const PLATFORM_SERVICE_ORIGIN = PLATFORM_SERVICE_URL.replace(/\/api\/?$/, ''); +export const BACKEND_API_URL = + process.env.NEXT_PUBLIC_BACKEND_API_URL ?? `http://localhost:${productIdentity.backendPort}/api`; +export const DIAGNOSTICS_URL = process.env.NEXT_PUBLIC_DIAGNOSTICS_URL ?? PLATFORM_SERVICE_ORIGIN; +export const TELEMETRY_TRANSPORT = + process.env.NEXT_PUBLIC_TELEMETRY_TRANSPORT === 'beacon' ? 'beacon' : 'fetch'; +``` + +**Repos to standardize:** + +- FlowMonk web: uses inline object literal → adopt `shared/product.json` import +- ActionTrail web: same +- JarvisJr web: uses custom `getPlatformBaseURL()` → standardize +- ChronoMind web: uses `auth-api.ts` for product config → separate into `product-config.ts` + +**Per-repo commit:** `refactor(web): standardize product-config.ts to shared/product.json pattern` + +### 4.4 Verification checkpoint + +```bash +# Build updated packages +cd /Users/sd9235/code/mygh/learning_ai_common_plat +pnpm --filter @bytelyst/telemetry-client build +pnpm --filter @bytelyst/diagnostics-client build + +# Typecheck all web apps +for repo in learning_ai_fastgap/web learning_ai_notes/web learning_ai_clock/web \ + learning_ai_jarvis_jr/web learning_ai_flowmonk/web learning_ai_trails/web; do + echo "=== $repo ===" && cd /Users/sd9235/code/mygh/$repo && npx tsc --noEmit 2>&1 | tail -3 && cd - +done +``` + +--- + +## Commit & Push Strategy + +Each phase should be committed and pushed independently: + +| Phase | Commits | Push Strategy | +| ----- | ------------------------------------------------------------------ | ------------------------------------------ | +| **0** | 2 commits per repo (errors cleanup + product-config) | Push all repos after phase complete | +| **1** | 1 commit in common-plat (new package) + 1 commit per product repo | Push common-plat first, then product repos | +| **2** | 1 commit in common-plat + 1-2 commits per product repo | Push common-plat first, then product repos | +| **3** | 3 commits in common-plat + 1 commit each in FlowMonk + ActionTrail | Push common-plat first | +| **4** | 2 commits in common-plat + 1 commit per web app | Push common-plat first | + +**Total commits: ~35-40 across all repos** + +--- + +## Rollback Plan + +Every migration follows the same rollback strategy: + +1. **Product repos reference shared packages via `file:` refs** — no npm publish needed +2. **Each phase is independently revertable** — revert the product repo commit and the old `lib/` files return +3. **Shared packages are additive** — creating `@bytelyst/fastify-auth` doesn't break anything that doesn't use it +4. **Test gate:** Every phase ends with a verification checkpoint. If any repo's tests fail, stop and fix before continuing. + +--- + +## Final State (After All Phases) + +### Backend `lib/` directory (target state per product repo) + +| File | Source | Lines | +| -------------------- | ------------------------------------------------------- | ----- | +| `config.ts` | `createBackendConfig()` from `@bytelyst/backend-config` | ~8-15 | +| `auth.ts` | `createExtractAuth()` from `@bytelyst/fastify-auth` | ~10 | +| `request-context.ts` | `createRequestContext()` from `@bytelyst/fastify-auth` | ~6 | +| `product-config.ts` | Reads `shared/product.json` | ~12 | +| `datastore.ts` | `@bytelyst/datastore` with `DB_PROVIDER` | ~15 | +| ~~`errors.ts`~~ | **DELETED** — import `@bytelyst/errors` directly | 0 | + +**Average backend `lib/` shrinks from ~250 lines to ~60 lines (76% reduction).** + +### New packages in common-plat + +| Package | Phase | Tests | Purpose | +| ----------------------------- | ----- | ----- | ----------------------------------------------- | +| `@bytelyst/fastify-auth` | 1 | ~20 | JWT auth + request context for Fastify backends | +| `@bytelyst/backend-config` | 2 | ~10 | Composable Zod env config factory | +| `@bytelyst/backend-flags` | 3 | ~8 | In-memory feature flag registry | +| `@bytelyst/backend-telemetry` | 3 | ~8 | Server-side telemetry buffer | +| `@bytelyst/domain-events` | 3 | ~12 | Event bus + SSE + webhook factory | + +**Total new shared tests: ~58** + +### Updated packages in common-plat + +| Package | Phase | Change | +| ------------------------------ | ----- | ------------------------------------ | +| `@bytelyst/telemetry-client` | 4 | `createWebTelemetry()` convenience | +| `@bytelyst/diagnostics-client` | 4 | `createWebDiagnostics()` convenience | + +--- + +## Success Metrics + +| Metric | Before | After | +| ------------------------------------------- | ------ | ----------------------------------- | +| Duplicated backend `lib/` LOC | ~2,400 | ~540 (thin wrappers) | +| Duplicated web `lib/` LOC | ~1,800 | ~400 (thin wrappers) | +| Total duplicated LOC | ~4,200 | ~940 | +| **LOC eliminated** | — | **~3,260** | +| Shared packages | 53 | 58 | +| Repos with `DB_PROVIDER` support | 6 | 9 | +| Repos with standardized `product-config.ts` | 4 | 9 | +| Backend test count | 1,217 | 1,217+ (unchanged + ~58 new shared) |