From d10322095a1eba25e741b54115242e0255f36f7c Mon Sep 17 00:00:00 2001 From: saravanakumardb1 Date: Fri, 20 Mar 2026 07:15:07 -0700 Subject: [PATCH] docs: fix 10 inaccuracies in DRY audit + roadmap after thorough review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Verified claims against actual codebase. Key corrections: Audit (CROSS_REPO_DRY_AUDIT.md): - Fix #5: all 9 repos already use @bytelyst/datastore — issue is only DB_PROVIDER missing from config.ts in 3 older repos (not a full migration) - Fix #6: only 3 repos have product-config.ts (not 4) — NomGap was wrong - Fix #11: web telemetry.ts is in NomGap+NoteLett+ChronoMind+LysnrAI (not JarvisJr) - Fix #12: web diagnostics.ts includes LysnrAI user-dashboard-web (5 repos total) - Fix auth.ts LOC: exactly 79 lines × 9 repos = 711 (was '60-90') - Fix request-context.ts LOC: 30-49 lines range (was '~30-50') - Fix package count: 50 packages (not 53) - Add items 15-16: web auth.ts + billing-client.ts (noted as keep-as-is) - Fix LOC math: ~2,700 total (was inflated ~4,200) - Add cosmos-init.ts note (5 repos, product-specific, not consolidation candidates) Roadmap (CROSS_REPO_DRY_MIGRATION_ROADMAP.md): - Phase 0.2: 6 repos need product-config.ts (add NomGap) - Phase 2.3: rewritten from 'migrate to datastore' to 'add DB_PROVIDER to config' - Phase 4: add LysnrAI user-dashboard-web to affected repos, remove JarvisJr from telemetry - Fix product-config.ts template path (was ../../../../, now ../../../ with depth note) - Fix success metrics: packages 50→55, product-config 3→9, LOC ~1,760 - Fix overview table: Phase 2 name, Phase 4 repo count --- docs/CROSS_REPO_DRY_AUDIT.md | 75 ++++++++---- .../CROSS_REPO_DRY_MIGRATION_ROADMAP.md | 114 ++++++++---------- 2 files changed, 99 insertions(+), 90 deletions(-) diff --git a/docs/CROSS_REPO_DRY_AUDIT.md b/docs/CROSS_REPO_DRY_AUDIT.md index 8641daff..79da6975 100644 --- a/docs/CROSS_REPO_DRY_AUDIT.md +++ b/docs/CROSS_REPO_DRY_AUDIT.md @@ -1,16 +1,16 @@ # Cross-Repo DRY Audit — Migration & Integration Candidates > **Date:** 2026-03-20 -> **Scope:** 9 product backends + web dashboards vs. 53 `@bytelyst/*` shared packages +> **Scope:** 9 product backends + web dashboards vs. 50 `@bytelyst/*` shared packages > **Goal:** Identify duplicated patterns across product repos that should be consolidated into common platform packages. --- ## Executive Summary -Audited all 9 product repos with Fastify backends. Found **6 high-priority** and **4 medium-priority** duplication patterns across backends and web clients. The most impactful candidates involve backend `lib/` files that are **copy-pasted identically** (or near-identically) across all repos — differing only by product ID, port number, or service name. +Audited all 9 product repos with Fastify backends + 5 Next.js web dashboards. Found **6 high-priority** and **4 medium-priority** backend duplication patterns, plus **6 web-side** duplication patterns. The most impactful candidates involve backend `lib/` files that are **copy-pasted identically** (or near-identically) across all repos — differing only by product ID, port number, or service name. -**Estimated LOC savings:** ~2,400 lines backend + ~1,800 lines web = **~4,200 lines** of duplicated code that could be replaced by shared packages. +**Estimated LOC savings:** ~1,700 lines backend + ~1,000 lines web = **~2,700 lines** of duplicated code that could be replaced by shared packages or eliminated entirely. --- @@ -18,7 +18,7 @@ Audited all 9 product repos with Fastify backends. Found **6 high-priority** and ### 1. `auth.ts` — JWT verification + JWKS + role checking -**Status:** Identical across all 9 repos (60-90 lines each = ~700 LOC total) +**Status:** Identical across all 9 repos (79 lines each = 711 LOC total) **Pattern:** RS256 JWKS verification → HS256 fallback → `extractAuth()` + `requireRole()` **Only difference:** imports `config` from local `./config.js` @@ -37,7 +37,7 @@ export function requireRole(req: FastifyRequest, ...roles: string[]): Promise **Created:** 2026-03-20 > **Audit:** See [`../CROSS_REPO_DRY_AUDIT.md`](../CROSS_REPO_DRY_AUDIT.md) > **Total effort:** ~11 days across 5 phases -> **Total LOC eliminated:** ~4,200 duplicated lines across 9 product repos +> **Total LOC eliminated:** ~2,700 duplicated lines across 9 product repos + 5 web dashboards --- @@ -12,11 +12,11 @@ | Phase | Name | New Packages | Repos Touched | Est. Effort | Status | | ----- | --------------------------------------------------- | ------------ | ------------- | ----------- | ------ | -| **0** | Quick Wins (no new packages) | 0 | 8 | 1 day | ⬜ | +| **0** | Quick Wins (no new packages) | 0 | 9 | 1 day | ⬜ | | **1** | `@bytelyst/fastify-auth` | 1 | 10 | 3 days | ⬜ | -| **2** | `@bytelyst/backend-config` + datastore migration | 1 | 12 | 3 days | ⬜ | +| **2** | `@bytelyst/backend-config` + DB_PROVIDER cleanup | 1 | 12 | 3 days | ⬜ | | **3** | Backend utilities (flags, telemetry, domain events) | 3 | 3 | 2 days | ⬜ | -| **4** | Web client DRY (telemetry, diagnostics, config) | 0 | 6 | 2 days | ⬜ | +| **4** | Web client DRY (telemetry, diagnostics, config) | 0 | 7 | 2 days | ⬜ | **Repos involved (9 product backends):** @@ -69,7 +69,7 @@ These files exist in 8 repos and contain only re-exports from `@bytelyst/errors` ### 0.2 Add `product-config.ts` to repos that hardcode product IDs -Five repos hardcode `const PRODUCT_ID = ''` in `request-context.ts` instead of reading from `shared/product.json`. +Six repos hardcode `const PRODUCT_ID = ''` in `request-context.ts` instead of reading from `shared/product.json`. **Repos needing `product-config.ts`:** @@ -79,6 +79,7 @@ Five repos hardcode `const PRODUCT_ID = ''` in `request-context.ts` instea | MindLyst | `'mindlyst'` in `request-context.ts` | 4014 | | ChronoMind | `'chronomind'` in `request-context.ts` | 4011 | | JarvisJr | `'jarvisjr'` in `request-context.ts` | 4012 | +| NomGap | `'nomgap'` in `request-context.ts` | 4013 | | PeakPulse | `'peakpulse'` in `request-context.ts` | 4010 | **Steps per repo:** @@ -92,9 +93,10 @@ Five repos hardcode `const PRODUCT_ID = ''` in `request-context.ts` instea import { fileURLToPath } from 'node:url'; const __dirname = dirname(fileURLToPath(import.meta.url)); - const raw = JSON.parse( - readFileSync(resolve(__dirname, '../../../../shared/product.json'), 'utf8') - ); + // Path works from both src/ (dev with tsx) and dist/ (compiled): + // src/lib/ -> ../../shared/ OR dist/lib/ -> ../../shared/ + // Both resolve to /shared/product.json since backend/ is a subdirectory. + const raw = JSON.parse(readFileSync(resolve(__dirname, '../../../shared/product.json'), 'utf8')); export const PRODUCT_ID: string = raw.productId; export const productConfig = raw as { @@ -106,6 +108,12 @@ Five repos hardcode `const PRODUCT_ID = ''` in `request-context.ts` instea }; ``` + > **IMPORTANT:** The relative path `../../../shared/product.json` resolves from + > `backend/src/lib/` (3 levels up = repo root) and also from `backend/dist/lib/` + > (3 levels up = repo root). Verify with: `ls $(dirname $0)/../../../shared/product.json`. + > For repos where `backend/` is at repo root, the path may need to be `../../../../shared/product.json` + > — check the existing `product-config.ts` in FlowMonk/ActionTrail/NoteLett for the correct depth. + 3. Update `request-context.ts`: replace `const PRODUCT_ID = ''` with `import { PRODUCT_ID } from './product-config.js'` 4. Update `config.ts` if it hardcodes the port: use `productConfig.backendPort` 5. Run: `cd backend && npm run typecheck && npm test` @@ -408,59 +416,27 @@ export const config = createBackendConfig({ **Per-repo commit:** `refactor(backend): migrate config to @bytelyst/backend-config` -### 2.3 Migrate LysnrAI, MindLyst, ChronoMind to `@bytelyst/datastore` +### 2.3 Add `DB_PROVIDER` to config schema in 3 older repos -These 3 repos use direct `@azure/cosmos` calls with custom `cosmos-init.ts` files instead of the `@bytelyst/datastore` abstraction that the 6 newer repos use. +All 9 repos already use `@bytelyst/datastore` in their `datastore.ts`. However, 3 older repos (LysnrAI, MindLyst, ChronoMind) read `DB_PROVIDER` directly from `process.env` inside `datastore.ts` instead of declaring it in their Zod config schema. This means `DB_PROVIDER` isn't validated at startup. -**Repos to migrate:** +**Repos to fix:** -| Repo | Current Pattern | Target | -| ---------- | ----------------------------------------- | ------------------------------------- | -| LysnrAI | Direct `@azure/cosmos` + `cosmos-init.ts` | `@bytelyst/datastore` + `DB_PROVIDER` | -| MindLyst | Direct `@azure/cosmos` + `cosmos-init.ts` | `@bytelyst/datastore` + `DB_PROVIDER` | -| ChronoMind | Direct `@azure/cosmos` + `cosmos-init.ts` | `@bytelyst/datastore` + `DB_PROVIDER` | +| Repo | Current Pattern | Target | +| ---------- | ------------------------------------------- | -------------------------------------------------- | +| LysnrAI | `process.env.DB_PROVIDER` in `datastore.ts` | `config.DB_PROVIDER` via Zod schema in `config.ts` | +| MindLyst | `process.env.DB_PROVIDER` in `datastore.ts` | `config.DB_PROVIDER` via Zod schema in `config.ts` | +| ChronoMind | `process.env.DB_PROVIDER` in `datastore.ts` | `config.DB_PROVIDER` via Zod schema in `config.ts` | **Steps per repo:** -1. Add `"@bytelyst/datastore": "file:../../learning_ai_common_plat/packages/datastore"` to `package.json` -2. Add `DB_PROVIDER` to config schema (default: `'cosmos'`) -3. Create/update `backend/src/lib/datastore.ts` matching the 6 newer repos' pattern: +1. Add `DB_PROVIDER: z.enum(['cosmos', 'memory']).default('cosmos')` to `config.ts` Zod schema +2. Update `datastore.ts` to read `config.DB_PROVIDER` instead of `process.env.DB_PROVIDER` +3. `cd backend && npm run typecheck && npm test` - ```ts - import { - CosmosDatastoreProvider, - MemoryDatastoreProvider, - type DatastoreProvider, - } from '@bytelyst/datastore'; - import { config } from './config.js'; +**Note:** Do NOT delete `cosmos-init.ts` files — these register product-specific Cosmos containers and are still needed alongside `@bytelyst/datastore`. - let provider: DatastoreProvider; - - export function getDatastoreProvider(): DatastoreProvider { - if (!provider) { - provider = - config.DB_PROVIDER === 'memory' - ? new MemoryDatastoreProvider() - : new CosmosDatastoreProvider({ - endpoint: config.COSMOS_ENDPOINT, - key: config.COSMOS_KEY, - database: config.COSMOS_DATABASE, - }); - } - return provider; - } - - export function getCollection(name: string) { - return getDatastoreProvider().getCollection(name); - } - ``` - -4. Update each repository file to use `getCollection()` instead of direct Cosmos container calls -5. Delete `cosmos-init.ts` if fully replaced -6. Add `DB_PROVIDER=memory` to test env for zero-dependency test runs -7. `cd backend && npm install && npm run typecheck && npm test` - -**Per-repo commit:** `refactor(backend): migrate to @bytelyst/datastore with DB_PROVIDER` +**Per-repo commit:** `refactor(backend): add DB_PROVIDER to config schema` ### 2.4 Verification checkpoint @@ -477,7 +453,7 @@ for repo in learning_voice_ai_agent learning_multimodal_memory_agents learning_a done ``` -**Expected:** All 1,217+ backend tests pass. 3 repos now support `DB_PROVIDER=memory` for offline testing. +**Expected:** All 1,217+ backend tests pass. All 9 repos now have `DB_PROVIDER` in their validated config schema. --- @@ -665,7 +641,9 @@ export function createWebTelemetry(opts: { export { client as telemetryClient, init as initTelemetry, trackPageView }; ``` -**Repos:** NomGap, NoteLett, ChronoMind, JarvisJr +**Repos:** NomGap, NoteLett, ChronoMind, LysnrAI (user-dashboard-web) + +**Note:** JarvisJr does NOT have a `telemetry.ts` wrapper file. **Commit:** `feat(telemetry-client): add createWebTelemetry() convenience` @@ -688,7 +666,7 @@ export function createWebDiagnostics(opts: { **Migration per web app:** Replace ~40-line file with ~8-line wrapper. -**Repos:** NomGap, NoteLett, ChronoMind, JarvisJr +**Repos:** NomGap, NoteLett, ChronoMind, JarvisJr, LysnrAI (user-dashboard-web) **Commit:** `feat(diagnostics-client): add createWebDiagnostics() convenience` @@ -719,6 +697,7 @@ export const TELEMETRY_TRANSPORT = - ActionTrail web: same - JarvisJr web: uses custom `getPlatformBaseURL()` → standardize - ChronoMind web: uses `auth-api.ts` for product config → separate into `product-config.ts` +- LysnrAI user-dashboard-web: uses `@bytelyst/config` `loadProductIdentity()` → align with `shared/product.json` pattern **Per-repo commit:** `refactor(web): standardize product-config.ts to shared/product.json pattern` @@ -732,7 +711,8 @@ pnpm --filter @bytelyst/diagnostics-client build # Typecheck all web apps for repo in learning_ai_fastgap/web learning_ai_notes/web learning_ai_clock/web \ - learning_ai_jarvis_jr/web learning_ai_flowmonk/web learning_ai_trails/web; do + learning_ai_jarvis_jr/web learning_ai_flowmonk/web learning_ai_trails/web \ + learning_voice_ai_agent/user-dashboard-web; do echo "=== $repo ===" && cd /Users/sd9235/code/mygh/$repo && npx tsc --noEmit 2>&1 | tail -3 && cd - done ``` @@ -779,7 +759,7 @@ Every migration follows the same rollback strategy: | `datastore.ts` | `@bytelyst/datastore` with `DB_PROVIDER` | ~15 | | ~~`errors.ts`~~ | **DELETED** — import `@bytelyst/errors` directly | 0 | -**Average backend `lib/` shrinks from ~250 lines to ~60 lines (76% reduction).** +**Average backend `lib/` shrinks from ~200 lines to ~60 lines (70% reduction).** ### New packages in common-plat @@ -806,11 +786,15 @@ Every migration follows the same rollback strategy: | Metric | Before | After | | ------------------------------------------- | ------ | ----------------------------------- | -| Duplicated backend `lib/` LOC | ~2,400 | ~540 (thin wrappers) | -| Duplicated web `lib/` LOC | ~1,800 | ~400 (thin wrappers) | -| Total duplicated LOC | ~4,200 | ~940 | -| **LOC eliminated** | — | **~3,260** | -| Shared packages | 53 | 58 | -| Repos with `DB_PROVIDER` support | 6 | 9 | -| Repos with standardized `product-config.ts` | 4 | 9 | +| Duplicated backend `lib/` LOC | ~1,700 | ~540 (thin wrappers) | +| Duplicated web `lib/` LOC | ~1,000 | ~400 (thin wrappers) | +| Total duplicated LOC | ~2,700 | ~940 | +| **LOC eliminated** | — | **~1,760** | +| Shared packages | 50 | 55 | +| Repos with `DB_PROVIDER` in config schema | 6 | 9 | +| Repos with standardized `product-config.ts` | 3 | 9 | | Backend test count | 1,217 | 1,217+ (unchanged + ~58 new shared) | + +> **Note on package count:** The 50-package baseline counts directories in +> `learning_ai_common_plat/packages/` that contain a `package.json`. The 5 new +> packages proposed in this roadmap bring the total to 55.