docs: cross-repo DRY audit + end-to-end migration roadmap
- CROSS_REPO_DRY_AUDIT.md: identified 14 duplication patterns across 9 product backends + web clients (~4,200 LOC duplicated) - CROSS_REPO_DRY_MIGRATION_ROADMAP.md: 5-phase execution plan with concrete file lists, migration steps, verification checkpoints, rollback strategy, and success metrics - Phase 0: quick wins (delete errors.ts re-exports, standardize product-config) - Phase 1: @bytelyst/fastify-auth (auth + request-context for 9 repos) - Phase 2: @bytelyst/backend-config + datastore migration for 3 older repos - Phase 3: backend-flags, backend-telemetry, domain-events (FlowMonk + ActionTrail) - Phase 4: web client DRY (telemetry, diagnostics, product-config conveniences) - Estimated: 11 days, 5 new packages, ~3,260 LOC eliminated
This commit is contained in:
saravanakumardb1
parent
10c288857d
commit
a769f671e2
277
docs/CROSS_REPO_DRY_AUDIT.md
Normal file
277
docs/CROSS_REPO_DRY_AUDIT.md
Normal file
@ -0,0 +1,277 @@
|
|||||||
|
# Cross-Repo DRY Audit — Migration & Integration Candidates
|
||||||
|
|
||||||
|
> **Date:** 2026-03-20
|
||||||
|
> **Scope:** 9 product backends + web dashboards vs. 53 `@bytelyst/*` shared packages
|
||||||
|
> **Goal:** Identify duplicated patterns across product repos that should be consolidated into common platform packages.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Executive Summary
|
||||||
|
|
||||||
|
Audited all 9 product repos with Fastify backends. Found **6 high-priority** and **4 medium-priority** duplication patterns across backends and web clients. The most impactful candidates involve backend `lib/` files that are **copy-pasted identically** (or near-identically) across all repos — differing only by product ID, port number, or service name.
|
||||||
|
|
||||||
|
**Estimated LOC savings:** ~2,400 lines backend + ~1,800 lines web = **~4,200 lines** of duplicated code that could be replaced by shared packages.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## HIGH Priority — Backend `lib/` Duplication (9 repos × 6 files)
|
||||||
|
|
||||||
|
### 1. `auth.ts` — JWT verification + JWKS + role checking
|
||||||
|
|
||||||
|
**Status:** Identical across all 9 repos (60-90 lines each = ~700 LOC total)
|
||||||
|
**Pattern:** RS256 JWKS verification → HS256 fallback → `extractAuth()` + `requireRole()`
|
||||||
|
**Only difference:** imports `config` from local `./config.js`
|
||||||
|
|
||||||
|
**Recommendation:** Create `@bytelyst/fastify-auth` package:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
// New package: packages/fastify-auth/
|
||||||
|
export function createAuthPlugin(opts: { jwtSecret: string; jwksUrl?: string }) { ... }
|
||||||
|
export function extractAuth(req: FastifyRequest): Promise<AuthPayload> { ... }
|
||||||
|
export function requireRole(req: FastifyRequest, ...roles: string[]): Promise<AuthPayload> { ... }
|
||||||
|
```
|
||||||
|
|
||||||
|
**Affected repos:** LysnrAI, MindLyst, ChronoMind, JarvisJr, NomGap, PeakPulse, FlowMonk, NoteLett, ActionTrail
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 2. `request-context.ts` — productId validation + getUserId()
|
||||||
|
|
||||||
|
**Status:** Nearly identical across all 9 repos (~30-50 lines each = ~350 LOC total)
|
||||||
|
**Only difference:** `const PRODUCT_ID = '<product>'` string literal
|
||||||
|
|
||||||
|
**Recommendation:** Add to `@bytelyst/fastify-auth` or new `@bytelyst/request-context`:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
export function createRequestContext(productId: string) {
|
||||||
|
return {
|
||||||
|
getRequestProductId(req: FastifyRequest): string { ... },
|
||||||
|
getUserId(req: FastifyRequest): string { ... },
|
||||||
|
};
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Affected repos:** All 9 backends
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 3. `errors.ts` — Re-export of `@bytelyst/errors`
|
||||||
|
|
||||||
|
**Status:** Identical in 8/9 repos (FlowMonk imports directly)
|
||||||
|
**Pattern:** 7-12 line file that just re-exports from `@bytelyst/errors`
|
||||||
|
|
||||||
|
**Recommendation:** **DELETE these files.** Product repos should import `@bytelyst/errors` directly in their modules — the re-export layer adds no value.
|
||||||
|
|
||||||
|
**Affected repos:** LysnrAI, MindLyst, ChronoMind, JarvisJr, NomGap, PeakPulse, NoteLett, ActionTrail
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 4. `config.ts` — Zod env schema
|
||||||
|
|
||||||
|
**Status:** 80% identical across all 9 repos (~15-30 lines each = ~200 LOC total)
|
||||||
|
**Common fields:** PORT, HOST, NODE*ENV, CORS_ORIGIN, SERVICE_NAME, DB_PROVIDER, COSMOS*\*, JWT_SECRET, PLATFORM_JWKS_URL
|
||||||
|
**Product-specific fields:** PLATFORM_SERVICE_URL, EXTRACTION_SERVICE_URL, WEBHOOK_SECRET, LLM_API_KEY, etc.
|
||||||
|
|
||||||
|
**Recommendation:** Create `@bytelyst/backend-config` with a composable schema:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
import { createBackendConfig } from '@bytelyst/backend-config';
|
||||||
|
|
||||||
|
export const config = createBackendConfig({
|
||||||
|
serviceName: 'nomgap-backend',
|
||||||
|
defaultPort: 4013,
|
||||||
|
extraSchema: z.object({
|
||||||
|
EXTRACTION_SERVICE_URL: z.string().default('http://localhost:4005'),
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
**Affected repos:** All 9 backends
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 5. `datastore.ts` — DB_PROVIDER switch (Cosmos vs Memory)
|
||||||
|
|
||||||
|
**Status:** Identical in 6/9 repos; 3 older repos use direct Cosmos without the provider abstraction
|
||||||
|
**Pattern:** Import `@bytelyst/datastore`, configure based on `DB_PROVIDER` env var
|
||||||
|
|
||||||
|
**Recommendation:** Already have `@bytelyst/datastore` package. The 3 older repos (LysnrAI, MindLyst, ChronoMind) should be **migrated** to use `@bytelyst/datastore` with `DB_PROVIDER` support instead of direct `@azure/cosmos` calls.
|
||||||
|
|
||||||
|
**Migration needed:** LysnrAI, MindLyst, ChronoMind backends
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 6. `product-config.ts` — Backend product identity
|
||||||
|
|
||||||
|
**Status:** Present in 4/9 repos (FlowMonk, ActionTrail, NoteLett, NomGap), hardcoded in others
|
||||||
|
**Pattern:** Read `shared/product.json`, export `PRODUCT_ID`, `productConfig`
|
||||||
|
|
||||||
|
**Recommendation:** Already solved by `@bytelyst/config` (loadProductIdentity). The 5 repos without `product-config.ts` should adopt `@bytelyst/config/product-identity` instead of hardcoding product IDs.
|
||||||
|
|
||||||
|
**Migration needed:** LysnrAI, MindLyst, ChronoMind, JarvisJr, PeakPulse
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## MEDIUM Priority — Backend Patterns (2-3 repos)
|
||||||
|
|
||||||
|
### 7. `events.ts` — Domain event bus + SSE + webhook dispatch
|
||||||
|
|
||||||
|
**Status:** FlowMonk and ActionTrail have near-identical event bus infrastructure (~150 lines each)
|
||||||
|
**Pattern:** `@bytelyst/event-store` + `@bytelyst/fastify-sse` + `@bytelyst/webhook-dispatch` wired together with typed domain events + SSE hub + webhook targets
|
||||||
|
|
||||||
|
**Recommendation:** Create `@bytelyst/domain-events` factory:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
export function createDomainEventBus<TEvent extends BaseEvent>(opts: {
|
||||||
|
productId: string;
|
||||||
|
eventTypes: TEvent['type'][];
|
||||||
|
webhookSecret?: string;
|
||||||
|
}) { ... }
|
||||||
|
```
|
||||||
|
|
||||||
|
Product repos would only define their event type interfaces and call `createDomainEventBus()`.
|
||||||
|
|
||||||
|
**Affected repos:** FlowMonk, ActionTrail (and future products that need real-time events)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 8. `feature-flags.ts` — Backend in-memory flag registry
|
||||||
|
|
||||||
|
**Status:** FlowMonk and ActionTrail have **identical** implementations (25 lines each)
|
||||||
|
**Pattern:** `Map<string, boolean>` with `isFeatureEnabled()`, `getAllFlags()`, `setFlag()`
|
||||||
|
|
||||||
|
**Recommendation:** Create `@bytelyst/backend-flags`:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
export function createFlagRegistry(defaults: Record<string, boolean>) { ... }
|
||||||
|
```
|
||||||
|
|
||||||
|
**Affected repos:** FlowMonk, ActionTrail (expandable to all backends)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 9. `telemetry.ts` — Backend telemetry buffer
|
||||||
|
|
||||||
|
**Status:** FlowMonk and ActionTrail have **byte-for-byte identical** implementations (33 lines each)
|
||||||
|
**Pattern:** In-memory buffer with `trackEvent()`, `getBufferedEvents()`, `flushEvents()`
|
||||||
|
|
||||||
|
**Recommendation:** Add to existing `@bytelyst/events` or new `@bytelyst/backend-telemetry`:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
export function createTelemetryBuffer(opts: { enabled: boolean }) { ... }
|
||||||
|
```
|
||||||
|
|
||||||
|
**Affected repos:** FlowMonk, ActionTrail
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 10. `scheduler.ts` — In-process job scheduler
|
||||||
|
|
||||||
|
**Status:** ActionTrail has a full scheduler; FlowMonk has a scheduling engine
|
||||||
|
**Pattern:** Cron parsing, job registry, runner with diagnostics
|
||||||
|
|
||||||
|
**Recommendation:** ActionTrail's scheduler could generalize to `@bytelyst/cron-scheduler` for any product needing periodic jobs. Currently `@bytelyst/queue` exists but is a different pattern (durable job queue).
|
||||||
|
|
||||||
|
**Affected repos:** ActionTrail (FlowMonk's scheduler is domain-specific — planning, not cron)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## HIGH Priority — Web `lib/` Duplication (5+ repos)
|
||||||
|
|
||||||
|
### 11. `telemetry.ts` — Web telemetry init
|
||||||
|
|
||||||
|
**Status:** NomGap, NoteLett, ChronoMind, JarvisJr have near-identical wrappers (~35 lines each)
|
||||||
|
**Only difference:** `channel` name string
|
||||||
|
|
||||||
|
**Recommendation:** The `@bytelyst/telemetry-client` package already provides `createTelemetryClient()`. Products should call it directly in their `providers.tsx` instead of maintaining a wrapper file. Alternatively, add a `createWebTelemetry(productId, channel)` convenience to the package.
|
||||||
|
|
||||||
|
**Affected repos:** NomGap, NoteLett, ChronoMind, JarvisJr web apps
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 12. `diagnostics.ts` — Web diagnostics init
|
||||||
|
|
||||||
|
**Status:** NomGap, NoteLett, ChronoMind, JarvisJr have near-identical wrappers (~40 lines each)
|
||||||
|
**Only difference:** `channel` name, auth token retrieval method
|
||||||
|
|
||||||
|
**Recommendation:** Add `createWebDiagnostics(opts)` convenience to `@bytelyst/diagnostics-client` that handles install ID generation, localStorage auth token, and default config.
|
||||||
|
|
||||||
|
**Affected repos:** NomGap, NoteLett, ChronoMind, JarvisJr web apps
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 13. `feature-flags.ts` + `kill-switch.ts` — Web client wrappers
|
||||||
|
|
||||||
|
**Status:** NomGap and NoteLett have identical wrappers; ChronoMind has similar
|
||||||
|
|
||||||
|
**Recommendation:** These are thin wrappers around `@bytelyst/feature-flag-client` and `@bytelyst/kill-switch-client`. Consider adding React hook convenience exports directly to the packages.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 14. `product-config.ts` — Web product identity
|
||||||
|
|
||||||
|
**Status:** 5 different patterns across repos:
|
||||||
|
|
||||||
|
- NomGap/NoteLett: import `shared/product.json` + env var overrides
|
||||||
|
- FlowMonk/ActionTrail: inline object literal
|
||||||
|
- JarvisJr/ChronoMind: custom exports
|
||||||
|
|
||||||
|
**Recommendation:** Standardize on `shared/product.json` import pattern. Could add `@bytelyst/web-config` convenience that reads product.json and provides typed exports with NEXT*PUBLIC* env var overrides.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Migration Priority Matrix
|
||||||
|
|
||||||
|
| # | Candidate | LOC Saved | Repos | Effort | Priority |
|
||||||
|
| --- | -------------------------------------------------------- | --------- | ----- | -------- | -------- |
|
||||||
|
| 1 | `auth.ts` → `@bytelyst/fastify-auth` | ~700 | 9 | 2 days | **P0** |
|
||||||
|
| 2 | `request-context.ts` → merge into fastify-auth | ~350 | 9 | 1 day | **P0** |
|
||||||
|
| 3 | `errors.ts` → delete (import directly) | ~100 | 8 | 0.5 day | **P0** |
|
||||||
|
| 4 | `config.ts` → `@bytelyst/backend-config` | ~200 | 9 | 2 days | **P1** |
|
||||||
|
| 5 | Older repos → `@bytelyst/datastore` | ~300 | 3 | 1 day | **P1** |
|
||||||
|
| 6 | Hardcoded productId → `@bytelyst/config` | ~50 | 5 | 0.5 day | **P1** |
|
||||||
|
| 7 | `events.ts` → `@bytelyst/domain-events` | ~300 | 2 | 1.5 days | **P2** |
|
||||||
|
| 8 | `feature-flags.ts` → `@bytelyst/backend-flags` | ~50 | 2 | 0.5 day | **P2** |
|
||||||
|
| 9 | `telemetry.ts` → `@bytelyst/backend-telemetry` | ~70 | 2 | 0.5 day | **P2** |
|
||||||
|
| 10 | Web telemetry/diagnostics wrappers → package convenience | ~400 | 4 | 1 day | **P2** |
|
||||||
|
| 11 | Web `product-config.ts` → standardize pattern | ~200 | 5+ | 1 day | **P2** |
|
||||||
|
|
||||||
|
**Total estimated effort: ~11 days for full DRY migration**
|
||||||
|
**Total LOC eliminated: ~2,700+ duplicated lines**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Repos NOT Yet Using Common Platform
|
||||||
|
|
||||||
|
### `learning_ai_local_memory_gpt`
|
||||||
|
|
||||||
|
- Standalone Express + SQLite app — no Cosmos, no auth, no platform-service
|
||||||
|
- **No integration needed** — this is intentionally a standalone local tool
|
||||||
|
|
||||||
|
### `learning_ai_productivity_web`
|
||||||
|
|
||||||
|
- No backend directory found
|
||||||
|
- Needs investigation to determine if it should adopt platform patterns
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Recommended Execution Order
|
||||||
|
|
||||||
|
### Sprint 1 (P0 — 3.5 days)
|
||||||
|
|
||||||
|
1. Create `@bytelyst/fastify-auth` package with `extractAuth()`, `requireRole()`, `createRequestContext()`
|
||||||
|
2. Migrate all 9 backends to use it (delete local `auth.ts` + `request-context.ts`)
|
||||||
|
3. Delete all `errors.ts` re-export files, update imports to `@bytelyst/errors` directly
|
||||||
|
|
||||||
|
### Sprint 2 (P1 — 3.5 days)
|
||||||
|
|
||||||
|
4. Create `@bytelyst/backend-config` with composable Zod schema factory
|
||||||
|
5. Migrate LysnrAI, MindLyst, ChronoMind backends to `@bytelyst/datastore`
|
||||||
|
6. Migrate 5 repos to `@bytelyst/config/product-identity` for productId
|
||||||
|
|
||||||
|
### Sprint 3 (P2 — 4 days)
|
||||||
|
|
||||||
|
7. Create `@bytelyst/domain-events` factory (from FlowMonk/ActionTrail pattern)
|
||||||
|
8. Create `@bytelyst/backend-flags` and `@bytelyst/backend-telemetry`
|
||||||
|
9. Add convenience functions to web-side shared packages
|
||||||
|
10. Standardize web `product-config.ts` pattern
|
||||||
816
docs/roadmaps/CROSS_REPO_DRY_MIGRATION_ROADMAP.md
Normal file
816
docs/roadmaps/CROSS_REPO_DRY_MIGRATION_ROADMAP.md
Normal file
@ -0,0 +1,816 @@
|
|||||||
|
# Cross-Repo DRY Migration Roadmap
|
||||||
|
|
||||||
|
> **Status:** Planning
|
||||||
|
> **Created:** 2026-03-20
|
||||||
|
> **Audit:** See [`../CROSS_REPO_DRY_AUDIT.md`](../CROSS_REPO_DRY_AUDIT.md)
|
||||||
|
> **Total effort:** ~11 days across 5 phases
|
||||||
|
> **Total LOC eliminated:** ~4,200 duplicated lines across 9 product repos
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase Overview
|
||||||
|
|
||||||
|
| Phase | Name | New Packages | Repos Touched | Est. Effort | Status |
|
||||||
|
| ----- | --------------------------------------------------- | ------------ | ------------- | ----------- | ------ |
|
||||||
|
| **0** | Quick Wins (no new packages) | 0 | 8 | 1 day | ⬜ |
|
||||||
|
| **1** | `@bytelyst/fastify-auth` | 1 | 10 | 3 days | ⬜ |
|
||||||
|
| **2** | `@bytelyst/backend-config` + datastore migration | 1 | 12 | 3 days | ⬜ |
|
||||||
|
| **3** | Backend utilities (flags, telemetry, domain events) | 3 | 3 | 2 days | ⬜ |
|
||||||
|
| **4** | Web client DRY (telemetry, diagnostics, config) | 0 | 6 | 2 days | ⬜ |
|
||||||
|
|
||||||
|
**Repos involved (9 product backends):**
|
||||||
|
|
||||||
|
| Short Name | Repo | Backend Port | Backend Tests |
|
||||||
|
| ----------- | ----------------------------------- | ------------ | ------------- |
|
||||||
|
| LysnrAI | `learning_voice_ai_agent` | 4015 | 67 |
|
||||||
|
| MindLyst | `learning_multimodal_memory_agents` | 4014 | 63 |
|
||||||
|
| ChronoMind | `learning_ai_clock` | 4011 | 176 |
|
||||||
|
| JarvisJr | `learning_ai_jarvis_jr` | 4012 | 203 |
|
||||||
|
| NomGap | `learning_ai_fastgap` | 4013 | 203 |
|
||||||
|
| PeakPulse | `learning_ai_peakpulse` | 4010 | 59 |
|
||||||
|
| FlowMonk | `learning_ai_flowmonk` | 4017 | 181 |
|
||||||
|
| NoteLett | `learning_ai_notes` | 4016 | 80 |
|
||||||
|
| ActionTrail | `learning_ai_trails` | 4018 | 185 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 0 — Quick Wins (No New Packages)
|
||||||
|
|
||||||
|
> **Goal:** Delete dead re-export files, standardize product identity loading.
|
||||||
|
> **Effort:** ~1 day
|
||||||
|
> **Risk:** Low — mechanical changes only.
|
||||||
|
|
||||||
|
### 0.1 Delete `errors.ts` re-export files
|
||||||
|
|
||||||
|
These files exist in 8 repos and contain only re-exports from `@bytelyst/errors`. Every module that imports from `./errors.js` should import from `@bytelyst/errors` directly (FlowMonk already does this).
|
||||||
|
|
||||||
|
**Files to delete:**
|
||||||
|
|
||||||
|
| Repo | File |
|
||||||
|
| ----------- | --------------------------- |
|
||||||
|
| LysnrAI | `backend/src/lib/errors.ts` |
|
||||||
|
| MindLyst | `backend/src/lib/errors.ts` |
|
||||||
|
| ChronoMind | `backend/src/lib/errors.ts` |
|
||||||
|
| JarvisJr | `backend/src/lib/errors.ts` |
|
||||||
|
| NomGap | `backend/src/lib/errors.ts` |
|
||||||
|
| PeakPulse | `backend/src/lib/errors.ts` |
|
||||||
|
| NoteLett | `backend/src/lib/errors.ts` |
|
||||||
|
| ActionTrail | `backend/src/lib/errors.ts` |
|
||||||
|
|
||||||
|
**Steps per repo:**
|
||||||
|
|
||||||
|
1. `grep -rn "from './errors.js'" backend/src/` → list all importing files
|
||||||
|
2. Replace `from './errors.js'` with `from '@bytelyst/errors'` in each file
|
||||||
|
3. Also replace `from '../lib/errors.js'` and `from '../../lib/errors.js'` patterns
|
||||||
|
4. Delete `backend/src/lib/errors.ts`
|
||||||
|
5. Run: `cd backend && npm run typecheck && npm test`
|
||||||
|
|
||||||
|
**Commit:** `refactor(backend): remove errors.ts re-export, import @bytelyst/errors directly`
|
||||||
|
|
||||||
|
### 0.2 Add `product-config.ts` to repos that hardcode product IDs
|
||||||
|
|
||||||
|
Five repos hardcode `const PRODUCT_ID = '<name>'` in `request-context.ts` instead of reading from `shared/product.json`.
|
||||||
|
|
||||||
|
**Repos needing `product-config.ts`:**
|
||||||
|
|
||||||
|
| Repo | Hardcoded PRODUCT_ID | Port |
|
||||||
|
| ---------- | -------------------------------------- | ---- |
|
||||||
|
| LysnrAI | `'lysnrai'` in `request-context.ts` | 4015 |
|
||||||
|
| MindLyst | `'mindlyst'` in `request-context.ts` | 4014 |
|
||||||
|
| ChronoMind | `'chronomind'` in `request-context.ts` | 4011 |
|
||||||
|
| JarvisJr | `'jarvisjr'` in `request-context.ts` | 4012 |
|
||||||
|
| PeakPulse | `'peakpulse'` in `request-context.ts` | 4010 |
|
||||||
|
|
||||||
|
**Steps per repo:**
|
||||||
|
|
||||||
|
1. Verify `shared/product.json` exists with correct `productId`
|
||||||
|
2. Create `backend/src/lib/product-config.ts`:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
import { readFileSync } from 'node:fs';
|
||||||
|
import { resolve, dirname } from 'node:path';
|
||||||
|
import { fileURLToPath } from 'node:url';
|
||||||
|
|
||||||
|
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||||
|
const raw = JSON.parse(
|
||||||
|
readFileSync(resolve(__dirname, '../../../../shared/product.json'), 'utf8')
|
||||||
|
);
|
||||||
|
|
||||||
|
export const PRODUCT_ID: string = raw.productId;
|
||||||
|
export const productConfig = raw as {
|
||||||
|
productId: string;
|
||||||
|
displayName: string;
|
||||||
|
domain: string;
|
||||||
|
backendPort: number;
|
||||||
|
[key: string]: unknown;
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Update `request-context.ts`: replace `const PRODUCT_ID = '<hardcoded>'` with `import { PRODUCT_ID } from './product-config.js'`
|
||||||
|
4. Update `config.ts` if it hardcodes the port: use `productConfig.backendPort`
|
||||||
|
5. Run: `cd backend && npm run typecheck && npm test`
|
||||||
|
|
||||||
|
**Commit:** `refactor(backend): load product identity from shared/product.json`
|
||||||
|
|
||||||
|
### 0.3 Verification checkpoint
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Run ALL backend tests across all 9 repos
|
||||||
|
for repo in learning_voice_ai_agent learning_multimodal_memory_agents learning_ai_clock \
|
||||||
|
learning_ai_jarvis_jr learning_ai_fastgap learning_ai_peakpulse \
|
||||||
|
learning_ai_flowmonk learning_ai_notes learning_ai_trails; do
|
||||||
|
echo "=== $repo ===" && cd /Users/sd9235/code/mygh/$repo/backend && npm test 2>&1 | tail -3 && cd -
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
**Expected:** All 1,217 backend tests pass (unchanged count).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 1 — `@bytelyst/fastify-auth` Package
|
||||||
|
|
||||||
|
> **Goal:** Extract the duplicated JWT auth + request context into a single shared package. Delete `auth.ts` and `request-context.ts` from all 9 product repos.
|
||||||
|
> **Effort:** ~3 days
|
||||||
|
> **Risk:** Medium — auth is critical path. Must verify every repo's tests pass.
|
||||||
|
|
||||||
|
### 1.1 Create `packages/fastify-auth/`
|
||||||
|
|
||||||
|
**Location:** `learning_ai_common_plat/packages/fastify-auth/`
|
||||||
|
|
||||||
|
**Package API:**
|
||||||
|
|
||||||
|
```ts
|
||||||
|
// packages/fastify-auth/src/index.ts
|
||||||
|
|
||||||
|
export interface FastifyAuthConfig {
|
||||||
|
/** HS256 JWT secret (required) */
|
||||||
|
jwtSecret: string;
|
||||||
|
/** RS256 JWKS URL from platform-service (optional, tried first) */
|
||||||
|
jwksUrl?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface RequestContextConfig {
|
||||||
|
/** Product ID to enforce (e.g., 'nomgap') */
|
||||||
|
productId: string;
|
||||||
|
/** If true, returns 'demo-user' in non-production when no JWT present */
|
||||||
|
allowDemoUser?: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Re-export AuthPayload type from @bytelyst/auth
|
||||||
|
export type { AuthPayload } from '@bytelyst/auth';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Extract and verify JWT from Authorization: Bearer header.
|
||||||
|
* Tries RS256 via JWKS first, falls back to HS256.
|
||||||
|
*/
|
||||||
|
export function createExtractAuth(
|
||||||
|
config: FastifyAuthConfig
|
||||||
|
): (req: { headers: { authorization?: string } }) => Promise<AuthPayload>;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Require specific roles. Calls extractAuth first, then checks role.
|
||||||
|
*/
|
||||||
|
export function createRequireRole(
|
||||||
|
config: FastifyAuthConfig
|
||||||
|
): (req: { headers: { authorization?: string } }, ...roles: string[]) => Promise<AuthPayload>;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Create product-scoped request context helpers.
|
||||||
|
*/
|
||||||
|
export function createRequestContext(config: RequestContextConfig): {
|
||||||
|
getUserId(req: FastifyRequest): string;
|
||||||
|
getRequestProductId(req: FastifyRequest): string;
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
**Files to create:**
|
||||||
|
|
||||||
|
| File | Purpose |
|
||||||
|
| ------------------------------------------------------------- | --------------------------------------------------- |
|
||||||
|
| `packages/fastify-auth/package.json` | Package manifest (`@bytelyst/fastify-auth`) |
|
||||||
|
| `packages/fastify-auth/tsconfig.json` | Extends `../../tsconfig.base.json` |
|
||||||
|
| `packages/fastify-auth/src/index.ts` | Main exports |
|
||||||
|
| `packages/fastify-auth/src/extract-auth.ts` | JWT verification (RS256 + HS256 fallback) |
|
||||||
|
| `packages/fastify-auth/src/request-context.ts` | Product-scoped getUserId + getRequestProductId |
|
||||||
|
| `packages/fastify-auth/src/types.ts` | Fastify module augmentation (JwtPayload on request) |
|
||||||
|
| `packages/fastify-auth/src/__tests__/extract-auth.test.ts` | Unit tests |
|
||||||
|
| `packages/fastify-auth/src/__tests__/request-context.test.ts` | Unit tests |
|
||||||
|
|
||||||
|
**Dependencies:**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"dependencies": {
|
||||||
|
"@bytelyst/errors": "workspace:*"
|
||||||
|
},
|
||||||
|
"peerDependencies": {
|
||||||
|
"jose": ">=5.0.0",
|
||||||
|
"fastify": ">=5.0.0",
|
||||||
|
"@bytelyst/auth": "workspace:*"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Steps:**
|
||||||
|
|
||||||
|
1. Create package files (scaffold from existing `@bytelyst/auth` package structure)
|
||||||
|
2. Extract the canonical `extractAuth()` from any repo (they're identical)
|
||||||
|
3. Parameterize: accept `{ jwtSecret, jwksUrl }` instead of importing `config`
|
||||||
|
4. Extract `createRequestContext()` — parameterize with `{ productId, allowDemoUser }`
|
||||||
|
5. Add Fastify module augmentation for `req.jwtPayload` and `req.apiKeyUserId`
|
||||||
|
6. Write tests (port existing `auth.test.ts` from any repo)
|
||||||
|
7. Build: `pnpm --filter @bytelyst/fastify-auth build`
|
||||||
|
8. Run tests: `pnpm --filter @bytelyst/fastify-auth test`
|
||||||
|
|
||||||
|
**Commit:** `feat(fastify-auth): new package — extractAuth, requireRole, createRequestContext`
|
||||||
|
|
||||||
|
### 1.2 Migrate all 9 product backends
|
||||||
|
|
||||||
|
**For each repo, the migration is identical:**
|
||||||
|
|
||||||
|
**Files to modify:**
|
||||||
|
|
||||||
|
| Action | File | Change |
|
||||||
|
| ----------- | ------------------------------------ | -------------------------------------------------------------------------------------------- |
|
||||||
|
| **Add dep** | `backend/package.json` | Add `"@bytelyst/fastify-auth": "file:../../learning_ai_common_plat/packages/fastify-auth"` |
|
||||||
|
| **Rewrite** | `backend/src/lib/auth.ts` | Replace 60-90 lines with ~10-line wrapper that calls `createExtractAuth()` with local config |
|
||||||
|
| **Rewrite** | `backend/src/lib/request-context.ts` | Replace 30-50 lines with ~5-line wrapper that calls `createRequestContext()` |
|
||||||
|
| **Verify** | — | `cd backend && npm install && npm run typecheck && npm test` |
|
||||||
|
|
||||||
|
**New `auth.ts` (per repo, ~10 lines):**
|
||||||
|
|
||||||
|
```ts
|
||||||
|
import { createExtractAuth, createRequireRole } from '@bytelyst/fastify-auth';
|
||||||
|
import { config } from './config.js';
|
||||||
|
export type { AuthPayload } from '@bytelyst/fastify-auth';
|
||||||
|
|
||||||
|
export const extractAuth = createExtractAuth({
|
||||||
|
jwtSecret: config.JWT_SECRET,
|
||||||
|
jwksUrl: config.PLATFORM_JWKS_URL,
|
||||||
|
});
|
||||||
|
|
||||||
|
export const requireRole = createRequireRole({
|
||||||
|
jwtSecret: config.JWT_SECRET,
|
||||||
|
jwksUrl: config.PLATFORM_JWKS_URL,
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
**New `request-context.ts` (per repo, ~10 lines):**
|
||||||
|
|
||||||
|
```ts
|
||||||
|
import { createRequestContext } from '@bytelyst/fastify-auth';
|
||||||
|
import { PRODUCT_ID } from './product-config.js';
|
||||||
|
export type { JwtPayload } from '@bytelyst/fastify-auth';
|
||||||
|
|
||||||
|
const ctx = createRequestContext({ productId: PRODUCT_ID, allowDemoUser: true });
|
||||||
|
export const getUserId = ctx.getUserId;
|
||||||
|
export const getRequestProductId = ctx.getRequestProductId;
|
||||||
|
```
|
||||||
|
|
||||||
|
**Migration order (by test count — smallest first for safe iteration):**
|
||||||
|
|
||||||
|
| Order | Repo | Tests | Notes |
|
||||||
|
| ----- | ----------- | ----- | ------------------------------------------------------------ |
|
||||||
|
| 1 | PeakPulse | 59 | Simplest, good canary |
|
||||||
|
| 2 | MindLyst | 63 | |
|
||||||
|
| 3 | LysnrAI | 67 | Has `audit.ts` extra import |
|
||||||
|
| 4 | NoteLett | 80 | |
|
||||||
|
| 5 | ChronoMind | 176 | |
|
||||||
|
| 6 | FlowMonk | 181 | Already imports `@bytelyst/errors` directly |
|
||||||
|
| 7 | ActionTrail | 185 | Has `apiKeyUserId` on request — include in type augmentation |
|
||||||
|
| 8 | JarvisJr | 203 | |
|
||||||
|
| 9 | NomGap | 203 | |
|
||||||
|
|
||||||
|
**Per-repo commit:** `refactor(backend): migrate auth to @bytelyst/fastify-auth`
|
||||||
|
|
||||||
|
### 1.3 Delete `auth.test.ts` from repos that test shared logic
|
||||||
|
|
||||||
|
After migration, the `auth.test.ts` files in each repo test the same JWT logic that's now in the shared package. Keep only product-specific auth behavior tests; delete the ones testing `extractAuth()` and `requireRole()` directly (those now live in `packages/fastify-auth`).
|
||||||
|
|
||||||
|
**Files to evaluate (keep if product-specific, delete if duplicate):**
|
||||||
|
|
||||||
|
| Repo | File | Expected Action |
|
||||||
|
| ----- | ------------------------------ | ----------------------------------------------------------- |
|
||||||
|
| All 9 | `backend/src/lib/auth.test.ts` | Review — likely delete (logic now tested in shared package) |
|
||||||
|
|
||||||
|
### 1.4 Verification checkpoint
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Build shared package
|
||||||
|
cd /Users/sd9235/code/mygh/learning_ai_common_plat && pnpm --filter @bytelyst/fastify-auth build && pnpm --filter @bytelyst/fastify-auth test
|
||||||
|
|
||||||
|
# Run ALL backend tests
|
||||||
|
for repo in learning_voice_ai_agent learning_multimodal_memory_agents learning_ai_clock \
|
||||||
|
learning_ai_jarvis_jr learning_ai_fastgap learning_ai_peakpulse \
|
||||||
|
learning_ai_flowmonk learning_ai_notes learning_ai_trails; do
|
||||||
|
echo "=== $repo ===" && cd /Users/sd9235/code/mygh/$repo/backend && npm install && npm run typecheck && npm test 2>&1 | tail -3 && cd -
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
**Expected:** All 1,217+ backend tests pass. Shared package has its own test suite (~15-20 tests).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 2 — `@bytelyst/backend-config` + Datastore Migration
|
||||||
|
|
||||||
|
> **Goal:** Extract the duplicated Zod env config schema into a composable factory. Migrate 3 older repos to `@bytelyst/datastore`.
|
||||||
|
> **Effort:** ~3 days
|
||||||
|
> **Risk:** Medium — config parsing is startup-critical.
|
||||||
|
|
||||||
|
### 2.1 Create `packages/backend-config/`
|
||||||
|
|
||||||
|
**Location:** `learning_ai_common_plat/packages/backend-config/`
|
||||||
|
|
||||||
|
**Package API:**
|
||||||
|
|
||||||
|
```ts
|
||||||
|
// packages/backend-config/src/index.ts
|
||||||
|
import { z } from 'zod';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Base schema shared across ALL product backends.
|
||||||
|
* Includes PORT, HOST, NODE_ENV, CORS_ORIGIN, SERVICE_NAME,
|
||||||
|
* DB_PROVIDER, COSMOS_*, JWT_SECRET, PLATFORM_JWKS_URL.
|
||||||
|
*/
|
||||||
|
export const baseBackendSchema = z.object({
|
||||||
|
PORT: z.coerce.number(),
|
||||||
|
HOST: z.string().default('0.0.0.0'),
|
||||||
|
NODE_ENV: z.enum(['development', 'production', 'test']).default('development'),
|
||||||
|
CORS_ORIGIN: z.string().optional(),
|
||||||
|
SERVICE_NAME: z.string(),
|
||||||
|
DB_PROVIDER: z.enum(['cosmos', 'memory']).default('cosmos'),
|
||||||
|
COSMOS_ENDPOINT: z.string().default(''),
|
||||||
|
COSMOS_KEY: z.string().default(''),
|
||||||
|
COSMOS_DATABASE: z.string().default('lysnrai'),
|
||||||
|
JWT_SECRET: z.string().default('dev-secret-do-not-use-in-prod'),
|
||||||
|
PLATFORM_JWKS_URL: z.string().url().optional(),
|
||||||
|
PLATFORM_SERVICE_URL: z.string().default('http://localhost:4003'),
|
||||||
|
});
|
||||||
|
|
||||||
|
export type BaseBackendConfig = z.infer<typeof baseBackendSchema>;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Create a product-specific config by extending the base schema.
|
||||||
|
*/
|
||||||
|
export function createBackendConfig<T extends z.ZodRawShape>(opts: {
|
||||||
|
serviceName: string;
|
||||||
|
defaultPort: number;
|
||||||
|
defaultDatabase?: string;
|
||||||
|
extraSchema?: z.ZodObject<T>;
|
||||||
|
}): BaseBackendConfig & z.infer<z.ZodObject<T>>;
|
||||||
|
```
|
||||||
|
|
||||||
|
**Files to create:**
|
||||||
|
|
||||||
|
| File | Purpose |
|
||||||
|
| ------------------------------------------------------ | --------------------------------------------- |
|
||||||
|
| `packages/backend-config/package.json` | Package manifest |
|
||||||
|
| `packages/backend-config/tsconfig.json` | TS config |
|
||||||
|
| `packages/backend-config/src/index.ts` | Base schema + `createBackendConfig()` factory |
|
||||||
|
| `packages/backend-config/src/__tests__/config.test.ts` | Tests |
|
||||||
|
|
||||||
|
**Steps:**
|
||||||
|
|
||||||
|
1. Create package scaffold
|
||||||
|
2. Extract common schema fields from the 9 existing `config.ts` files
|
||||||
|
3. Build the `createBackendConfig()` factory that merges base + extra schemas
|
||||||
|
4. Handle the Cosmos credential validation (required when `DB_PROVIDER=cosmos`)
|
||||||
|
5. Write tests
|
||||||
|
6. Build + test
|
||||||
|
|
||||||
|
**Commit:** `feat(backend-config): new package — composable Zod env schema factory`
|
||||||
|
|
||||||
|
### 2.2 Migrate all 9 product backends to `@bytelyst/backend-config`
|
||||||
|
|
||||||
|
**New `config.ts` per repo (example: NomGap):**
|
||||||
|
|
||||||
|
```ts
|
||||||
|
import { z } from 'zod';
|
||||||
|
import { createBackendConfig } from '@bytelyst/backend-config';
|
||||||
|
|
||||||
|
export const config = createBackendConfig({
|
||||||
|
serviceName: 'nomgap-backend',
|
||||||
|
defaultPort: 4013,
|
||||||
|
extraSchema: z.object({
|
||||||
|
CORS_ORIGIN: z.string().default('http://localhost:3040'),
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
**Migration order:** Same as Phase 1 (smallest test count first).
|
||||||
|
|
||||||
|
**Per-repo steps:**
|
||||||
|
|
||||||
|
1. Add `"@bytelyst/backend-config": "file:../../learning_ai_common_plat/packages/backend-config"` to `package.json`
|
||||||
|
2. Rewrite `config.ts` to use `createBackendConfig()`
|
||||||
|
3. Ensure all modules still find the config fields they need
|
||||||
|
4. `cd backend && npm install && npm run typecheck && npm test`
|
||||||
|
|
||||||
|
**Per-repo commit:** `refactor(backend): migrate config to @bytelyst/backend-config`
|
||||||
|
|
||||||
|
### 2.3 Migrate LysnrAI, MindLyst, ChronoMind to `@bytelyst/datastore`
|
||||||
|
|
||||||
|
These 3 repos use direct `@azure/cosmos` calls with custom `cosmos-init.ts` files instead of the `@bytelyst/datastore` abstraction that the 6 newer repos use.
|
||||||
|
|
||||||
|
**Repos to migrate:**
|
||||||
|
|
||||||
|
| Repo | Current Pattern | Target |
|
||||||
|
| ---------- | ----------------------------------------- | ------------------------------------- |
|
||||||
|
| LysnrAI | Direct `@azure/cosmos` + `cosmos-init.ts` | `@bytelyst/datastore` + `DB_PROVIDER` |
|
||||||
|
| MindLyst | Direct `@azure/cosmos` + `cosmos-init.ts` | `@bytelyst/datastore` + `DB_PROVIDER` |
|
||||||
|
| ChronoMind | Direct `@azure/cosmos` + `cosmos-init.ts` | `@bytelyst/datastore` + `DB_PROVIDER` |
|
||||||
|
|
||||||
|
**Steps per repo:**
|
||||||
|
|
||||||
|
1. Add `"@bytelyst/datastore": "file:../../learning_ai_common_plat/packages/datastore"` to `package.json`
|
||||||
|
2. Add `DB_PROVIDER` to config schema (default: `'cosmos'`)
|
||||||
|
3. Create/update `backend/src/lib/datastore.ts` matching the 6 newer repos' pattern:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
import {
|
||||||
|
CosmosDatastoreProvider,
|
||||||
|
MemoryDatastoreProvider,
|
||||||
|
type DatastoreProvider,
|
||||||
|
} from '@bytelyst/datastore';
|
||||||
|
import { config } from './config.js';
|
||||||
|
|
||||||
|
let provider: DatastoreProvider;
|
||||||
|
|
||||||
|
export function getDatastoreProvider(): DatastoreProvider {
|
||||||
|
if (!provider) {
|
||||||
|
provider =
|
||||||
|
config.DB_PROVIDER === 'memory'
|
||||||
|
? new MemoryDatastoreProvider()
|
||||||
|
: new CosmosDatastoreProvider({
|
||||||
|
endpoint: config.COSMOS_ENDPOINT,
|
||||||
|
key: config.COSMOS_KEY,
|
||||||
|
database: config.COSMOS_DATABASE,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return provider;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function getCollection(name: string) {
|
||||||
|
return getDatastoreProvider().getCollection(name);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
4. Update each repository file to use `getCollection()` instead of direct Cosmos container calls
|
||||||
|
5. Delete `cosmos-init.ts` if fully replaced
|
||||||
|
6. Add `DB_PROVIDER=memory` to test env for zero-dependency test runs
|
||||||
|
7. `cd backend && npm install && npm run typecheck && npm test`
|
||||||
|
|
||||||
|
**Per-repo commit:** `refactor(backend): migrate to @bytelyst/datastore with DB_PROVIDER`
|
||||||
|
|
||||||
|
### 2.4 Verification checkpoint
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Build shared packages
|
||||||
|
cd /Users/sd9235/code/mygh/learning_ai_common_plat
|
||||||
|
pnpm --filter @bytelyst/backend-config build && pnpm --filter @bytelyst/backend-config test
|
||||||
|
|
||||||
|
# Run ALL backend tests
|
||||||
|
for repo in learning_voice_ai_agent learning_multimodal_memory_agents learning_ai_clock \
|
||||||
|
learning_ai_jarvis_jr learning_ai_fastgap learning_ai_peakpulse \
|
||||||
|
learning_ai_flowmonk learning_ai_notes learning_ai_trails; do
|
||||||
|
echo "=== $repo ===" && cd /Users/sd9235/code/mygh/$repo/backend && npm install && npm run typecheck && npm test 2>&1 | tail -3 && cd -
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
**Expected:** All 1,217+ backend tests pass. 3 repos now support `DB_PROVIDER=memory` for offline testing.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 3 — Backend Utilities (Flags, Telemetry, Domain Events)
|
||||||
|
|
||||||
|
> **Goal:** Extract the 3 remaining duplicated backend patterns from FlowMonk + ActionTrail.
|
||||||
|
> **Effort:** ~2 days
|
||||||
|
> **Risk:** Low — only 2 repos affected, patterns are small.
|
||||||
|
|
||||||
|
### 3.1 Create `packages/backend-flags/`
|
||||||
|
|
||||||
|
**Source:** Identical code in FlowMonk + ActionTrail `backend/src/lib/feature-flags.ts`
|
||||||
|
|
||||||
|
**Package API:**
|
||||||
|
|
||||||
|
```ts
|
||||||
|
export interface FlagRegistry {
|
||||||
|
isFeatureEnabled(flag: string, userId?: string): boolean;
|
||||||
|
getAllFlags(): Record<string, boolean>;
|
||||||
|
setFlag(flag: string, value: boolean): void;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function createFlagRegistry(defaults: Record<string, boolean>): FlagRegistry;
|
||||||
|
```
|
||||||
|
|
||||||
|
**Files to create:** `package.json`, `tsconfig.json`, `src/index.ts`, `src/__tests__/flags.test.ts`
|
||||||
|
|
||||||
|
**Migration:**
|
||||||
|
|
||||||
|
- FlowMonk: rewrite `feature-flags.ts` to `import { createFlagRegistry } from '@bytelyst/backend-flags'`
|
||||||
|
- ActionTrail: same
|
||||||
|
|
||||||
|
**Commit:** `feat(backend-flags): new package — in-memory feature flag registry`
|
||||||
|
|
||||||
|
### 3.2 Create `packages/backend-telemetry/`
|
||||||
|
|
||||||
|
**Source:** Byte-for-byte identical code in FlowMonk + ActionTrail `backend/src/lib/telemetry.ts`
|
||||||
|
|
||||||
|
**Package API:**
|
||||||
|
|
||||||
|
```ts
|
||||||
|
export interface TelemetryBuffer {
|
||||||
|
trackEvent(event: string, userId?: string, properties?: Record<string, unknown>): void;
|
||||||
|
getBufferedEvents(): TelemetryEvent[];
|
||||||
|
flushEvents(): TelemetryEvent[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export function createTelemetryBuffer(opts: { enabled: boolean }): TelemetryBuffer;
|
||||||
|
```
|
||||||
|
|
||||||
|
**Files to create:** `package.json`, `tsconfig.json`, `src/index.ts`, `src/__tests__/telemetry.test.ts`
|
||||||
|
|
||||||
|
**Migration:**
|
||||||
|
|
||||||
|
- FlowMonk: rewrite `telemetry.ts` to import from `@bytelyst/backend-telemetry`
|
||||||
|
- ActionTrail: same
|
||||||
|
|
||||||
|
**Commit:** `feat(backend-telemetry): new package — server-side telemetry event buffer`
|
||||||
|
|
||||||
|
### 3.3 Create `packages/domain-events/`
|
||||||
|
|
||||||
|
**Source:** Near-identical event bus infrastructure in FlowMonk + ActionTrail (~150 lines each)
|
||||||
|
|
||||||
|
**Package API:**
|
||||||
|
|
||||||
|
```ts
|
||||||
|
export interface DomainEventBusConfig {
|
||||||
|
productId: string;
|
||||||
|
webhookSecret?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface DomainEventBus<TEvent extends { type: string }> {
|
||||||
|
emit(event: TEvent): void;
|
||||||
|
on(type: TEvent['type'], handler: (event: TEvent) => void): void;
|
||||||
|
getSSEHub(): SSEHub;
|
||||||
|
getEventStore(): EventStore;
|
||||||
|
registerWebhookTargets(targets: WebhookTarget[]): void;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function createDomainEventBus<TEvent extends { type: string }>(
|
||||||
|
config: DomainEventBusConfig
|
||||||
|
): DomainEventBus<TEvent>;
|
||||||
|
```
|
||||||
|
|
||||||
|
**Product repos would only define their event type interfaces and call `createDomainEventBus()` — eliminating ~120 lines of wiring boilerplate per repo.**
|
||||||
|
|
||||||
|
**Files to create:** `package.json`, `tsconfig.json`, `src/index.ts`, `src/bus.ts`, `src/__tests__/bus.test.ts`
|
||||||
|
|
||||||
|
**Dependencies:**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"dependencies": {
|
||||||
|
"@bytelyst/event-store": "workspace:*",
|
||||||
|
"@bytelyst/fastify-sse": "workspace:*",
|
||||||
|
"@bytelyst/webhook-dispatch": "workspace:*"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Migration:**
|
||||||
|
|
||||||
|
- FlowMonk: replace 150-line `events.ts` with ~30-line type definitions + `createDomainEventBus()`
|
||||||
|
- ActionTrail: same
|
||||||
|
|
||||||
|
**Commit:** `feat(domain-events): new package — typed event bus + SSE + webhook wiring`
|
||||||
|
|
||||||
|
### 3.4 Verification checkpoint
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /Users/sd9235/code/mygh/learning_ai_common_plat
|
||||||
|
pnpm --filter @bytelyst/backend-flags build && pnpm --filter @bytelyst/backend-flags test
|
||||||
|
pnpm --filter @bytelyst/backend-telemetry build && pnpm --filter @bytelyst/backend-telemetry test
|
||||||
|
pnpm --filter @bytelyst/domain-events build && pnpm --filter @bytelyst/domain-events test
|
||||||
|
|
||||||
|
# Test affected repos
|
||||||
|
for repo in learning_ai_flowmonk learning_ai_trails; do
|
||||||
|
echo "=== $repo ===" && cd /Users/sd9235/code/mygh/$repo/backend && npm install && npm run typecheck && npm test 2>&1 | tail -3 && cd -
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
**Expected:** FlowMonk (181 tests) + ActionTrail (185 tests) still pass. 3 new packages with their own test suites.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 4 — Web Client DRY
|
||||||
|
|
||||||
|
> **Goal:** Reduce duplication in web app `lib/` directories. Add convenience functions to existing packages rather than creating new ones.
|
||||||
|
> **Effort:** ~2 days
|
||||||
|
> **Risk:** Low — web wrappers are thin, changes are cosmetic.
|
||||||
|
|
||||||
|
### 4.1 Add `createWebTelemetry()` to `@bytelyst/telemetry-client`
|
||||||
|
|
||||||
|
**Current state:** NomGap, NoteLett, ChronoMind, JarvisJr each have ~35-line `telemetry.ts` that wraps `createTelemetryClient()` with identical boilerplate.
|
||||||
|
|
||||||
|
**Add to `packages/telemetry-client/src/index.ts`:**
|
||||||
|
|
||||||
|
```ts
|
||||||
|
export function createWebTelemetry(opts: {
|
||||||
|
productId: string;
|
||||||
|
channel: string;
|
||||||
|
baseUrl?: string;
|
||||||
|
transport?: 'fetch' | 'beacon';
|
||||||
|
}) {
|
||||||
|
let initialized = false;
|
||||||
|
const client = createTelemetryClient({
|
||||||
|
productId: opts.productId,
|
||||||
|
baseUrl: opts.baseUrl ?? 'http://localhost:4003/api',
|
||||||
|
endpoint: '/telemetry/events',
|
||||||
|
platform: 'web',
|
||||||
|
channel: opts.channel,
|
||||||
|
transport: opts.transport ?? 'fetch',
|
||||||
|
appVersion: '0.1.0',
|
||||||
|
buildNumber: '1',
|
||||||
|
releaseChannel: 'dev',
|
||||||
|
osFamily: 'other',
|
||||||
|
});
|
||||||
|
|
||||||
|
return {
|
||||||
|
client,
|
||||||
|
init() {
|
||||||
|
if (initialized) return client;
|
||||||
|
client.init();
|
||||||
|
client.trackEvent('info', 'app_shell', 'web_app_initialized');
|
||||||
|
initialized = true;
|
||||||
|
return client;
|
||||||
|
},
|
||||||
|
trackPageView(page: string) {
|
||||||
|
client.trackEvent('info', 'navigation', 'page_view', { feature: page });
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Migration per web app:**
|
||||||
|
|
||||||
|
- Replace ~35-line `telemetry.ts` with ~5-line wrapper:
|
||||||
|
```ts
|
||||||
|
'use client';
|
||||||
|
import { createWebTelemetry } from '@bytelyst/telemetry-client';
|
||||||
|
const { client, init, trackPageView } = createWebTelemetry({
|
||||||
|
productId: 'nomgap',
|
||||||
|
channel: 'nomgap_web',
|
||||||
|
});
|
||||||
|
export { client as telemetryClient, init as initTelemetry, trackPageView };
|
||||||
|
```
|
||||||
|
|
||||||
|
**Repos:** NomGap, NoteLett, ChronoMind, JarvisJr
|
||||||
|
|
||||||
|
**Commit:** `feat(telemetry-client): add createWebTelemetry() convenience`
|
||||||
|
|
||||||
|
### 4.2 Add `createWebDiagnostics()` to `@bytelyst/diagnostics-client`
|
||||||
|
|
||||||
|
**Current state:** NomGap, NoteLett, ChronoMind, JarvisJr each have ~40-line `diagnostics.ts` with identical DiagnosticsClient init + install ID generation.
|
||||||
|
|
||||||
|
**Add to `packages/diagnostics-client/`:**
|
||||||
|
|
||||||
|
```ts
|
||||||
|
export function createWebDiagnostics(opts: {
|
||||||
|
productId: string;
|
||||||
|
channel: string;
|
||||||
|
serverUrl: string;
|
||||||
|
getAuthToken: () => string;
|
||||||
|
}): { init(): void; stop(): void };
|
||||||
|
```
|
||||||
|
|
||||||
|
**Handles:** install ID generation, localStorage, default config, error swallowing.
|
||||||
|
|
||||||
|
**Migration per web app:** Replace ~40-line file with ~8-line wrapper.
|
||||||
|
|
||||||
|
**Repos:** NomGap, NoteLett, ChronoMind, JarvisJr
|
||||||
|
|
||||||
|
**Commit:** `feat(diagnostics-client): add createWebDiagnostics() convenience`
|
||||||
|
|
||||||
|
### 4.3 Standardize web `product-config.ts` pattern
|
||||||
|
|
||||||
|
**Current state:** 5 different patterns across repos.
|
||||||
|
|
||||||
|
**Target pattern (already used by NomGap + NoteLett):**
|
||||||
|
|
||||||
|
```ts
|
||||||
|
import productIdentity from '../../../shared/product.json';
|
||||||
|
|
||||||
|
export const PRODUCT_NAME = process.env.NEXT_PUBLIC_PRODUCT_NAME ?? productIdentity.displayName;
|
||||||
|
export const PRODUCT_ID = process.env.NEXT_PUBLIC_PRODUCT_ID ?? productIdentity.productId;
|
||||||
|
export const PLATFORM_SERVICE_URL =
|
||||||
|
process.env.NEXT_PUBLIC_PLATFORM_SERVICE_URL ?? 'http://localhost:4003/api';
|
||||||
|
export const PLATFORM_SERVICE_ORIGIN = PLATFORM_SERVICE_URL.replace(/\/api\/?$/, '');
|
||||||
|
export const BACKEND_API_URL =
|
||||||
|
process.env.NEXT_PUBLIC_BACKEND_API_URL ?? `http://localhost:${productIdentity.backendPort}/api`;
|
||||||
|
export const DIAGNOSTICS_URL = process.env.NEXT_PUBLIC_DIAGNOSTICS_URL ?? PLATFORM_SERVICE_ORIGIN;
|
||||||
|
export const TELEMETRY_TRANSPORT =
|
||||||
|
process.env.NEXT_PUBLIC_TELEMETRY_TRANSPORT === 'beacon' ? 'beacon' : 'fetch';
|
||||||
|
```
|
||||||
|
|
||||||
|
**Repos to standardize:**
|
||||||
|
|
||||||
|
- FlowMonk web: uses inline object literal → adopt `shared/product.json` import
|
||||||
|
- ActionTrail web: same
|
||||||
|
- JarvisJr web: uses custom `getPlatformBaseURL()` → standardize
|
||||||
|
- ChronoMind web: uses `auth-api.ts` for product config → separate into `product-config.ts`
|
||||||
|
|
||||||
|
**Per-repo commit:** `refactor(web): standardize product-config.ts to shared/product.json pattern`
|
||||||
|
|
||||||
|
### 4.4 Verification checkpoint
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Build updated packages
|
||||||
|
cd /Users/sd9235/code/mygh/learning_ai_common_plat
|
||||||
|
pnpm --filter @bytelyst/telemetry-client build
|
||||||
|
pnpm --filter @bytelyst/diagnostics-client build
|
||||||
|
|
||||||
|
# Typecheck all web apps
|
||||||
|
for repo in learning_ai_fastgap/web learning_ai_notes/web learning_ai_clock/web \
|
||||||
|
learning_ai_jarvis_jr/web learning_ai_flowmonk/web learning_ai_trails/web; do
|
||||||
|
echo "=== $repo ===" && cd /Users/sd9235/code/mygh/$repo && npx tsc --noEmit 2>&1 | tail -3 && cd -
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Commit & Push Strategy
|
||||||
|
|
||||||
|
Each phase should be committed and pushed independently:
|
||||||
|
|
||||||
|
| Phase | Commits | Push Strategy |
|
||||||
|
| ----- | ------------------------------------------------------------------ | ------------------------------------------ |
|
||||||
|
| **0** | 2 commits per repo (errors cleanup + product-config) | Push all repos after phase complete |
|
||||||
|
| **1** | 1 commit in common-plat (new package) + 1 commit per product repo | Push common-plat first, then product repos |
|
||||||
|
| **2** | 1 commit in common-plat + 1-2 commits per product repo | Push common-plat first, then product repos |
|
||||||
|
| **3** | 3 commits in common-plat + 1 commit each in FlowMonk + ActionTrail | Push common-plat first |
|
||||||
|
| **4** | 2 commits in common-plat + 1 commit per web app | Push common-plat first |
|
||||||
|
|
||||||
|
**Total commits: ~35-40 across all repos**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Rollback Plan
|
||||||
|
|
||||||
|
Every migration follows the same rollback strategy:
|
||||||
|
|
||||||
|
1. **Product repos reference shared packages via `file:` refs** — no npm publish needed
|
||||||
|
2. **Each phase is independently revertable** — revert the product repo commit and the old `lib/` files return
|
||||||
|
3. **Shared packages are additive** — creating `@bytelyst/fastify-auth` doesn't break anything that doesn't use it
|
||||||
|
4. **Test gate:** Every phase ends with a verification checkpoint. If any repo's tests fail, stop and fix before continuing.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Final State (After All Phases)
|
||||||
|
|
||||||
|
### Backend `lib/` directory (target state per product repo)
|
||||||
|
|
||||||
|
| File | Source | Lines |
|
||||||
|
| -------------------- | ------------------------------------------------------- | ----- |
|
||||||
|
| `config.ts` | `createBackendConfig()` from `@bytelyst/backend-config` | ~8-15 |
|
||||||
|
| `auth.ts` | `createExtractAuth()` from `@bytelyst/fastify-auth` | ~10 |
|
||||||
|
| `request-context.ts` | `createRequestContext()` from `@bytelyst/fastify-auth` | ~6 |
|
||||||
|
| `product-config.ts` | Reads `shared/product.json` | ~12 |
|
||||||
|
| `datastore.ts` | `@bytelyst/datastore` with `DB_PROVIDER` | ~15 |
|
||||||
|
| ~~`errors.ts`~~ | **DELETED** — import `@bytelyst/errors` directly | 0 |
|
||||||
|
|
||||||
|
**Average backend `lib/` shrinks from ~250 lines to ~60 lines (76% reduction).**
|
||||||
|
|
||||||
|
### New packages in common-plat
|
||||||
|
|
||||||
|
| Package | Phase | Tests | Purpose |
|
||||||
|
| ----------------------------- | ----- | ----- | ----------------------------------------------- |
|
||||||
|
| `@bytelyst/fastify-auth` | 1 | ~20 | JWT auth + request context for Fastify backends |
|
||||||
|
| `@bytelyst/backend-config` | 2 | ~10 | Composable Zod env config factory |
|
||||||
|
| `@bytelyst/backend-flags` | 3 | ~8 | In-memory feature flag registry |
|
||||||
|
| `@bytelyst/backend-telemetry` | 3 | ~8 | Server-side telemetry buffer |
|
||||||
|
| `@bytelyst/domain-events` | 3 | ~12 | Event bus + SSE + webhook factory |
|
||||||
|
|
||||||
|
**Total new shared tests: ~58**
|
||||||
|
|
||||||
|
### Updated packages in common-plat
|
||||||
|
|
||||||
|
| Package | Phase | Change |
|
||||||
|
| ------------------------------ | ----- | ------------------------------------ |
|
||||||
|
| `@bytelyst/telemetry-client` | 4 | `createWebTelemetry()` convenience |
|
||||||
|
| `@bytelyst/diagnostics-client` | 4 | `createWebDiagnostics()` convenience |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Success Metrics
|
||||||
|
|
||||||
|
| Metric | Before | After |
|
||||||
|
| ------------------------------------------- | ------ | ----------------------------------- |
|
||||||
|
| Duplicated backend `lib/` LOC | ~2,400 | ~540 (thin wrappers) |
|
||||||
|
| Duplicated web `lib/` LOC | ~1,800 | ~400 (thin wrappers) |
|
||||||
|
| Total duplicated LOC | ~4,200 | ~940 |
|
||||||
|
| **LOC eliminated** | — | **~3,260** |
|
||||||
|
| Shared packages | 53 | 58 |
|
||||||
|
| Repos with `DB_PROVIDER` support | 6 | 9 |
|
||||||
|
| Repos with standardized `product-config.ts` | 4 | 9 |
|
||||||
|
| Backend test count | 1,217 | 1,217+ (unchanged + ~58 new shared) |
|
||||||
Loading…
Reference in New Issue
Block a user