diff --git a/__LOCAL_LLMs/AI_IDE_CHAT_HISTORY/WINDSURF/.last-refresh.log b/__LOCAL_LLMs/AI_IDE_CHAT_HISTORY/WINDSURF/.last-refresh.log index 2cd6c6fb..4d1ea674 100644 --- a/__LOCAL_LLMs/AI_IDE_CHAT_HISTORY/WINDSURF/.last-refresh.log +++ b/__LOCAL_LLMs/AI_IDE_CHAT_HISTORY/WINDSURF/.last-refresh.log @@ -1,9 +1,9 @@ -Last refresh: 2026-03-21T21:44:43Z (2026-03-21 14:44:44 PDT) -Cascade conversations: 50 (387M) -Memories: 95 +Last refresh: 2026-03-22T18:43:11Z (2026-03-22 11:43:11 PDT) +Cascade conversations: 50 (417M) +Memories: 98 Implicit context: 20 -Code tracker dirs: 89 -File edit history: 3457 entries -Workspace storage: 36 workspaces +Code tracker dirs: 139 +File edit history: 3547 entries +Workspace storage: 34 workspaces Repo docs: 7 files across 2 repos Repo workflows: 43 files across 10 repos diff --git a/docs/devops/SINGLE_VM_DEPLOYMENT.md b/docs/devops/SINGLE_VM_DEPLOYMENT.md index 7f50fcd1..bcc584fd 100644 --- a/docs/devops/SINGLE_VM_DEPLOYMENT.md +++ b/docs/devops/SINGLE_VM_DEPLOYMENT.md @@ -4,6 +4,25 @@ --- +## Package-Manager Strategy (current transition plan) + +- `learning_ai_common_plat` is already the canonical **`pnpm` workspace** monorepo for shared packages, services, and dashboards. +- Node/TypeScript product repos are moving toward **`pnpm` as the long-term standard**, but that migration is still **repo-by-repo** and **incremental**. +- During the transition, each repo's Docker/build flow must follow the repo's own: + - `packageManager` field + - lockfile + - Dockerfile + - `docker-prep.sh` behavior +- This plan does **not** merge all repos into one mega-monorepo. Product repos remain independent repositories. +- Once a repo migrates to `pnpm`, it must be fully aligned in the same change set: + - no `pnpm-lock.yaml` with `npm ci` + - no stale `package-lock.json` + - no mixed package-manager assumptions in CI, Docker, or docs + +> **Migration-impact note:** The deployment architecture in this guide stays the same during the `pnpm` migration (Compose, K3s, ingress, namespaces, VM sizing). The main maintenance surface is Docker/build instructions and dependency-prep flow. The biggest operational risk is stale templates or stale docs after an individual repo migrates. + +--- + ## 1. Service Inventory ### Shared Infrastructure (common-plat) @@ -112,7 +131,7 @@ ### Phase 1: Docker Compose (after prerequisite work) -> **⚠️ Prerequisite:** ALL product repos must run `docker-prep.sh` before building Docker images (see §12 Audit Findings). All Dockerfiles and `output: 'standalone'` configs are now in place (completed 2026-03-22). +> **⚠️ Prerequisite:** ALL product repos must run `docker-prep.sh` before building Docker images (see §12 Audit Findings). All Dockerfiles and `output: 'standalone'` configs are now in place (completed 2026-03-22). During the package-manager transition, each repo's Docker build must follow that repo's declared package manager and lockfile semantics rather than assuming `npm` or `pnpm` globally. Create a **unified** `docker-compose.ecosystem.yml` that brings everything up. @@ -158,6 +177,8 @@ Create `docker-compose.ecosystem.yml` at workspace root (`~/code/mygh/`) that co # Pack @bytelyst/* file: dependencies into tarballs for each product repo. # Every product repo has file: refs to ../learning_ai_common_plat/packages/* # which don't resolve inside Docker build context. docker-prep.sh packs them. +# The prep flow must preserve each repo's package-manager semantics while rewriting +# file: refs for Docker contexts. for repo in learning_voice_ai_agent learning_multimodal_memory_agents learning_ai_clock \ learning_ai_jarvis_jr learning_ai_peakpulse learning_ai_flowmonk \ learning_ai_fastgap learning_ai_notes learning_ai_trails learning_ai_local_memory_gpt; do @@ -796,30 +817,32 @@ kubectl get pods -A ## 10. Dockerization Status (all complete) -| Repo | Backend Dockerfile | Web Dockerfile | `docker-prep.sh` | `output:'standalone'` | Status | -| --------------- | ------------------ | ------------------- | ---------------- | --------------------- | ------------------------------------ | -| **LysnrAI** | ✅ | ✅ user-dashboard | ✅ | ✅ (conditional) | ✅ Ready | -| **MindLyst** | ✅ | ✅ | ✅ | ✅ (conditional) | ✅ Ready | -| **ChronoMind** | ✅ | ✅ | ✅ | ✅ (conditional) | ✅ Ready | -| **JarvisJr** | ✅ | ✅ | ✅ | ✅ (conditional) | ✅ Ready | -| **PeakPulse** | ✅ | — (no web) | ✅ | — | ✅ Ready | -| **FlowMonk** | ✅ | ✅ | ✅ | ✅ (conditional) | ✅ Ready | -| **NomGap** | ✅ | ✅ | ✅ | ✅ | ✅ Fixed (added `.tarballs/` COPY) | -| **NoteLett** | ✅ | ✅ | ✅ | ✅ | ✅ Fixed (explicit COPY, not `.`) | -| **ActionTrail** | ✅ | ✅ | ✅ | ✅ | ✅ Ready (uses `.tarballs/` pattern) | -| **LocalMemGPT** | ✅ | ✅ | ✅ | ✅ | ✅ Ready (repo-root build context) | -| **admin-web** | — | ✅ (in common-plat) | N/A (pnpm) | ✅ (conditional) | ✅ Ready | -| **tracker-web** | — | ✅ (in common-plat) | N/A (pnpm) | ✅ (conditional) | ✅ Ready | +| Repo | Backend Dockerfile | Web Dockerfile | `docker-prep.sh` | `output:'standalone'` | Package manager state | Lockfile state | Docker template type | Status | +| --------------- | ------------------ | ------------------- | ---------------- | --------------------- | ---------------------------------------- | ---------------------------------- | ------------------------------- | ------------------------------------ | +| **LysnrAI** | ✅ | ✅ user-dashboard | ✅ | ✅ (conditional) | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready | +| **MindLyst** | ✅ | ✅ | ✅ | ✅ (conditional) | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready | +| **ChronoMind** | ✅ | ✅ | ✅ | ✅ (conditional) | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready | +| **JarvisJr** | ✅ | ✅ | ✅ | ✅ (conditional) | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready | +| **PeakPulse** | ✅ | — (no web) | ✅ | — | No Node web surface in this repo | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready | +| **FlowMonk** | ✅ | ✅ | ✅ | ✅ (conditional) | **Pilot candidate** for `pnpm` migration | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready | +| **NomGap** | ✅ | ✅ | ✅ | ✅ | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Fixed (added `.tarballs/` COPY) | +| **NoteLett** | ✅ | ✅ | ✅ | ✅ | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Fixed (explicit COPY, not `.`) | +| **ActionTrail** | ✅ | ✅ | ✅ | ✅ | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready (uses `.tarballs/` pattern) | +| **LocalMemGPT** | ✅ | ✅ | ✅ | ✅ | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready (repo-root build context) | +| **admin-web** | — | ✅ (in common-plat) | N/A (`pnpm`) | ✅ (conditional) | `pnpm` workspace today | `pnpm-lock.yaml` via common-plat | `pnpm` workspace template | ✅ Ready | +| **tracker-web** | — | ✅ (in common-plat) | N/A (`pnpm`) | ✅ (conditional) | `pnpm` workspace today | `pnpm-lock.yaml` via common-plat | `pnpm` workspace template | ✅ Ready | **All 10 product repos now have Dockerfiles, `docker-prep.sh`, and `output:'standalone'`.** Created 2026-03-22. +> **Note:** The table above tracks Docker readiness, not completed package-manager migration. For product repos, use each repo's actual `packageManager` field and lockfile until that repo is explicitly migrated to `pnpm`. + --- ## 11. Dockerfile Template (reference) -> **Critical:** These templates assume you run `docker-prep.sh` first to pack `@bytelyst/*` file: deps into `.tarballs/`. Without this, `npm ci` will fail because `file:../../learning_ai_common_plat/packages/*` doesn't exist inside the Docker build context. +> **Critical:** Run `docker-prep.sh` first for product repos that use `@bytelyst/*` `file:` dependencies. The prep step packs those dependencies into `.tarballs/` so Docker builds can resolve them inside the repo's own build context. During the migration window, Dockerfiles must match the repo's package manager and lockfile instead of assuming a single global install command. -### Backend (Fastify 5 + TypeScript) +### Backend / service template — `npm` repo variant ```dockerfile # Pre-requisite: run ./scripts/docker-prep.sh to pack @bytelyst/* tarballs @@ -851,7 +874,41 @@ EXPOSE ${PORT:-4010} CMD ["node", "dist/server.js"] ``` -### Web (Next.js 16) +### Backend / service template — `pnpm` repo variant + +```dockerfile +# Pre-requisite: run ./scripts/docker-prep.sh if this repo rewrites @bytelyst/* file: deps +FROM node:22-alpine AS builder +WORKDIR /app + +RUN corepack enable && corepack prepare pnpm@10 --activate + +COPY package.json pnpm-lock.yaml ./ +COPY .tarballs/ ./.tarballs/ +RUN pnpm install --frozen-lockfile --ignore-scripts + +COPY tsconfig.json ./ +COPY src/ ./src/ +RUN pnpm run build + +FROM node:22-alpine +WORKDIR /app +ENV NODE_ENV=production + +RUN corepack enable && corepack prepare pnpm@10 --activate + +COPY package.json pnpm-lock.yaml ./ +COPY .tarballs/ ./.tarballs/ +RUN pnpm install --frozen-lockfile --prod --ignore-scripts + +COPY --from=builder /app/dist ./dist +COPY shared/ ./shared/ 2>/dev/null || true + +EXPOSE ${PORT:-4010} +CMD ["node", "dist/server.js"] +``` + +### Web (Next.js 16) — `npm` repo variant > **Prerequisite:** `next.config.ts` MUST have `output: 'standalone'` for the standalone Dockerfile pattern to work. Without it, `.next/standalone/` won't be generated and the COPY will fail. @@ -884,15 +941,121 @@ EXPOSE 3000 CMD ["node", "server.js"] ``` +### Web (Next.js 16) — `pnpm` repo variant + +> **Prerequisite:** `next.config.ts` MUST have `output: 'standalone'` for the standalone Dockerfile pattern to work. Keep the repo's `build` script authoritative, including `--webpack` where required. + +```dockerfile +# Pre-requisite: run ./scripts/docker-prep.sh to pack @bytelyst/* tarballs when applicable +FROM node:22-alpine AS builder +WORKDIR /app + +RUN corepack enable && corepack prepare pnpm@10 --activate + +COPY package.json pnpm-lock.yaml ./ +COPY .tarballs/ ./.tarballs/ +RUN pnpm install --frozen-lockfile + +COPY . . + +# Dummy env vars for Next.js build-time static page collection +ENV NEXT_PUBLIC_BACKEND_URL=http://localhost:4010 +ENV NEXT_PUBLIC_PLATFORM_SERVICE_URL=http://localhost:4003 + +RUN pnpm run build + +FROM node:22-alpine +WORKDIR /app +ENV NODE_ENV=production + +COPY --from=builder /app/.next/standalone ./ +COPY --from=builder /app/.next/static ./.next/static +COPY --from=builder /app/public ./public 2>/dev/null || true + +EXPOSE 3000 +CMD ["node", "server.js"] +``` + +> **Template selection rule:** +> +> - Use the `npm` variant only for repos that are still on `npm` with `package-lock.json` and matching Docker/CI scripts. +> - Use the `pnpm` variant for repos that have migrated to `pnpm` and carry `pnpm-lock.yaml` plus aligned CI/Docker/docs. +> - Do **not** leave a repo in mixed state after migration. + ### docker-prep.sh (for repos that don't have one yet) Copy from `learning_ai_trails/scripts/docker-prep.sh` — it handles both `backend/` and `web/` targets, packs all `file:` refs into `.tarballs/`, and rewrites `package.json` to point at them. +The important rule is **behavior**, not shell-script ancestry: + +- `docker-prep.sh` must support both legacy `npm` repos and migrated `pnpm` repos. +- It must **not** hardcode `npm` assumptions into tarball rewrite flow. +- It must preserve the repo's package-manager semantics after prep: + - keep the correct lockfile + - keep the correct install command in Docker/CI + - keep `.tarballs/` handling compatible with the repo's active package manager + ```bash cp learning_ai_trails/scripts/docker-prep.sh /scripts/docker-prep.sh chmod +x /scripts/docker-prep.sh ``` +## 11.1 Long-Term Package-Manager Migration Roadmap + +### End-state + +- `learning_ai_common_plat` remains the canonical **`pnpm` workspace** monorepo. +- Node-based product repos migrate to **`pnpm` over time**. +- Product repos remain **independent repositories**, not one combined workspace. +- Current `.tarballs/` handling for `@bytelyst/*` remains supported unless it is explicitly simplified later. + +### Migration principles + +- No big-bang migration. +- One repo at a time. +- Fully green before moving to the next repo. +- Do not combine package-manager migration with unrelated dependency upgrades. +- Migrate CI, Docker, and docs together in the same repo migration. +- No mixed lockfile/package-manager state after migration. + +### Phase 0 — policy and checklist + +- Define package-manager policy. +- Define migration checklist. +- Define validation gates. + +### Pilot + +- `learning_ai_flowmonk` + +### Wave 1 + +- `learning_ai_trails` +- `learning_ai_local_memory_gpt` + +### Wave 2 + +- `learning_ai_notes` +- `learning_ai_fastgap` +- `learning_ai_clock` + +### Wave 3 + +- `learning_ai_jarvis_jr` +- `learning_voice_ai_agent` + +### Validation gates per migrated repo + +A repo is only considered migrated when all of the following are aligned and passing: + +- install +- test +- typecheck +- build +- Docker build +- local shared package resolution +- docs/CI updated + --- ## 12. Audit Findings (Review 2026-03-22) @@ -921,9 +1084,12 @@ Systematic code review of all claims in this document against the actual codebas 1. Runs `pnpm build` in common-plat 2. Packs each `@bytelyst/*` package into a `.tarballs/*.tgz` 3. Rewrites package.json `file:` refs → `file:.tarballs/bytelyst-*.tgz` +4. Preserves the product repo's active package-manager semantics during the rewrite **All 10 repos now have `docker-prep.sh`** (created 2026-03-22). Previously only ActionTrail, LocalMemGPT, NoteLett, NomGap had them. +> **Long-term note:** As product repos migrate to `pnpm`, this pattern remains valid. What changes is the repo-local install/runtime contract (`pnpm install --frozen-lockfile` instead of `npm ci`), not the deployment architecture or the need to package `@bytelyst/*` dependencies for isolated Docker contexts. + ### F3. NomGap Backend Dockerfile Ignores `file:` Deps (BUG) `@/learning_ai_fastgap/backend/Dockerfile` does `COPY package.json → npm ci` but doesn't copy `.tarballs/`. The `file:` refs will fail. Needs the `.tarballs/` COPY step added. @@ -967,7 +1133,7 @@ Existing Dockerfiles use mixed Node versions: ### F8. Missing `--webpack` Flag for Next.js Builds -Several web apps require `--webpack` flag for builds (Serwist PWA incompatible with Turbopack, or `@bytelyst/*` file: ref transpilation). The Dockerfile template uses `npm run build` which should map to `next build --webpack` in package.json — verify each repo's `build` script. +Several web apps require `--webpack` flag for builds (Serwist PWA incompatible with Turbopack, or `@bytelyst/*` file: ref transpilation). The Dockerfile template should call the repo's package-manager-appropriate build command (`npm run build` or `pnpm run build`) and that script should map to `next build --webpack` where required. ### F9. Missing `.env.ecosystem` Template @@ -1319,6 +1485,7 @@ esac | ------------------------------ | -------------------------------------------------------------------------------------------------------------- | | **Can deploy on single VM?** | **Yes.** All ~25 services fit in 32 GB RAM. | | **All Dockerized?** | **Yes.** All 10 product repos now have Dockerfiles + docker-prep.sh. | +| **Package-manager direction?** | **`pnpm` is the long-term standard** for Node/TS repos, but migration is phased repo-by-repo, not big-bang. | | **K8s practice on single VM?** | **Docker Desktop K8s** (Mac/Windows) or **K3s** (Linux). Same manifests scale to AKS/EKS/GKE. | | **Recommended VM?** | 8 vCPU / 32 GB (min) or 16 vCPU / 64 GB (with Ollama). Hetzner ~$45/mo for dev. | | **Time to production K8s?** | Phase 1 (compose) → Phase 2 (Docker Desktop / K3s) → Phase 3 (multi-node) → Phase 4 (managed). Same manifests. |