diff --git a/docs/devops/SINGLE_VM_DEPLOYMENT.md b/docs/devops/SINGLE_VM_DEPLOYMENT.md index 3fdafb4e..e66be73b 100644 --- a/docs/devops/SINGLE_VM_DEPLOYMENT.md +++ b/docs/devops/SINGLE_VM_DEPLOYMENT.md @@ -4,22 +4,13 @@ --- -## Package-Manager Strategy (current transition plan) +## Package-Manager Strategy -- `learning_ai_common_plat` is already the canonical **`pnpm` workspace** monorepo for shared packages, services, and dashboards. -- Node/TypeScript product repos are moving toward **`pnpm` as the long-term standard**, but that migration is still **repo-by-repo** and **incremental**. -- During the transition, each repo's Docker/build flow must follow the repo's own: - - `packageManager` field - - lockfile - - Dockerfile - - `docker-prep.sh` behavior -- This plan does **not** merge all repos into one mega-monorepo. Product repos remain independent repositories. -- Once a repo migrates to `pnpm`, it must be fully aligned in the same change set: - - no `pnpm-lock.yaml` with `npm ci` - - no stale `package-lock.json` - - no mixed package-manager assumptions in CI, Docker, or docs - -> **Migration-impact note:** The deployment architecture in this guide stays the same during the `pnpm` migration (Compose, K3s, ingress, namespaces, VM sizing). The main maintenance surface is Docker/build instructions and dependency-prep flow. The biggest operational risk is stale templates or stale docs after an individual repo migrates. +- `learning_ai_common_plat` is the canonical **`pnpm` workspace** monorepo for shared packages, services, and dashboards. +- All 10 Node/TypeScript product repos use **`pnpm`** with `pnpm-lock.yaml`. +- Product repos remain **independent repositories**, not one combined workspace. +- All repos consume `@bytelyst/*` packages from the local **Gitea npm registry** (49 packages published). +- All Dockerfiles use **pnpm + BuildKit secret mount** for registry authentication. --- @@ -132,11 +123,9 @@ ## 3. Architecture: Docker Compose β†’ K3s Migration Path -### Phase 1: Docker Compose (after prerequisite work) +### Phase 1: Docker Compose -> **βœ… Prerequisite RESOLVED (2026-03-24):** All 10 repos now consume `@bytelyst/*` packages from the Gitea npm registry. `docker-prep.sh` has been deleted from all repos. Docker builds use pnpm + BuildKit secret mount pattern. See [`GITEA_NPM_REGISTRY_MIGRATION.md`](GITEA_NPM_REGISTRY_MIGRATION.md) Β§14-17 for details. -> -> **πŸ“‹ Enhanced plan:** See [`SINGLE_VM_ENHANCED_PLAN.md`](SINGLE_VM_ENHANCED_PLAN.md) for the updated deployment plan with Coolify, Valkey, Uptime Kuma, and other open-source tooling additions. +> **πŸ“‹ Enhanced plan:** See [`SINGLE_VM_ENHANCED_PLAN.md`](SINGLE_VM_ENHANCED_PLAN.md) for the deployment plan with Coolify, Valkey, Uptime Kuma, and other open-source tooling. Create a **unified** `docker-compose.ecosystem.yml` that brings everything up. @@ -176,25 +165,9 @@ Docker Desktop includes a built-in **kind** (Kubernetes IN Docker) cluster. Enab Create `docker-compose.ecosystem.yml` at workspace root (`~/code/mygh/`) that composes all services: -**⚠️ Critical prerequisite β€” run BEFORE `docker compose build`:** - -```bash -# Pack @bytelyst/* file: dependencies into tarballs for each product repo. -# Every product repo has file: refs to ../learning_ai_common_plat/packages/* -# which don't resolve inside Docker build context. docker-prep.sh packs them. -# The prep flow must preserve each repo's package-manager semantics while rewriting -# file: refs for Docker contexts. -for repo in learning_voice_ai_agent learning_multimodal_memory_agents learning_ai_clock \ - learning_ai_jarvis_jr learning_ai_peakpulse \ - learning_ai_fastgap learning_ai_notes learning_ai_trails learning_ai_local_memory_gpt; do - (cd $repo && ./scripts/docker-prep.sh) -done -``` - ```yaml # ~/code/mygh/docker-compose.ecosystem.yml -# NOTE: Most product backends/webs still rely on file: deps to @bytelyst/* packages. -# Run docker-prep.sh only for the repos that still require tarball prep (see above). +# All repos consume @bytelyst/* from local Gitea npm registry. services: # ══════════════════════════════════════════════════════ @@ -317,7 +290,6 @@ services: # ══════════════════════════════════════════════════════ # PRODUCT BACKENDS - # Most still have file: deps β†’ run docker-prep.sh first unless the repo is already registry-backed. # ActionTrail + LocalMemGPT Dockerfiles use repo-root context. # Others use backend/ subdir context. # ══════════════════════════════════════════════════════ @@ -675,89 +647,46 @@ spec: --- -## 5.1 Scaling-Readiness Reality Check +## 5.1 Scaling-Readiness -The ecosystem code and infra design are already **mostly** aligned for a low-effort scaling path. +The ecosystem is designed for low-effort scaling: -That does **not** mean the whole end-to-end deployment path is proven yet. +- Service identity, ports, and inter-service URLs are env/config driven across all backends and dashboards +- Compose service boundaries map cleanly to Kubernetes Deployments + Services +- Traefik-based routing maps cleanly to Kubernetes Ingress resources +- The namespace split (`infra`, `platform`, `products`, `web`) provides a usable organizational model for K8s +- All repos consume `@bytelyst/*` from Gitea npm registry β€” Docker builds use standard `pnpm install` with BuildKit secret mount +- Moving from 1 replica to N replicas for stateless services is infra config work, not application rewrites +- Moving from Docker Desktop K8s to K3s or managed K8s reuses the same manifest model -### What is already designed for low-change scaling +### Scaling the running system means -- service identity, ports, and inter-service URLs are already env/config driven in most backends and dashboards -- Compose service boundaries already map cleanly to Kubernetes Deployments + Services -- Traefik-based routing already maps cleanly to Kubernetes Ingress resources -- resource requests/limits and replica counts are already represented in a K8s-friendly way in the documented manifest examples -- the namespace split (`infra`, `platform`, `products`, `web`) already gives a usable organizational model for K8s -- moving from 1 replica to N replicas for stateless services should mostly be infra config work, not application rewrites -- moving from Docker Desktop K8s to K3s or later managed K8s should mostly reuse the same manifest model - -### What is not yet proven enough to call low-change - -- the shared package distribution path for container builds after the Gitea npm registry migration -- the pilot Docker build path from the local Gitea registry -- the final removal of `docker-prep.sh` / tarball fallback across the wider ecosystem -- the complete image build/publish/deploy workflow for all repos under one consistent registry strategy - -### Practical interpretation - -If we solve the package registry + image build path cleanly, then scaling the running system should mostly mean: - -- increasing replica counts -- applying HPAs where justified -- adjusting resource requests and limits -- moving from local-path storage to stronger persistent storage where needed -- adding worker nodes or moving to a managed cluster - -That is the sense in which the ecosystem is already mostly configurable and scale-friendly. - -The remaining work is concentrated more in the **build and package-distribution layer** than in the application service code. +- Increasing replica counts +- Applying HPAs where justified +- Adjusting resource requests and limits +- Moving from local-path storage to persistent storage where needed +- Adding worker nodes or moving to a managed cluster --- -## 5.2 Remaining Gaps Before Azure VM / K3s Rollout +## 5.2 Remaining Work Before K3s Rollout -### Local Mac rehearsal β€” validated βœ… +### Validated βœ… -- Host-side `pnpm install` against local Gitea registry -- FlowMonk backend/web Docker builds with BuildKit secret + `--add-host localhost:host-gateway` (`NETWORK=corp` local-dev topology) -- Local Gitea CI green for FlowMonk and common-platform on current commits -- Runner re-registered against `127.0.0.1` (avoids IPv6 `[::1]` declaration failure) +- Host-side `pnpm install` against local Gitea registry (all 10 repos) +- Docker builds for all 10 repos with BuildKit secret mount +- 49 `@bytelyst/*` packages published to Gitea npm registry +- Local Gitea CI green for 8/8 repos with workflows +- Docker builds verified for MindLyst (backend + web) and LysnrAI (backend + dashboard) +- 1,591 backend tests passing, 9/9 web typechecks clean -The `--add-host localhost:host-gateway` pattern is the **correct local-dev recipe** when `NETWORK=corp`. On the target VM, Gitea's `ROOT_URL` will be the real hostname or compose service name, so tarball URLs resolve natively β€” no `--add-host` needed. +### K3s readiness β€” still needed -### Local Mac rehearsal β€” gaps still open - -- Local Gitea Actions has not yet been fully validated for the package build β†’ publish β†’ consumer flow -- Not all `@bytelyst/*` packages have been revalidated under the local Gitea registry with a fresh consumer install -- Mobile remains outside the registry-backed pilot slice - -### Azure VM readiness gaps - -Before calling the Azure VM rollout low-risk, we still need one validated answer for each of these: - -- where Gitea runs and what `ROOT_URL` / package URL shape it advertises -- how Docker image builds authenticate to and consume the package registry -- whether package builds happen on the VM directly, inside CI runners, or via a dedicated package-publish pipeline -- how image publishing is handled for K3s / later multi-node rollout -- which repos are fully registry-backed versus still temporarily using tarball fallback during migration - -### K3s / scaling readiness gaps - -The application architecture is largely ready for scale-by-configuration, but these operational items still need proof or stronger artifacts: - -- concrete K8s manifests or Helm values for the full ecosystem, not just examples -- image registry strategy for K3s and later multi-node or managed Kubernetes rollout -- persistent volume strategy for Gitea, Grafana, Loki, Azurite, and any stateful product workloads -- autoscaling thresholds and resource defaults validated under representative load -- deployment ordering and health-check gating for infra β†’ platform β†’ products β†’ web surfaces - -### Recommended order to close these gaps - -1. clear the local Gitea package URL and Docker build problem -2. validate one full pilot path: package publish β†’ Docker build β†’ runtime start -3. validate the same path in local Gitea Actions -4. choose the image registry / image distribution pattern for Azure VM and K3s -5. generate and validate concrete K8s/Helm artifacts for the platform + one pilot product first +- Concrete K8s manifests or Helm values for the full ecosystem (not just examples) +- Image registry strategy for K3s (Gitea container registry or local import) +- Persistent volume strategy for Gitea, Grafana, Loki, Azurite, and stateful workloads +- Autoscaling thresholds and resource defaults validated under representative load +- Deployment ordering and health-check gating for infra β†’ platform β†’ products β†’ web --- @@ -769,8 +698,8 @@ These exercises simulate real production scenarios: ```bash # Build new image, deploy with zero downtime -docker build -t bytelyst/lysnrai-backend:v2 ./learning_voice_ai_agent/backend -kubectl set image deploy/lysnrai-backend lysnrai-backend=bytelyst/lysnrai-backend:v2 -n bytelyst-products +docker build -t bytelyst/lysnrai-backend:1.1.0 ./learning_voice_ai_agent/backend +kubectl set image deploy/lysnrai-backend lysnrai-backend=bytelyst/lysnrai-backend:1.1.0 -n bytelyst-products kubectl rollout status deploy/lysnrai-backend -n bytelyst-products ``` @@ -937,78 +866,45 @@ kubectl get pods -A --- -## 10. Dockerization Status (all complete) +## 10. Dockerization Status -| Repo | Backend Dockerfile | Web Dockerfile | `docker-prep.sh` | `output:'standalone'` | Package manager state | Lockfile state | Docker template type | Status | -| ------------------ | ------------------ | ------------------- | ---------------- | --------------------- | ---------------------------------------- | ---------------------------------- | ------------------------------- | ------------------------------------ | -| **LysnrAI** | βœ… | βœ… user-dashboard | βœ… | βœ… (conditional) | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | βœ… Ready | -| **MindLyst** | βœ… | βœ… | βœ… | βœ… (conditional) | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | βœ… Ready | -| **ChronoMind** | βœ… | βœ… | βœ… | βœ… (conditional) | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | βœ… Ready | -| **JarvisJr** | βœ… | βœ… | βœ… | βœ… (conditional) | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | βœ… Ready | -| **PeakPulse** | βœ… | β€” (no web) | βœ… | β€” | No Node web surface in this repo | Follow repo-local current lockfile | Repo-specific during transition | βœ… Ready | -| **FlowMonk** | βœ… | βœ… | βœ… | βœ… (conditional) | **Pilot candidate** for `pnpm` migration | Follow repo-local current lockfile | Repo-specific during transition | βœ… Ready | -| **NomGap** | βœ… | βœ… | βœ… | βœ… | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | βœ… Fixed (added `.tarballs/` COPY) | -| **NoteLett** | βœ… | βœ… | βœ… | βœ… | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | βœ… Fixed (explicit COPY, not `.`) | -| **ActionTrail** | βœ… | βœ… | βœ… | βœ… | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | βœ… Ready (uses `.tarballs/` pattern) | -| **LocalMemGPT** | βœ… | βœ… | βœ… | βœ… | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | βœ… Ready (repo-root build context) | -| **admin-web** | β€” | βœ… (in common-plat) | N/A (`pnpm`) | βœ… (conditional) | `pnpm` workspace today | `pnpm-lock.yaml` via common-plat | `pnpm` workspace template | βœ… Ready | -| **tracker-web** | β€” | βœ… (in common-plat) | N/A (`pnpm`) | βœ… (conditional) | `pnpm` workspace today | `pnpm-lock.yaml` via common-plat | `pnpm` workspace template | βœ… Ready | -| **user-dashboard** | β€” | βœ… (in common-plat) | N/A (`pnpm`) | βœ… (conditional) | `pnpm` workspace today | `pnpm-lock.yaml` via common-plat | `pnpm` workspace template | βœ… Ready | +| Repo | Backend Dockerfile | Web Dockerfile | `output:'standalone'` | Build Context | Status | +| ------------------ | ------------------ | ------------------- | --------------------- | -------------- | --------- | +| **LysnrAI** | βœ… | βœ… user-dashboard | βœ… (conditional) | repo root | βœ… Ready | +| **MindLyst** | βœ… | βœ… | βœ… (conditional) | repo root | βœ… Ready | +| **ChronoMind** | βœ… | βœ… | βœ… (conditional) | backend/ | βœ… Ready | +| **JarvisJr** | βœ… | βœ… | βœ… (conditional) | backend/ | βœ… Ready | +| **PeakPulse** | βœ… | β€” (no web) | β€” | backend/ | βœ… Ready | +| **FlowMonk** | βœ… | βœ… | βœ… (conditional) | repo root | βœ… Ready | +| **NomGap** | βœ… | βœ… | βœ… | backend/ | βœ… Ready | +| **NoteLett** | βœ… | βœ… | βœ… | backend/ | βœ… Ready | +| **ActionTrail** | βœ… | βœ… | βœ… | repo root | βœ… Ready | +| **LocalMemGPT** | βœ… | βœ… | βœ… | repo root | βœ… Ready | +| **admin-web** | β€” | βœ… (in common-plat) | βœ… (conditional) | pnpm workspace | βœ… Ready | +| **tracker-web** | β€” | βœ… (in common-plat) | βœ… (conditional) | pnpm workspace | βœ… Ready | +| **user-dashboard** | β€” | βœ… (in common-plat) | βœ… (conditional) | pnpm workspace | βœ… Ready | -**All 10 product repos now have Dockerfiles, and most still retain `docker-prep.sh` plus `output:'standalone'` where applicable.** Created 2026-03-22. - -> **Note:** The table above tracks Docker readiness, not completed package-manager migration. For product repos, use each repo's actual `packageManager` field and lockfile until that repo is explicitly migrated to `pnpm`. +All repos use pnpm with Gitea npm registry. Docker builds use BuildKit secret mount for registry auth. --- -## 11. Dockerfile Template (reference) +## 11. Dockerfile Templates (reference) -> **Critical:** Run `docker-prep.sh` first for product repos that use `@bytelyst/*` `file:` dependencies. The prep step packs those dependencies into `.tarballs/` so Docker builds can resolve them inside the repo's own build context. During the migration window, Dockerfiles must match the repo's package manager and lockfile instead of assuming a single global install command. +All Dockerfiles use pnpm with BuildKit secret mount for Gitea npm registry authentication. `next.config.ts` MUST have `output: 'standalone'` for web Dockerfiles. -### Backend / service template β€” `npm` repo variant +### Backend template ```dockerfile -# Pre-requisite: run ./scripts/docker-prep.sh to pack @bytelyst/* tarballs -FROM node:22-alpine AS builder -WORKDIR /app - -COPY package.json package-lock.json ./ -COPY .tarballs/ ./.tarballs/ -RUN npm ci --ignore-scripts - -COPY tsconfig.json ./ -COPY src/ ./src/ -RUN npx tsc - -# Production stage -FROM node:22-alpine -WORKDIR /app -ENV NODE_ENV=production - -COPY package.json package-lock.json ./ -COPY .tarballs/ ./.tarballs/ -RUN npm ci --omit=dev --ignore-scripts - -COPY --from=builder /app/dist ./dist -# Copy shared/product.json if the backend reads it at runtime -COPY shared/ ./shared/ 2>/dev/null || true - -EXPOSE ${PORT:-4010} -CMD ["node", "dist/server.js"] -``` - -### Backend / service template β€” `pnpm` repo variant - -```dockerfile -# Pre-requisite: run ./scripts/docker-prep.sh if this repo rewrites @bytelyst/* file: deps FROM node:22-alpine AS builder WORKDIR /app RUN corepack enable && corepack prepare pnpm@10 --activate +COPY .npmrc.docker ./.npmrc COPY package.json pnpm-lock.yaml ./ -COPY .tarballs/ ./.tarballs/ -RUN pnpm install --frozen-lockfile --ignore-scripts +RUN --mount=type=secret,id=gitea_npm_token \ + export GITEA_NPM_TOKEN="$(cat /run/secrets/gitea_npm_token)" && \ + pnpm install --frozen-lockfile --ignore-scripts COPY tsconfig.json ./ COPY src/ ./src/ @@ -1020,72 +916,37 @@ ENV NODE_ENV=production RUN corepack enable && corepack prepare pnpm@10 --activate +COPY .npmrc.docker ./.npmrc COPY package.json pnpm-lock.yaml ./ -COPY .tarballs/ ./.tarballs/ -RUN pnpm install --frozen-lockfile --prod --ignore-scripts +RUN --mount=type=secret,id=gitea_npm_token \ + export GITEA_NPM_TOKEN="$(cat /run/secrets/gitea_npm_token)" && \ + pnpm install --frozen-lockfile --prod --ignore-scripts COPY --from=builder /app/dist ./dist -COPY shared/ ./shared/ 2>/dev/null || true +COPY shared/product.json ../shared/product.json EXPOSE ${PORT:-4010} CMD ["node", "dist/server.js"] ``` -### Web (Next.js 16) β€” `npm` repo variant - -> **Prerequisite:** `next.config.ts` MUST have `output: 'standalone'` for the standalone Dockerfile pattern to work. Without it, `.next/standalone/` won't be generated and the COPY will fail. +### Web (Next.js 16) template ```dockerfile -# Pre-requisite: run ./scripts/docker-prep.sh to pack @bytelyst/* tarballs -FROM node:22-alpine AS builder -WORKDIR /app - -COPY package.json package-lock.json ./ -COPY .tarballs/ ./.tarballs/ -RUN npm ci - -COPY . . - -# Dummy env vars for Next.js build-time static page collection -ENV NEXT_PUBLIC_BACKEND_URL=http://localhost:4010 -ENV NEXT_PUBLIC_PLATFORM_SERVICE_URL=http://localhost:4003 - -RUN npm run build - -FROM node:22-alpine -WORKDIR /app -ENV NODE_ENV=production - -COPY --from=builder /app/.next/standalone ./ -COPY --from=builder /app/.next/static ./.next/static -COPY --from=builder /app/public ./public 2>/dev/null || true - -EXPOSE 3000 -CMD ["node", "server.js"] -``` - -### Web (Next.js 16) β€” `pnpm` repo variant - -> **Prerequisite:** `next.config.ts` MUST have `output: 'standalone'` for the standalone Dockerfile pattern to work. Keep the repo's `build` script authoritative, including `--webpack` where required. - -```dockerfile -# Pre-requisite: run ./scripts/docker-prep.sh to pack @bytelyst/* tarballs when applicable FROM node:22-alpine AS builder WORKDIR /app RUN corepack enable && corepack prepare pnpm@10 --activate +COPY .npmrc.docker ./.npmrc COPY package.json pnpm-lock.yaml ./ -COPY .tarballs/ ./.tarballs/ -RUN pnpm install --frozen-lockfile +RUN --mount=type=secret,id=gitea_npm_token \ + export GITEA_NPM_TOKEN="$(cat /run/secrets/gitea_npm_token)" && \ + pnpm install --frozen-lockfile COPY . . -# Dummy env vars for Next.js build-time static page collection -ENV NEXT_PUBLIC_BACKEND_URL=http://localhost:4010 -ENV NEXT_PUBLIC_PLATFORM_SERVICE_URL=http://localhost:4003 - -RUN pnpm run build +ENV NEXT_TELEMETRY_DISABLED=1 +RUN npx next build --webpack FROM node:22-alpine WORKDIR /app @@ -1093,251 +954,59 @@ ENV NODE_ENV=production COPY --from=builder /app/.next/standalone ./ COPY --from=builder /app/.next/static ./.next/static -COPY --from=builder /app/public ./public 2>/dev/null || true +COPY --from=builder /app/public ./public EXPOSE 3000 CMD ["node", "server.js"] ``` -> **Template selection rule:** -> -> - Use the `npm` variant only for repos that are still on `npm` with `package-lock.json` and matching Docker/CI scripts. -> - Use the `pnpm` variant for repos that have migrated to `pnpm` and carry `pnpm-lock.yaml` plus aligned CI/Docker/docs. -> - Do **not** leave a repo in mixed state after migration. - -### docker-prep.sh (for repos that don't have one yet) - -Copy from `learning_ai_trails/scripts/docker-prep.sh` β€” it handles both `backend/` and `web/` targets, packs all `file:` refs into `.tarballs/`, and rewrites `package.json` to point at them. - -The important rule is **behavior**, not shell-script ancestry: - -- `docker-prep.sh` must support both legacy `npm` repos and migrated `pnpm` repos. -- It must **not** hardcode `npm` assumptions into tarball rewrite flow. -- It must preserve the repo's package-manager semantics after prep: - - keep the correct lockfile - - keep the correct install command in Docker/CI - - keep `.tarballs/` handling compatible with the repo's active package manager +### Docker build command ```bash -cp learning_ai_trails/scripts/docker-prep.sh /scripts/docker-prep.sh -chmod +x /scripts/docker-prep.sh +export TOKEN="$(cat ~/.gitea-npm-token)" +docker build \ + --add-host localhost:host-gateway \ + --build-arg GITEA_NPM_HOST=host.docker.internal \ + --secret id=gitea_npm_token,env=TOKEN \ + -f backend/Dockerfile \ + -t bytelyst/product-backend:latest \ + . ``` -## 11.1 Long-Term Package-Manager Migration Roadmap - -### End-state - -- `learning_ai_common_plat` remains the canonical **`pnpm` workspace** monorepo. -- Node-based product repos migrate to **`pnpm` over time**. -- Product repos remain **independent repositories**, not one combined workspace. -- Current `.tarballs/` handling for `@bytelyst/*` remains supported unless it is explicitly simplified later. - -### Migration principles - -- No big-bang migration. -- One repo at a time. -- Fully green before moving to the next repo. -- Do not combine package-manager migration with unrelated dependency upgrades. -- Migrate CI, Docker, and docs together in the same repo migration. -- No mixed lockfile/package-manager state after migration. - -### Phase 0 β€” policy and checklist - -- Define package-manager policy. -- Define migration checklist. -- Define validation gates. - -### Pilot - -- `learning_ai_flowmonk` - -### Wave 1 - -- `learning_ai_trails` -- `learning_ai_local_memory_gpt` - -### Wave 2 - -- `learning_ai_notes` -- `learning_ai_fastgap` -- `learning_ai_clock` - -### Wave 3 - -- `learning_ai_jarvis_jr` -- `learning_voice_ai_agent` - -### Validation gates per migrated repo - -A repo is only considered migrated when all of the following are aligned and passing: - -- install -- test -- typecheck -- build -- Docker build -- local shared package resolution -- docs/CI updated - --- ## 11.2 Gitea β€” Self-Hosted Git + CI + Package Registry -### Why Gitea on the VM? - Gitea is a lightweight, self-hosted Git forge (~150 MB RAM) that provides three capabilities in one container: 1. **Git hosting** β€” mirror or primary for all 13+ repos 2. **CI/CD** β€” GitHub Actions–compatible workflows via `act_runner` -3. **npm package registry** β€” built-in, eliminates the `docker-prep.sh` tarball workflow +3. **npm package registry** β€” 49 `@bytelyst/*` packages published, consumed by all 10 product repos -### Current pain: `docker-prep.sh` + `.tgz` tarballs - -Every product repo has `file:../../learning_ai_common_plat/packages/*` dependencies. These don't resolve inside Docker build contexts, so each repo needs a `docker-prep.sh` that: - -1. Builds all `@bytelyst/*` packages in common-plat -2. Packs each into a `.tarballs/*.tgz` file -3. Rewrites package.json `file:` refs β†’ `file:.tarballs/bytelyst-*.tgz` -4. Preserves the repo's active package-manager semantics during the rewrite - -This is fragile, slow, and must be repeated for every image rebuild. - -### Solution: Gitea npm Package Registry - -Gitea has a built-in npm registry at `http://gitea:3300/api/packages//npm/`. Publishing `@bytelyst/*` packages there means **all product repos install from the registry like any normal npm dependency** β€” no tarballs, no rewriting, no `docker-prep.sh`. - -> **Current local rehearsal status on this Mac:** host-side publishing, host-side consumer installs, and FlowMonk backend/web Docker builds are all validated. The `--add-host localhost:host-gateway` pattern is the correct local-dev recipe when `NETWORK=corp`. On the target VM, Gitea's `ROOT_URL` will be the real hostname or compose service name, so no `--add-host` workaround is needed. - -#### Setup (one-time) +### Registry configuration ```bash -# 1. Create a Gitea API token (Settings β†’ Applications β†’ Generate Token) -# Scopes: package:write, package:read - -# 2. Configure npm/pnpm to use Gitea registry for @bytelyst scope -# In ~/.npmrc or project .npmrc: +# .npmrc (host-side) β€” points @bytelyst scope to Gitea registry @bytelyst:registry=http://localhost:3300/api/packages/bytelyst/npm/ -//localhost:3300/api/packages/bytelyst/npm/:_authToken= +//localhost:3300/api/packages/bytelyst/npm/:_authToken=${GITEA_NPM_TOKEN} -# For Docker builds, use the compose service name: -@bytelyst:registry=http://gitea:3300/api/packages/bytelyst/npm/ +# .npmrc.docker (Docker-side) β€” uses compose service name or build arg +@bytelyst:registry=http://${GITEA_NPM_HOST:-localhost}:3300/api/packages/bytelyst/npm/ +//localhost:3300/api/packages/bytelyst/npm/:_authToken=${GITEA_NPM_TOKEN} ``` -#### Publish workflow (CI or manual) +### Publish workflow ```bash -# In learning_ai_common_plat β€” publish all packages after build -cd /path/to/learning_ai_common_plat - -# Build all packages +# Build and publish all @bytelyst/* packages to Gitea +cd learning_ai_common_plat pnpm -r --filter './packages/**' build - -# Publish via the local helper that uses the proven pnpm-pack-based flow -GITEA_NPM_TOKEN='' bash ./scripts/publish-local-gitea-packages.sh +GITEA_NPM_TOKEN='' bash ./scripts/publish-local-gitea-packages.sh ``` -This helper is important because raw `npm publish` is not sufficient for packages that contain internal `workspace:*` references. The validated local path uses `pnpm pack` first so workspace dependencies are normalized before the tarball is published to Gitea. +The publish helper uses `pnpm pack` first so `workspace:*` references are normalized before publishing to Gitea. -Or add a CI step in `learning_ai_common_plat/.gitea/workflows/ci.yml`: - -```yaml -publish-packages: - name: Publish @bytelyst/* to Gitea npm registry - runs-on: ubuntu-latest - needs: [build-test-typecheck] - steps: - - name: Pull latest - working-directory: /path/to/learning_ai_common_plat - run: git pull --ff-only origin main || true - - - name: Build packages - working-directory: /path/to/learning_ai_common_plat - run: pnpm -r --filter './packages/**' build - - - name: Publish to Gitea registry - working-directory: /path/to/learning_ai_common_plat - env: - NPM_TOKEN: ${{ secrets.GITEA_NPM_TOKEN }} - run: | - echo "@bytelyst:registry=http://gitea:3300/api/packages/bytelyst/npm/" > .npmrc - echo "//gitea:3300/api/packages/bytelyst/npm/:_authToken=${NPM_TOKEN}" >> .npmrc - for pkg in packages/*/; do - (cd "$pkg" && npm publish --registry http://gitea:3300/api/packages/bytelyst/npm/ 2>/dev/null || true) - done -``` - -#### Product repo migration - -For each product repo, replace `file:` refs with versioned (or `latest`) registry refs: - -```jsonc -// Before (package.json) -{ - "@bytelyst/auth": "file:../../learning_ai_common_plat/packages/auth", - "@bytelyst/config": "file:../../learning_ai_common_plat/packages/config", - "@bytelyst/cosmos": "file:../../learning_ai_common_plat/packages/cosmos" -} - -// After -{ - "@bytelyst/auth": "^0.1.0", - "@bytelyst/config": "^0.1.0", - "@bytelyst/cosmos": "^0.1.0" -} -``` - -With `.npmrc` pointing `@bytelyst` scope to Gitea, host-side `pnpm install` / `npm install` resolves from the local registry successfully in the current local rehearsal. - -For Docker builds, the final single-VM target is still the same, but local validation on this Mac showed that a Homebrew-hosted Gitea instance may require extra reachability and `ROOT_URL` alignment work before BuildKit can consume package metadata and tarballs reliably. - -#### Dockerfile simplification (after registry migration) - -```dockerfile -# NO MORE docker-prep.sh or .tarballs/ needed! -FROM node:22-alpine AS builder -WORKDIR /app - -# .npmrc baked in (or passed as build arg) -COPY .npmrc ./ -COPY package.json pnpm-lock.yaml ./ -RUN corepack enable && pnpm install --frozen-lockfile - -COPY tsconfig.json ./ -COPY src/ ./src/ -RUN pnpm run build - -FROM node:22-alpine -WORKDIR /app -ENV NODE_ENV=production -COPY .npmrc ./ -COPY package.json pnpm-lock.yaml ./ -RUN corepack enable && pnpm install --frozen-lockfile --prod -COPY --from=builder /app/dist ./dist -EXPOSE ${PORT:-4010} -CMD ["node", "dist/server.js"] -``` - -#### Comparison: before vs after - -| Aspect | Current (`docker-prep.sh`) | With Gitea npm registry | -| ------------------- | ----------------------------------- | ---------------------------------------- | -| **Pre-build step** | Run `docker-prep.sh` per repo | None (registry resolves deps) | -| **Package format** | `.tgz` tarballs in `.tarballs/` | Standard npm registry install | -| **`package.json`** | Rewritten with `file:.tarballs/...` | Normal `^0.1.0` semver refs | -| **Docker build** | Must COPY `.tarballs/` into context | Standard `pnpm install` | -| **Version pinning** | Whatever was packed at build time | Explicit semver in lockfile | -| **Local dev** | `file:` symlinks (fast, direct) | Can keep `file:` for dev OR use registry | -| **CI publish** | N/A | Auto-publish on common-plat push | -| **Disk overhead** | ~50 MB tarballs per repo | Centralized in Gitea storage | - -#### Migration path (incremental) - -1. **Phase 0:** Gitea running locally with packages enabled -2. **Phase 1:** Publish `@bytelyst/*` packages to local Gitea using the `pnpm pack`-based helper flow -3. **Phase 2:** Migrate one pilot repo on the host: replace tarball/file refs with semver refs and validate install/typecheck/test -4. **Phase 3:** Validate Docker builds against the local registry -5. **Phase 4:** Only after local Docker validation is green, remove `docker-prep.sh` patterns repo-by-repo -6. **Keep `file:` refs available** for local dev via overrides or link-based workflows if needed +For full details, see [`GITEA_NPM_REGISTRY_MIGRATION.md`](GITEA_NPM_REGISTRY_MIGRATION.md). ### Gitea Container Registry (bonus) @@ -1373,32 +1042,19 @@ Systematic code review of all claims in this document against the actual codebas **Fix:** Set `PORT` env var in compose for each, or use host:container port remapping. -### F2. `file:` Dependencies Break Docker Builds (CRITICAL) +### F2. `@bytelyst/*` Package Distribution for Docker Builds -**Every** product backend and web has `file:../../learning_ai_common_plat/packages/*` dependencies in package.json. These resolve locally via symlinks but **fail inside Docker** because the sibling repo isn't in the build context. +All repos now consume `@bytelyst/*` from the Gitea npm registry (`^0.1.0` semver refs). Docker builds use BuildKit secret mount for registry authentication. -**Pattern:** Each repo needs a `docker-prep.sh` that: +**Status:** βœ… Resolved. See [`GITEA_NPM_REGISTRY_MIGRATION.md`](GITEA_NPM_REGISTRY_MIGRATION.md) for details. -1. Runs `pnpm build` in common-plat -2. Packs each `@bytelyst/*` package into a `.tarballs/*.tgz` -3. Rewrites package.json `file:` refs β†’ `file:.tarballs/bytelyst-*.tgz` -4. Preserves the repo's active package-manager semantics during the rewrite +### F3. NomGap Backend Dockerfile -**All 10 repos now have `docker-prep.sh`** (created 2026-03-22). Previously only ActionTrail, LocalMemGPT, NoteLett, NomGap had them. +**Status:** βœ… Fixed. Uses Gitea registry pattern. -> **Long-term note:** As product repos migrate to `pnpm`, this pattern remains valid. What changes is the repo-local install/runtime contract (`pnpm install --frozen-lockfile` instead of `npm ci`), not the deployment architecture or the need to package `@bytelyst/*` dependencies for isolated Docker contexts. +### F4. NoteLett Backend Dockerfile -### F3. NomGap Backend Dockerfile Ignores `file:` Deps (BUG) - -`@/learning_ai_fastgap/backend/Dockerfile` previously did `COPY package.json β†’ npm ci` without copying `.tarballs/`, which broke `file:` dependency resolution inside Docker. - -**Status:** βœ… Fixed. The backend Dockerfile was updated to include the `.tarballs/` pattern. - -### F4. NoteLett Backend Dockerfile Copies Everything (BUG) - -`@/learning_ai_notes/backend/Dockerfile` previously used `COPY . .` in the build stage, which pulled in broken `node_modules` symlinks from `file:` dependencies. - -**Status:** βœ… Fixed. The backend Dockerfile now uses explicit `COPY` steps instead of broad context copy. +**Status:** βœ… Fixed. Uses explicit `COPY` steps with Gitea registry pattern. ### F5. Missing `output: 'standalone'` in next.config.ts (CRITICAL) @@ -1470,19 +1126,18 @@ LocalMemGPT uses `OLLAMA_URL: 'http://host.docker.internal:11434'` β€” this work **Status:** βœ… Addressed in `docker-compose.ecosystem.yml` via `extra_hosts: ['host.docker.internal:host-gateway']` on `localmemgpt-backend`. -### Summary of Required Work Before Compose Works +### Summary of Audit Items -| Priority | Item | Count | Status | -| -------- | -------------------------------------------------------- | ------------- | ---------------------------------------------------------- | -| **P0** | Create missing `docker-prep.sh` | 6 repos | βœ… Done (3 created, 3 already existed) | -| **P0** | Create missing backend Dockerfiles | 6 repos | βœ… Done | -| **P0** | Create missing web Dockerfiles | 5 repos | βœ… Done (4 created, PeakPulse has no web) | -| **P0** | Add `output: 'standalone'` to next.config.ts | 3 webs | βœ… Done (4 webs: ChronoMind, JarvisJr, FlowMonk, MindLyst) | -| **P1** | Fix NomGap backend Dockerfile (add `.tarballs/` COPY) | 1 file | βœ… Done | -| **P1** | Fix NoteLett backend Dockerfile (explicit COPY, not `.`) | 1 file | βœ… Done | -| **P1** | Create `.env.ecosystem` template | 1 file | βœ… Done | -| **P2** | Standardize Node.js version to 22-alpine | 4 Dockerfiles | βœ… Done (all new Dockerfiles use 22-alpine) | -| **P2** | Add `extra_hosts` for Linux VM Ollama access | 1 service | βœ… Done (`localmemgpt-backend`) | +All audit findings have been resolved: + +| Priority | Item | Status | +| -------- | ---------------------------------------------- | -------- | +| **P0** | All backend + web Dockerfiles created | βœ… Done | +| **P0** | `output: 'standalone'` in all web configs | βœ… Done | +| **P0** | Gitea npm registry for `@bytelyst/*` packages | βœ… Done | +| **P1** | `.env.ecosystem` template | βœ… Done | +| **P2** | Standardize Node.js version to 22-alpine | βœ… Done | +| **P2** | `extra_hosts` for Linux VM Ollama access | βœ… Done | --- @@ -1514,11 +1169,12 @@ Use that document for: | Question | Answer | | ------------------------------ | -------------------------------------------------------------------------------------------------------------- | -| **Can deploy on single VM?** | **Yes.** All ~27 services fit in 32 GB RAM. | -| **All Dockerized?** | **Yes.** All 10 product repos now have Dockerfiles + docker-prep.sh. | -| **Package-manager direction?** | **`pnpm` is the long-term standard** for Node/TS repos, but migration is phased repo-by-repo, not big-bang. | +| **Can deploy on single VM?** | **Yes.** All ~35 containers fit in 32 GB RAM. | +| **All Dockerized?** | **Yes.** All 10 product repos + 3 shared services + 3 dashboards have Dockerfiles. | +| **Package manager?** | **pnpm** across all repos. `@bytelyst/*` packages from Gitea npm registry. | | **Self-hosted CI?** | **Yes.** Gitea + act_runner (~250 MB combined). GitHub Actions–compatible workflows. | -| **Package registry?** | **Gitea npm registry** replaces `docker-prep.sh` + `.tgz` tarballs. Also has Docker container registry. | +| **Package registry?** | **Gitea npm registry** β€” 49 `@bytelyst/*` packages published. Also has Docker container registry. | | **K8s practice on single VM?** | **Docker Desktop K8s** (Mac/Windows) or **K3s** (Linux). Same manifests scale to AKS/EKS/GKE. | -| **Recommended VM?** | 8 vCPU / 32 GB (min) or 16 vCPU / 64 GB (with Ollama). Hetzner ~$45/mo for dev. | +| **Recommended VM?** | 8 vCPU / 32 GB (min) or 16 vCPU / 64 GB (with Ollama). Hetzner ~€45/mo for dev. | +| **Enhanced tooling?** | See [`SINGLE_VM_ENHANCED_PLAN.md`](SINGLE_VM_ENHANCED_PLAN.md) β€” Coolify, Valkey, Uptime Kuma, SOPS. | | **Time to production K8s?** | Phase 1 (compose) β†’ Phase 2 (Docker Desktop / K3s) β†’ Phase 3 (multi-node) β†’ Phase 4 (managed). Same manifests. | diff --git a/docs/devops/SINGLE_VM_ENHANCED_PLAN.md b/docs/devops/SINGLE_VM_ENHANCED_PLAN.md index 441b2d05..b252a39a 100644 --- a/docs/devops/SINGLE_VM_ENHANCED_PLAN.md +++ b/docs/devops/SINGLE_VM_ENHANCED_PLAN.md @@ -1,24 +1,10 @@ # ByteLyst Ecosystem β€” Enhanced Single-VM Deployment Plan -> Supersedes the stale sections of `SINGLE_VM_DEPLOYMENT.md`. Incorporates lessons from the Gitea registry migration (2026-03-24) and introduces open-source tooling to minimize setup time while maximizing robustness. +> Deploy the entire ByteLyst ecosystem on a single VM using modern open-source tooling. All 10 product repos consume `@bytelyst/*` packages from a local Gitea npm registry (49 packages). All Dockerfiles use pnpm + BuildKit secret mount. 1,591 backend tests green, 9/9 web typechecks clean. --- -## 0. What Changed Since the Original Plan - -The original `SINGLE_VM_DEPLOYMENT.md` was written during the `file:` β†’ registry transition. These items are now **resolved**: - -- βœ… All 10 repos consume `@bytelyst/*` from Gitea npm registry (`^0.1.0`) -- βœ… `docker-prep.sh` deleted from all repos β€” no more tarball prep step -- βœ… All Dockerfiles use pnpm + BuildKit secret mount pattern -- βœ… 49 packages published, 1,591 backend tests green, 9/9 web typechecks clean -- βœ… Docker builds verified for MindLyst + LysnrAI (the two non-standard repos) - -**The prerequisite blocker in Β§4.1 of the original plan is gone.** We can now build any image with just `docker build` + registry auth. - ---- - -## 1. Recommended Open-Source Tooling Additions +## 1. Recommended Open-Source Tooling ### Tier 1 β€” Game Changers (add these first) @@ -392,7 +378,7 @@ sops -d .env.production.enc > .env.production --- -## 9. Revised Implementation Order +## 9. Implementation Order | Step | Time | What | Tools | |------|------|------|-------| @@ -406,11 +392,11 @@ sops -d .env.production.enc > .env.production | **8** | 30 min | Configure Restic backups for all stateful volumes | CLI + cron | | **9** | 30 min | Smoke test: hit all `/health` endpoints, verify Uptime Kuma green | Browser + curl | -**Total: ~4.5 hours** (down from 6-7 hours without Coolify) +**Total: ~4.5 hours** to go from bare VM to fully running ecosystem --- -## 10. VM Sizing (Updated) +## 10. VM Sizing ### Minimum (32 GB) β€” dev/staging, no Ollama @@ -430,7 +416,7 @@ sops -d .env.production.enc > .env.production Same as above + Ollama (~8 GB for llama3:8b) = ~16 GB active, ~48 GB headroom. -### Cloud Pricing (updated) +### Cloud Pricing | Provider | Instance | vCPU | RAM | Price | |----------|----------|------|-----|-------|