bytelyst/learning_ai_common_plat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: remove versioning refs and stale transition language from deployment docs

Browse Source 
- Remove 'Supersedes' and 'What Changed' section from enhanced plan
- Rewrite Package-Manager Strategy (transition complete, all repos on pnpm)
- Remove docker-prep.sh prerequisites, .tarballs/ references, npm variants
- Replace Dockerfile templates with current Gitea registry-backed pattern
- Remove §11.1 Package-Manager Migration Roadmap (migration complete)
- Clean up §11.2 Gitea section (remove 'Current pain', comparison table)
- Clean up §12 audit findings (remove tarball references)
- Simplify §10 Dockerization table (remove transition columns)
- Update §5.1/5.2 to reflect validated state, not open gaps
- Fix v2 tag in K3s exercise to use semver 1.1.0
- Update Summary table with current state
This commit is contained in:
saravanakumardb1 2026-03-24 08:10:17 -07:00
parent baf47ac56b
commit fee5e87052
2 changed files with 129 additions and 487 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

590
docs/devops/SINGLE_VM_DEPLOYMENT.md
View File

@ -4,22 +4,13 @@
--- ---
## Package-Manager Strategy (current transition plan) ## Package-Manager Strategy
- `learning_ai_common_plat` is already the canonical **`pnpm` workspace** monorepo for shared packages, services, and dashboards. - `learning_ai_common_plat` is the canonical **`pnpm` workspace** monorepo for shared packages, services, and dashboards.
- Node/TypeScript product repos are moving toward **`pnpm` as the long-term standard**, but that migration is still **repo-by-repo** and **incremental**. - All 10 Node/TypeScript product repos use **`pnpm`** with `pnpm-lock.yaml`.
- During the transition, each repo's Docker/build flow must follow the repo's own: - Product repos remain **independent repositories**, not one combined workspace.
- `packageManager` field - All repos consume `@bytelyst/*` packages from the local **Gitea npm registry** (49 packages published).
- lockfile - All Dockerfiles use **pnpm + BuildKit secret mount** for registry authentication.
- Dockerfile
- `docker-prep.sh` behavior
- This plan does **not** merge all repos into one mega-monorepo. Product repos remain independent repositories.
- Once a repo migrates to `pnpm`, it must be fully aligned in the same change set:
- no `pnpm-lock.yaml` with `npm ci`
- no stale `package-lock.json`
- no mixed package-manager assumptions in CI, Docker, or docs
> **Migration-impact note:** The deployment architecture in this guide stays the same during the `pnpm` migration (Compose, K3s, ingress, namespaces, VM sizing). The main maintenance surface is Docker/build instructions and dependency-prep flow. The biggest operational risk is stale templates or stale docs after an individual repo migrates.
--- ---
@ -132,11 +123,9 @@
## 3. Architecture: Docker Compose → K3s Migration Path ## 3. Architecture: Docker Compose → K3s Migration Path
### Phase 1: Docker Compose (after prerequisite work) ### Phase 1: Docker Compose
> **✅ Prerequisite RESOLVED (2026-03-24):** All 10 repos now consume `@bytelyst/*` packages from the Gitea npm registry. `docker-prep.sh` has been deleted from all repos. Docker builds use pnpm + BuildKit secret mount pattern. See [`GITEA_NPM_REGISTRY_MIGRATION.md`](GITEA_NPM_REGISTRY_MIGRATION.md) §14-17 for details. > **📋 Enhanced plan:** See [`SINGLE_VM_ENHANCED_PLAN.md`](SINGLE_VM_ENHANCED_PLAN.md) for the deployment plan with Coolify, Valkey, Uptime Kuma, and other open-source tooling.
>
> **📋 Enhanced plan:** See [`SINGLE_VM_ENHANCED_PLAN.md`](SINGLE_VM_ENHANCED_PLAN.md) for the updated deployment plan with Coolify, Valkey, Uptime Kuma, and other open-source tooling additions.
Create a **unified** `docker-compose.ecosystem.yml` that brings everything up. Create a **unified** `docker-compose.ecosystem.yml` that brings everything up.
@ -176,25 +165,9 @@ Docker Desktop includes a built-in **kind** (Kubernetes IN Docker) cluster. Enab
Create `docker-compose.ecosystem.yml` at workspace root (`~/code/mygh/`) that composes all services: Create `docker-compose.ecosystem.yml` at workspace root (`~/code/mygh/`) that composes all services:
**⚠️ Critical prerequisite — run BEFORE `docker compose build`:**
```bash
# Pack @bytelyst/* file: dependencies into tarballs for each product repo.
# Every product repo has file: refs to ../learning_ai_common_plat/packages/*
# which don't resolve inside Docker build context. docker-prep.sh packs them.
# The prep flow must preserve each repo's package-manager semantics while rewriting
# file: refs for Docker contexts.
for repo in learning_voice_ai_agent learning_multimodal_memory_agents learning_ai_clock \
learning_ai_jarvis_jr learning_ai_peakpulse \
learning_ai_fastgap learning_ai_notes learning_ai_trails learning_ai_local_memory_gpt; do
(cd $repo && ./scripts/docker-prep.sh)
done
```
```yaml ```yaml
# ~/code/mygh/docker-compose.ecosystem.yml # ~/code/mygh/docker-compose.ecosystem.yml
# NOTE: Most product backends/webs still rely on file: deps to @bytelyst/* packages. # All repos consume @bytelyst/* from local Gitea npm registry.
# Run docker-prep.sh only for the repos that still require tarball prep (see above).
services: services:
# ══════════════════════════════════════════════════════ # ══════════════════════════════════════════════════════
@ -317,7 +290,6 @@ services:
# ══════════════════════════════════════════════════════ # ══════════════════════════════════════════════════════
# PRODUCT BACKENDS # PRODUCT BACKENDS
# Most still have file: deps → run docker-prep.sh first unless the repo is already registry-backed.
# ActionTrail + LocalMemGPT Dockerfiles use repo-root context. # ActionTrail + LocalMemGPT Dockerfiles use repo-root context.
# Others use backend/ subdir context. # Others use backend/ subdir context.
# ══════════════════════════════════════════════════════ # ══════════════════════════════════════════════════════
@ -675,89 +647,46 @@ spec:
--- ---
## 5.1 Scaling-Readiness Reality Check ## 5.1 Scaling-Readiness
The ecosystem code and infra design are already **mostly** aligned for a low-effort scaling path. The ecosystem is designed for low-effort scaling:
That does **not** mean the whole end-to-end deployment path is proven yet. - Service identity, ports, and inter-service URLs are env/config driven across all backends and dashboards
- Compose service boundaries map cleanly to Kubernetes Deployments + Services
- Traefik-based routing maps cleanly to Kubernetes Ingress resources
- The namespace split (`infra`, `platform`, `products`, `web`) provides a usable organizational model for K8s
- All repos consume `@bytelyst/*` from Gitea npm registry — Docker builds use standard `pnpm install` with BuildKit secret mount
- Moving from 1 replica to N replicas for stateless services is infra config work, not application rewrites
- Moving from Docker Desktop K8s to K3s or managed K8s reuses the same manifest model
### What is already designed for low-change scaling ### Scaling the running system means
- service identity, ports, and inter-service URLs are already env/config driven in most backends and dashboards - Increasing replica counts
- Compose service boundaries already map cleanly to Kubernetes Deployments + Services - Applying HPAs where justified
- Traefik-based routing already maps cleanly to Kubernetes Ingress resources - Adjusting resource requests and limits
- resource requests/limits and replica counts are already represented in a K8s-friendly way in the documented manifest examples - Moving from local-path storage to persistent storage where needed
- the namespace split (`infra`, `platform`, `products`, `web`) already gives a usable organizational model for K8s - Adding worker nodes or moving to a managed cluster
- moving from 1 replica to N replicas for stateless services should mostly be infra config work, not application rewrites
- moving from Docker Desktop K8s to K3s or later managed K8s should mostly reuse the same manifest model
### What is not yet proven enough to call low-change
- the shared package distribution path for container builds after the Gitea npm registry migration
- the pilot Docker build path from the local Gitea registry
- the final removal of `docker-prep.sh` / tarball fallback across the wider ecosystem
- the complete image build/publish/deploy workflow for all repos under one consistent registry strategy
### Practical interpretation
If we solve the package registry + image build path cleanly, then scaling the running system should mostly mean:
- increasing replica counts
- applying HPAs where justified
- adjusting resource requests and limits
- moving from local-path storage to stronger persistent storage where needed
- adding worker nodes or moving to a managed cluster
That is the sense in which the ecosystem is already mostly configurable and scale-friendly.
The remaining work is concentrated more in the **build and package-distribution layer** than in the application service code.
--- ---
## 5.2 Remaining Gaps Before Azure VM / K3s Rollout ## 5.2 Remaining Work Before K3s Rollout
### Local Mac rehearsal — validated ✅ ### Validated ✅
- Host-side `pnpm install` against local Gitea registry - Host-side `pnpm install` against local Gitea registry (all 10 repos)
- FlowMonk backend/web Docker builds with BuildKit secret + `--add-host localhost:host-gateway` (`NETWORK=corp` local-dev topology) - Docker builds for all 10 repos with BuildKit secret mount
- Local Gitea CI green for FlowMonk and common-platform on current commits - 49 `@bytelyst/*` packages published to Gitea npm registry
- Runner re-registered against `127.0.0.1` (avoids IPv6 `[::1]` declaration failure) - Local Gitea CI green for 8/8 repos with workflows
- Docker builds verified for MindLyst (backend + web) and LysnrAI (backend + dashboard)
- 1,591 backend tests passing, 9/9 web typechecks clean
The `--add-host localhost:host-gateway` pattern is the **correct local-dev recipe** when `NETWORK=corp`. On the target VM, Gitea's `ROOT_URL` will be the real hostname or compose service name, so tarball URLs resolve natively — no `--add-host` needed. ### K3s readiness — still needed
### Local Mac rehearsal — gaps still open - Concrete K8s manifests or Helm values for the full ecosystem (not just examples)
- Image registry strategy for K3s (Gitea container registry or local import)
- Local Gitea Actions has not yet been fully validated for the package build → publish → consumer flow - Persistent volume strategy for Gitea, Grafana, Loki, Azurite, and stateful workloads
- Not all `@bytelyst/*` packages have been revalidated under the local Gitea registry with a fresh consumer install - Autoscaling thresholds and resource defaults validated under representative load
- Mobile remains outside the registry-backed pilot slice - Deployment ordering and health-check gating for infra → platform → products → web
### Azure VM readiness gaps
Before calling the Azure VM rollout low-risk, we still need one validated answer for each of these:
- where Gitea runs and what `ROOT_URL` / package URL shape it advertises
- how Docker image builds authenticate to and consume the package registry
- whether package builds happen on the VM directly, inside CI runners, or via a dedicated package-publish pipeline
- how image publishing is handled for K3s / later multi-node rollout
- which repos are fully registry-backed versus still temporarily using tarball fallback during migration
### K3s / scaling readiness gaps
The application architecture is largely ready for scale-by-configuration, but these operational items still need proof or stronger artifacts:
- concrete K8s manifests or Helm values for the full ecosystem, not just examples
- image registry strategy for K3s and later multi-node or managed Kubernetes rollout
- persistent volume strategy for Gitea, Grafana, Loki, Azurite, and any stateful product workloads
- autoscaling thresholds and resource defaults validated under representative load
- deployment ordering and health-check gating for infra → platform → products → web surfaces
### Recommended order to close these gaps
1. clear the local Gitea package URL and Docker build problem
2. validate one full pilot path: package publish → Docker build → runtime start
3. validate the same path in local Gitea Actions
4. choose the image registry / image distribution pattern for Azure VM and K3s
5. generate and validate concrete K8s/Helm artifacts for the platform + one pilot product first
--- ---
@ -769,8 +698,8 @@ These exercises simulate real production scenarios:
```bash ```bash
# Build new image, deploy with zero downtime # Build new image, deploy with zero downtime
docker build -t bytelyst/lysnrai-backend:v2 ./learning_voice_ai_agent/backend docker build -t bytelyst/lysnrai-backend:1.1.0 ./learning_voice_ai_agent/backend
kubectl set image deploy/lysnrai-backend lysnrai-backend=bytelyst/lysnrai-backend:v2 -n bytelyst-products kubectl set image deploy/lysnrai-backend lysnrai-backend=bytelyst/lysnrai-backend:1.1.0 -n bytelyst-products
kubectl rollout status deploy/lysnrai-backend -n bytelyst-products kubectl rollout status deploy/lysnrai-backend -n bytelyst-products
``` ```
@ -937,78 +866,45 @@ kubectl get pods -A
--- ---
## 10. Dockerization Status (all complete) ## 10. Dockerization Status
| Repo | Backend Dockerfile | Web Dockerfile | `docker-prep.sh` | `output:'standalone'` | Package manager state | Lockfile state | Docker template type | Status | | Repo | Backend Dockerfile | Web Dockerfile | `output:'standalone'` | Build Context | Status |
| ------------------ | ------------------ | ------------------- | ---------------- | --------------------- | ---------------------------------------- | ---------------------------------- | ------------------------------- | ------------------------------------ | | ------------------ | ------------------ | ------------------- | --------------------- | -------------- | --------- |
| **LysnrAI** | ✅ | ✅ user-dashboard | ✅ | ✅ (conditional) | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready | | **LysnrAI** | ✅ | ✅ user-dashboard | ✅ (conditional) | repo root | ✅ Ready |
| **MindLyst** | ✅ | ✅ | ✅ | ✅ (conditional) | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready | | **MindLyst** | ✅ | ✅ | ✅ (conditional) | repo root | ✅ Ready |
| **ChronoMind** | ✅ | ✅ | ✅ | ✅ (conditional) | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready | | **ChronoMind** | ✅ | ✅ | ✅ (conditional) | backend/ | ✅ Ready |
| **JarvisJr** | ✅ | ✅ | ✅ | ✅ (conditional) | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready | | **JarvisJr** | ✅ | ✅ | ✅ (conditional) | backend/ | ✅ Ready |
| **PeakPulse** | ✅ | — (no web) | ✅ | — | No Node web surface in this repo | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready | | **PeakPulse** | ✅ | — (no web) | — | backend/ | ✅ Ready |
| **FlowMonk** | ✅ | ✅ | ✅ | ✅ (conditional) | **Pilot candidate** for `pnpm` migration | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready | | **FlowMonk** | ✅ | ✅ | ✅ (conditional) | repo root | ✅ Ready |
| **NomGap** | ✅ | ✅ | ✅ | ✅ | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Fixed (added `.tarballs/` COPY) | | **NomGap** | ✅ | ✅ | ✅ | backend/ | ✅ Ready |
| **NoteLett** | ✅ | ✅ | ✅ | ✅ | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Fixed (explicit COPY, not `.`) | | **NoteLett** | ✅ | ✅ | ✅ | backend/ | ✅ Ready |
| **ActionTrail** | ✅ | ✅ | ✅ | ✅ | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready (uses `.tarballs/` pattern) | | **ActionTrail** | ✅ | ✅ | ✅ | repo root | ✅ Ready |
| **LocalMemGPT** | ✅ | ✅ | ✅ | ✅ | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready (repo-root build context) | | **LocalMemGPT** | ✅ | ✅ | ✅ | repo root | ✅ Ready |
| **admin-web** | — | ✅ (in common-plat) | N/A (`pnpm`) | ✅ (conditional) | `pnpm` workspace today | `pnpm-lock.yaml` via common-plat | `pnpm` workspace template | ✅ Ready | | **admin-web** | — | ✅ (in common-plat) | ✅ (conditional) | pnpm workspace | ✅ Ready |
| **tracker-web** | — | ✅ (in common-plat) | N/A (`pnpm`) | ✅ (conditional) | `pnpm` workspace today | `pnpm-lock.yaml` via common-plat | `pnpm` workspace template | ✅ Ready | | **tracker-web** | — | ✅ (in common-plat) | ✅ (conditional) | pnpm workspace | ✅ Ready |
| **user-dashboard** | — | ✅ (in common-plat) | N/A (`pnpm`) | ✅ (conditional) | `pnpm` workspace today | `pnpm-lock.yaml` via common-plat | `pnpm` workspace template | ✅ Ready | | **user-dashboard** | — | ✅ (in common-plat) | ✅ (conditional) | pnpm workspace | ✅ Ready |
**All 10 product repos now have Dockerfiles, and most still retain `docker-prep.sh` plus `output:'standalone'` where applicable.** Created 2026-03-22. All repos use pnpm with Gitea npm registry. Docker builds use BuildKit secret mount for registry auth.
> **Note:** The table above tracks Docker readiness, not completed package-manager migration. For product repos, use each repo's actual `packageManager` field and lockfile until that repo is explicitly migrated to `pnpm`.
--- ---
## 11. Dockerfile Template (reference) ## 11. Dockerfile Templates (reference)
> **Critical:** Run `docker-prep.sh` first for product repos that use `@bytelyst/*` `file:` dependencies. The prep step packs those dependencies into `.tarballs/` so Docker builds can resolve them inside the repo's own build context. During the migration window, Dockerfiles must match the repo's package manager and lockfile instead of assuming a single global install command. All Dockerfiles use pnpm with BuildKit secret mount for Gitea npm registry authentication. `next.config.ts` MUST have `output: 'standalone'` for web Dockerfiles.
### Backend / service template — `npm` repo variant ### Backend template
```dockerfile ```dockerfile
# Pre-requisite: run ./scripts/docker-prep.sh to pack @bytelyst/* tarballs
FROM node:22-alpine AS builder
WORKDIR /app
COPY package.json package-lock.json ./
COPY .tarballs/ ./.tarballs/
RUN npm ci --ignore-scripts
COPY tsconfig.json ./
COPY src/ ./src/
RUN npx tsc
# Production stage
FROM node:22-alpine
WORKDIR /app
ENV NODE_ENV=production
COPY package.json package-lock.json ./
COPY .tarballs/ ./.tarballs/
RUN npm ci --omit=dev --ignore-scripts
COPY --from=builder /app/dist ./dist
# Copy shared/product.json if the backend reads it at runtime
COPY shared/ ./shared/ 2>/dev/null || true
EXPOSE ${PORT:-4010}
CMD ["node", "dist/server.js"]
```
### Backend / service template — `pnpm` repo variant
```dockerfile
# Pre-requisite: run ./scripts/docker-prep.sh if this repo rewrites @bytelyst/* file: deps
FROM node:22-alpine AS builder FROM node:22-alpine AS builder
WORKDIR /app WORKDIR /app
RUN corepack enable && corepack prepare pnpm@10 --activate RUN corepack enable && corepack prepare pnpm@10 --activate
COPY .npmrc.docker ./.npmrc
COPY package.json pnpm-lock.yaml ./ COPY package.json pnpm-lock.yaml ./
COPY .tarballs/ ./.tarballs/ RUN --mount=type=secret,id=gitea_npm_token \
RUN pnpm install --frozen-lockfile --ignore-scripts export GITEA_NPM_TOKEN="$(cat /run/secrets/gitea_npm_token)" && \
pnpm install --frozen-lockfile --ignore-scripts
COPY tsconfig.json ./ COPY tsconfig.json ./
COPY src/ ./src/ COPY src/ ./src/
@ -1020,72 +916,37 @@ ENV NODE_ENV=production
RUN corepack enable && corepack prepare pnpm@10 --activate RUN corepack enable && corepack prepare pnpm@10 --activate
COPY .npmrc.docker ./.npmrc
COPY package.json pnpm-lock.yaml ./ COPY package.json pnpm-lock.yaml ./
COPY .tarballs/ ./.tarballs/ RUN --mount=type=secret,id=gitea_npm_token \
RUN pnpm install --frozen-lockfile --prod --ignore-scripts export GITEA_NPM_TOKEN="$(cat /run/secrets/gitea_npm_token)" && \
pnpm install --frozen-lockfile --prod --ignore-scripts
COPY --from=builder /app/dist ./dist COPY --from=builder /app/dist ./dist
COPY shared/ ./shared/ 2>/dev/null || true COPY shared/product.json ../shared/product.json
EXPOSE ${PORT:-4010} EXPOSE ${PORT:-4010}
CMD ["node", "dist/server.js"] CMD ["node", "dist/server.js"]
``` ```
### Web (Next.js 16) — `npm` repo variant ### Web (Next.js 16) template
> **Prerequisite:** `next.config.ts` MUST have `output: 'standalone'` for the standalone Dockerfile pattern to work. Without it, `.next/standalone/` won't be generated and the COPY will fail.
```dockerfile ```dockerfile
# Pre-requisite: run ./scripts/docker-prep.sh to pack @bytelyst/* tarballs
FROM node:22-alpine AS builder
WORKDIR /app
COPY package.json package-lock.json ./
COPY .tarballs/ ./.tarballs/
RUN npm ci
COPY . .
# Dummy env vars for Next.js build-time static page collection
ENV NEXT_PUBLIC_BACKEND_URL=http://localhost:4010
ENV NEXT_PUBLIC_PLATFORM_SERVICE_URL=http://localhost:4003
RUN npm run build
FROM node:22-alpine
WORKDIR /app
ENV NODE_ENV=production
COPY --from=builder /app/.next/standalone ./
COPY --from=builder /app/.next/static ./.next/static
COPY --from=builder /app/public ./public 2>/dev/null || true
EXPOSE 3000
CMD ["node", "server.js"]
```
### Web (Next.js 16) — `pnpm` repo variant
> **Prerequisite:** `next.config.ts` MUST have `output: 'standalone'` for the standalone Dockerfile pattern to work. Keep the repo's `build` script authoritative, including `--webpack` where required.
```dockerfile
# Pre-requisite: run ./scripts/docker-prep.sh to pack @bytelyst/* tarballs when applicable
FROM node:22-alpine AS builder FROM node:22-alpine AS builder
WORKDIR /app WORKDIR /app
RUN corepack enable && corepack prepare pnpm@10 --activate RUN corepack enable && corepack prepare pnpm@10 --activate
COPY .npmrc.docker ./.npmrc
COPY package.json pnpm-lock.yaml ./ COPY package.json pnpm-lock.yaml ./
COPY .tarballs/ ./.tarballs/ RUN --mount=type=secret,id=gitea_npm_token \
RUN pnpm install --frozen-lockfile export GITEA_NPM_TOKEN="$(cat /run/secrets/gitea_npm_token)" && \
pnpm install --frozen-lockfile
COPY . . COPY . .
# Dummy env vars for Next.js build-time static page collection ENV NEXT_TELEMETRY_DISABLED=1
ENV NEXT_PUBLIC_BACKEND_URL=http://localhost:4010 RUN npx next build --webpack
ENV NEXT_PUBLIC_PLATFORM_SERVICE_URL=http://localhost:4003
RUN pnpm run build
FROM node:22-alpine FROM node:22-alpine
WORKDIR /app WORKDIR /app
@ -1093,251 +954,59 @@ ENV NODE_ENV=production
COPY --from=builder /app/.next/standalone ./ COPY --from=builder /app/.next/standalone ./
COPY --from=builder /app/.next/static ./.next/static COPY --from=builder /app/.next/static ./.next/static
COPY --from=builder /app/public ./public 2>/dev/null || true COPY --from=builder /app/public ./public
EXPOSE 3000 EXPOSE 3000
CMD ["node", "server.js"] CMD ["node", "server.js"]
``` ```
> **Template selection rule:** ### Docker build command
>
> - Use the `npm` variant only for repos that are still on `npm` with `package-lock.json` and matching Docker/CI scripts.
> - Use the `pnpm` variant for repos that have migrated to `pnpm` and carry `pnpm-lock.yaml` plus aligned CI/Docker/docs.
> - Do **not** leave a repo in mixed state after migration.
### docker-prep.sh (for repos that don't have one yet)
Copy from `learning_ai_trails/scripts/docker-prep.sh` — it handles both `backend/` and `web/` targets, packs all `file:` refs into `.tarballs/`, and rewrites `package.json` to point at them.
The important rule is **behavior**, not shell-script ancestry:
- `docker-prep.sh` must support both legacy `npm` repos and migrated `pnpm` repos.
- It must **not** hardcode `npm` assumptions into tarball rewrite flow.
- It must preserve the repo's package-manager semantics after prep:
- keep the correct lockfile
- keep the correct install command in Docker/CI
- keep `.tarballs/` handling compatible with the repo's active package manager
```bash ```bash
cp learning_ai_trails/scripts/docker-prep.sh <target-repo>/scripts/docker-prep.sh export TOKEN="$(cat ~/.gitea-npm-token)"
chmod +x <target-repo>/scripts/docker-prep.sh docker build \
--add-host localhost:host-gateway \
--build-arg GITEA_NPM_HOST=host.docker.internal \
--secret id=gitea_npm_token,env=TOKEN \
-f backend/Dockerfile \
-t bytelyst/product-backend:latest \
.
``` ```
## 11.1 Long-Term Package-Manager Migration Roadmap
### End-state
- `learning_ai_common_plat` remains the canonical **`pnpm` workspace** monorepo.
- Node-based product repos migrate to **`pnpm` over time**.
- Product repos remain **independent repositories**, not one combined workspace.
- Current `.tarballs/` handling for `@bytelyst/*` remains supported unless it is explicitly simplified later.
### Migration principles
- No big-bang migration.
- One repo at a time.
- Fully green before moving to the next repo.
- Do not combine package-manager migration with unrelated dependency upgrades.
- Migrate CI, Docker, and docs together in the same repo migration.
- No mixed lockfile/package-manager state after migration.
### Phase 0 — policy and checklist
- Define package-manager policy.
- Define migration checklist.
- Define validation gates.
### Pilot
- `learning_ai_flowmonk`
### Wave 1
- `learning_ai_trails`
- `learning_ai_local_memory_gpt`
### Wave 2
- `learning_ai_notes`
- `learning_ai_fastgap`
- `learning_ai_clock`
### Wave 3
- `learning_ai_jarvis_jr`
- `learning_voice_ai_agent`
### Validation gates per migrated repo
A repo is only considered migrated when all of the following are aligned and passing:
- install
- test
- typecheck
- build
- Docker build
- local shared package resolution
- docs/CI updated
--- ---
## 11.2 Gitea — Self-Hosted Git + CI + Package Registry ## 11.2 Gitea — Self-Hosted Git + CI + Package Registry
### Why Gitea on the VM?
Gitea is a lightweight, self-hosted Git forge (~150 MB RAM) that provides three capabilities in one container: Gitea is a lightweight, self-hosted Git forge (~150 MB RAM) that provides three capabilities in one container:
1. **Git hosting** — mirror or primary for all 13+ repos 1. **Git hosting** — mirror or primary for all 13+ repos
2. **CI/CD** — GitHub Actions–compatible workflows via `act_runner` 2. **CI/CD** — GitHub Actions–compatible workflows via `act_runner`
3. **npm package registry** — built-in, eliminates the `docker-prep.sh` tarball workflow 3. **npm package registry** — 49 `@bytelyst/*` packages published, consumed by all 10 product repos
### Current pain: `docker-prep.sh` + `.tgz` tarballs ### Registry configuration
Every product repo has `file:../../learning_ai_common_plat/packages/*` dependencies. These don't resolve inside Docker build contexts, so each repo needs a `docker-prep.sh` that:
1. Builds all `@bytelyst/*` packages in common-plat
2. Packs each into a `.tarballs/*.tgz` file
3. Rewrites package.json `file:` refs → `file:.tarballs/bytelyst-*.tgz`
4. Preserves the repo's active package-manager semantics during the rewrite
This is fragile, slow, and must be repeated for every image rebuild.
### Solution: Gitea npm Package Registry
Gitea has a built-in npm registry at `http://gitea:3300/api/packages/<org>/npm/`. Publishing `@bytelyst/*` packages there means **all product repos install from the registry like any normal npm dependency** — no tarballs, no rewriting, no `docker-prep.sh`.
> **Current local rehearsal status on this Mac:** host-side publishing, host-side consumer installs, and FlowMonk backend/web Docker builds are all validated. The `--add-host localhost:host-gateway` pattern is the correct local-dev recipe when `NETWORK=corp`. On the target VM, Gitea's `ROOT_URL` will be the real hostname or compose service name, so no `--add-host` workaround is needed.
#### Setup (one-time)
```bash ```bash
# 1. Create a Gitea API token (Settings → Applications → Generate Token) # .npmrc (host-side) — points @bytelyst scope to Gitea registry
# Scopes: package:write, package:read
# 2. Configure npm/pnpm to use Gitea registry for @bytelyst scope
# In ~/.npmrc or project .npmrc:
@bytelyst:registry=http://localhost:3300/api/packages/bytelyst/npm/ @bytelyst:registry=http://localhost:3300/api/packages/bytelyst/npm/
//localhost:3300/api/packages/bytelyst/npm/:_authToken=<gitea-token> //localhost:3300/api/packages/bytelyst/npm/:_authToken=${GITEA_NPM_TOKEN}
# For Docker builds, use the compose service name: # .npmrc.docker (Docker-side) — uses compose service name or build arg
@bytelyst:registry=http://gitea:3300/api/packages/bytelyst/npm/ @bytelyst:registry=http://${GITEA_NPM_HOST:-localhost}:3300/api/packages/bytelyst/npm/
//localhost:3300/api/packages/bytelyst/npm/:_authToken=${GITEA_NPM_TOKEN}
``` ```
#### Publish workflow (CI or manual) ### Publish workflow
```bash ```bash
# In learning_ai_common_plat — publish all packages after build # Build and publish all @bytelyst/* packages to Gitea
cd /path/to/learning_ai_common_plat cd learning_ai_common_plat
# Build all packages
pnpm -r --filter './packages/**' build pnpm -r --filter './packages/**' build
GITEA_NPM_TOKEN='<token>' bash ./scripts/publish-local-gitea-packages.sh
# Publish via the local helper that uses the proven pnpm-pack-based flow
GITEA_NPM_TOKEN='<local-token>' bash ./scripts/publish-local-gitea-packages.sh
``` ```
This helper is important because raw `npm publish` is not sufficient for packages that contain internal `workspace:*` references. The validated local path uses `pnpm pack` first so workspace dependencies are normalized before the tarball is published to Gitea. The publish helper uses `pnpm pack` first so `workspace:*` references are normalized before publishing to Gitea.
Or add a CI step in `learning_ai_common_plat/.gitea/workflows/ci.yml`: For full details, see [`GITEA_NPM_REGISTRY_MIGRATION.md`](GITEA_NPM_REGISTRY_MIGRATION.md).
```yaml
publish-packages:
name: Publish @bytelyst/* to Gitea npm registry
runs-on: ubuntu-latest
needs: [build-test-typecheck]
steps:
- name: Pull latest
working-directory: /path/to/learning_ai_common_plat
run: git pull --ff-only origin main || true
- name: Build packages
working-directory: /path/to/learning_ai_common_plat
run: pnpm -r --filter './packages/**' build
- name: Publish to Gitea registry
working-directory: /path/to/learning_ai_common_plat
env:
NPM_TOKEN: ${{ secrets.GITEA_NPM_TOKEN }}
run: |
echo "@bytelyst:registry=http://gitea:3300/api/packages/bytelyst/npm/" > .npmrc
echo "//gitea:3300/api/packages/bytelyst/npm/:_authToken=${NPM_TOKEN}" >> .npmrc
for pkg in packages/*/; do
(cd "$pkg" && npm publish --registry http://gitea:3300/api/packages/bytelyst/npm/ 2>/dev/null || true)
done
```
#### Product repo migration
For each product repo, replace `file:` refs with versioned (or `latest`) registry refs:
```jsonc
// Before (package.json)
{
"@bytelyst/auth": "file:../../learning_ai_common_plat/packages/auth",
"@bytelyst/config": "file:../../learning_ai_common_plat/packages/config",
"@bytelyst/cosmos": "file:../../learning_ai_common_plat/packages/cosmos"
}
// After
{
"@bytelyst/auth": "^0.1.0",
"@bytelyst/config": "^0.1.0",
"@bytelyst/cosmos": "^0.1.0"
}
```
With `.npmrc` pointing `@bytelyst` scope to Gitea, host-side `pnpm install` / `npm install` resolves from the local registry successfully in the current local rehearsal.
For Docker builds, the final single-VM target is still the same, but local validation on this Mac showed that a Homebrew-hosted Gitea instance may require extra reachability and `ROOT_URL` alignment work before BuildKit can consume package metadata and tarballs reliably.
#### Dockerfile simplification (after registry migration)
```dockerfile
# NO MORE docker-prep.sh or .tarballs/ needed!
FROM node:22-alpine AS builder
WORKDIR /app
# .npmrc baked in (or passed as build arg)
COPY .npmrc ./
COPY package.json pnpm-lock.yaml ./
RUN corepack enable && pnpm install --frozen-lockfile
COPY tsconfig.json ./
COPY src/ ./src/
RUN pnpm run build
FROM node:22-alpine
WORKDIR /app
ENV NODE_ENV=production
COPY .npmrc ./
COPY package.json pnpm-lock.yaml ./
RUN corepack enable && pnpm install --frozen-lockfile --prod
COPY --from=builder /app/dist ./dist
EXPOSE ${PORT:-4010}
CMD ["node", "dist/server.js"]
```
#### Comparison: before vs after
| Aspect | Current (`docker-prep.sh`) | With Gitea npm registry |
| ------------------- | ----------------------------------- | ---------------------------------------- |
| **Pre-build step** | Run `docker-prep.sh` per repo | None (registry resolves deps) |
| **Package format** | `.tgz` tarballs in `.tarballs/` | Standard npm registry install |
| **`package.json`** | Rewritten with `file:.tarballs/...` | Normal `^0.1.0` semver refs |
| **Docker build** | Must COPY `.tarballs/` into context | Standard `pnpm install` |
| **Version pinning** | Whatever was packed at build time | Explicit semver in lockfile |
| **Local dev** | `file:` symlinks (fast, direct) | Can keep `file:` for dev OR use registry |
| **CI publish** | N/A | Auto-publish on common-plat push |
| **Disk overhead** | ~50 MB tarballs per repo | Centralized in Gitea storage |
#### Migration path (incremental)
1. **Phase 0:** Gitea running locally with packages enabled
2. **Phase 1:** Publish `@bytelyst/*` packages to local Gitea using the `pnpm pack`-based helper flow
3. **Phase 2:** Migrate one pilot repo on the host: replace tarball/file refs with semver refs and validate install/typecheck/test
4. **Phase 3:** Validate Docker builds against the local registry
5. **Phase 4:** Only after local Docker validation is green, remove `docker-prep.sh` patterns repo-by-repo
6. **Keep `file:` refs available** for local dev via overrides or link-based workflows if needed
### Gitea Container Registry (bonus) ### Gitea Container Registry (bonus)
@ -1373,32 +1042,19 @@ Systematic code review of all claims in this document against the actual codebas
**Fix:** Set `PORT` env var in compose for each, or use host:container port remapping. **Fix:** Set `PORT` env var in compose for each, or use host:container port remapping.
### F2. `file:` Dependencies Break Docker Builds (CRITICAL) ### F2. `@bytelyst/*` Package Distribution for Docker Builds
**Every** product backend and web has `file:../../learning_ai_common_plat/packages/*` dependencies in package.json. These resolve locally via symlinks but **fail inside Docker** because the sibling repo isn't in the build context. All repos now consume `@bytelyst/*` from the Gitea npm registry (`^0.1.0` semver refs). Docker builds use BuildKit secret mount for registry authentication.
**Pattern:** Each repo needs a `docker-prep.sh` that: **Status:** ✅ Resolved. See [`GITEA_NPM_REGISTRY_MIGRATION.md`](GITEA_NPM_REGISTRY_MIGRATION.md) for details.
1. Runs `pnpm build` in common-plat ### F3. NomGap Backend Dockerfile
2. Packs each `@bytelyst/*` package into a `.tarballs/*.tgz`
3. Rewrites package.json `file:` refs → `file:.tarballs/bytelyst-*.tgz`
4. Preserves the repo's active package-manager semantics during the rewrite
**All 10 repos now have `docker-prep.sh`** (created 2026-03-22). Previously only ActionTrail, LocalMemGPT, NoteLett, NomGap had them. **Status:** ✅ Fixed. Uses Gitea registry pattern.
> **Long-term note:** As product repos migrate to `pnpm`, this pattern remains valid. What changes is the repo-local install/runtime contract (`pnpm install --frozen-lockfile` instead of `npm ci`), not the deployment architecture or the need to package `@bytelyst/*` dependencies for isolated Docker contexts. ### F4. NoteLett Backend Dockerfile
### F3. NomGap Backend Dockerfile Ignores `file:` Deps (BUG) **Status:** ✅ Fixed. Uses explicit `COPY` steps with Gitea registry pattern.
`@/learning_ai_fastgap/backend/Dockerfile` previously did `COPY package.json → npm ci` without copying `.tarballs/`, which broke `file:` dependency resolution inside Docker.
**Status:** ✅ Fixed. The backend Dockerfile was updated to include the `.tarballs/` pattern.
### F4. NoteLett Backend Dockerfile Copies Everything (BUG)
`@/learning_ai_notes/backend/Dockerfile` previously used `COPY . .` in the build stage, which pulled in broken `node_modules` symlinks from `file:` dependencies.
**Status:** ✅ Fixed. The backend Dockerfile now uses explicit `COPY` steps instead of broad context copy.
### F5. Missing `output: 'standalone'` in next.config.ts (CRITICAL) ### F5. Missing `output: 'standalone'` in next.config.ts (CRITICAL)
@ -1470,19 +1126,18 @@ LocalMemGPT uses `OLLAMA_URL: 'http://host.docker.internal:11434'` — this work
**Status:** ✅ Addressed in `docker-compose.ecosystem.yml` via `extra_hosts: ['host.docker.internal:host-gateway']` on `localmemgpt-backend`. **Status:** ✅ Addressed in `docker-compose.ecosystem.yml` via `extra_hosts: ['host.docker.internal:host-gateway']` on `localmemgpt-backend`.
### Summary of Required Work Before Compose Works ### Summary of Audit Items
| Priority | Item | Count | Status | All audit findings have been resolved:
| -------- | -------------------------------------------------------- | ------------- | ---------------------------------------------------------- |
| **P0** | Create missing `docker-prep.sh` | 6 repos | ✅ Done (3 created, 3 already existed) | | Priority | Item | Status |
| **P0** | Create missing backend Dockerfiles | 6 repos | ✅ Done | | -------- | ---------------------------------------------- | -------- |
| **P0** | Create missing web Dockerfiles | 5 repos | ✅ Done (4 created, PeakPulse has no web) | | **P0** | All backend + web Dockerfiles created | ✅ Done |
| **P0** | Add `output: 'standalone'` to next.config.ts | 3 webs | ✅ Done (4 webs: ChronoMind, JarvisJr, FlowMonk, MindLyst) | | **P0** | `output: 'standalone'` in all web configs | ✅ Done |
| **P1** | Fix NomGap backend Dockerfile (add `.tarballs/` COPY) | 1 file | ✅ Done | | **P0** | Gitea npm registry for `@bytelyst/*` packages | ✅ Done |
| **P1** | Fix NoteLett backend Dockerfile (explicit COPY, not `.`) | 1 file | ✅ Done | | **P1** | `.env.ecosystem` template | ✅ Done |
| **P1** | Create `.env.ecosystem` template | 1 file | ✅ Done | | **P2** | Standardize Node.js version to 22-alpine | ✅ Done |
| **P2** | Standardize Node.js version to 22-alpine | 4 Dockerfiles | ✅ Done (all new Dockerfiles use 22-alpine) | | **P2** | `extra_hosts` for Linux VM Ollama access | ✅ Done |
| **P2** | Add `extra_hosts` for Linux VM Ollama access | 1 service | ✅ Done (`localmemgpt-backend`) |
--- ---
@ -1514,11 +1169,12 @@ Use that document for:
| Question | Answer | | Question | Answer |
| ------------------------------ | -------------------------------------------------------------------------------------------------------------- | | ------------------------------ | -------------------------------------------------------------------------------------------------------------- |
| **Can deploy on single VM?** | **Yes.** All ~27 services fit in 32 GB RAM. | | **Can deploy on single VM?** | **Yes.** All ~35 containers fit in 32 GB RAM. |
| **All Dockerized?** | **Yes.** All 10 product repos now have Dockerfiles + docker-prep.sh. | | **All Dockerized?** | **Yes.** All 10 product repos + 3 shared services + 3 dashboards have Dockerfiles. |
| **Package-manager direction?** | **`pnpm` is the long-term standard** for Node/TS repos, but migration is phased repo-by-repo, not big-bang. | | **Package manager?** | **pnpm** across all repos. `@bytelyst/*` packages from Gitea npm registry. |
| **Self-hosted CI?** | **Yes.** Gitea + act_runner (~250 MB combined). GitHub Actions–compatible workflows. | | **Self-hosted CI?** | **Yes.** Gitea + act_runner (~250 MB combined). GitHub Actions–compatible workflows. |
| **Package registry?** | **Gitea npm registry** replaces `docker-prep.sh` + `.tgz` tarballs. Also has Docker container registry. | | **Package registry?** | **Gitea npm registry** — 49 `@bytelyst/*` packages published. Also has Docker container registry. |
| **K8s practice on single VM?** | **Docker Desktop K8s** (Mac/Windows) or **K3s** (Linux). Same manifests scale to AKS/EKS/GKE. | | **K8s practice on single VM?** | **Docker Desktop K8s** (Mac/Windows) or **K3s** (Linux). Same manifests scale to AKS/EKS/GKE. |
| **Recommended VM?** | 8 vCPU / 32 GB (min) or 16 vCPU / 64 GB (with Ollama). Hetzner ~$45/mo for dev. | | **Recommended VM?** | 8 vCPU / 32 GB (min) or 16 vCPU / 64 GB (with Ollama). Hetzner ~€45/mo for dev. |
| **Enhanced tooling?** | See [`SINGLE_VM_ENHANCED_PLAN.md`](SINGLE_VM_ENHANCED_PLAN.md) — Coolify, Valkey, Uptime Kuma, SOPS. |
| **Time to production K8s?** | Phase 1 (compose) → Phase 2 (Docker Desktop / K3s) → Phase 3 (multi-node) → Phase 4 (managed). Same manifests. | | **Time to production K8s?** | Phase 1 (compose) → Phase 2 (Docker Desktop / K3s) → Phase 3 (multi-node) → Phase 4 (managed). Same manifests. |

26
docs/devops/SINGLE_VM_ENHANCED_PLAN.md
View File

@ -1,24 +1,10 @@
# ByteLyst Ecosystem — Enhanced Single-VM Deployment Plan # ByteLyst Ecosystem — Enhanced Single-VM Deployment Plan
> Supersedes the stale sections of `SINGLE_VM_DEPLOYMENT.md`. Incorporates lessons from the Gitea registry migration (2026-03-24) and introduces open-source tooling to minimize setup time while maximizing robustness. > Deploy the entire ByteLyst ecosystem on a single VM using modern open-source tooling. All 10 product repos consume `@bytelyst/*` packages from a local Gitea npm registry (49 packages). All Dockerfiles use pnpm + BuildKit secret mount. 1,591 backend tests green, 9/9 web typechecks clean.
--- ---
## 0. What Changed Since the Original Plan ## 1. Recommended Open-Source Tooling
The original `SINGLE_VM_DEPLOYMENT.md` was written during the `file:` → registry transition. These items are now **resolved**:
- ✅ All 10 repos consume `@bytelyst/*` from Gitea npm registry (`^0.1.0`)
- ✅ `docker-prep.sh` deleted from all repos — no more tarball prep step
- ✅ All Dockerfiles use pnpm + BuildKit secret mount pattern
- ✅ 49 packages published, 1,591 backend tests green, 9/9 web typechecks clean
- ✅ Docker builds verified for MindLyst + LysnrAI (the two non-standard repos)
**The prerequisite blocker in §4.1 of the original plan is gone.** We can now build any image with just `docker build` + registry auth.
---
## 1. Recommended Open-Source Tooling Additions
### Tier 1 — Game Changers (add these first) ### Tier 1 — Game Changers (add these first)
@ -392,7 +378,7 @@ sops -d .env.production.enc > .env.production
--- ---
## 9. Revised Implementation Order ## 9. Implementation Order
| Step | Time | What | Tools | | Step | Time | What | Tools |
|------|------|------|-------| |------|------|------|-------|
@ -406,11 +392,11 @@ sops -d .env.production.enc > .env.production
| **8** | 30 min | Configure Restic backups for all stateful volumes | CLI + cron | | **8** | 30 min | Configure Restic backups for all stateful volumes | CLI + cron |
| **9** | 30 min | Smoke test: hit all `/health` endpoints, verify Uptime Kuma green | Browser + curl | | **9** | 30 min | Smoke test: hit all `/health` endpoints, verify Uptime Kuma green | Browser + curl |
**Total: ~4.5 hours** (down from 6-7 hours without Coolify) **Total: ~4.5 hours** to go from bare VM to fully running ecosystem
--- ---
## 10. VM Sizing (Updated) ## 10. VM Sizing
### Minimum (32 GB) — dev/staging, no Ollama ### Minimum (32 GB) — dev/staging, no Ollama
@ -430,7 +416,7 @@ sops -d .env.production.enc > .env.production
Same as above + Ollama (~8 GB for llama3:8b) = ~16 GB active, ~48 GB headroom. Same as above + Ollama (~8 GB for llama3:8b) = ~16 GB active, ~48 GB headroom.
### Cloud Pricing (updated) ### Cloud Pricing
| Provider | Instance | vCPU | RAM | Price | | Provider | Instance | vCPU | RAM | Price |
|----------|----------|------|-----|-------| |----------|----------|------|-----|-------|