docs(devops): record local gitea registry rehearsal
This commit is contained in:
saravanakumardb1
parent
1d2003a30f
commit
7116749bbd
490
docs/devops/GITEA_NPM_REGISTRY_MIGRATION.md
Normal file
490
docs/devops/GITEA_NPM_REGISTRY_MIGRATION.md
Normal file
@ -0,0 +1,490 @@
|
||||
# Gitea npm Package Registry Migration
|
||||
|
||||
> **Goal:** Replace the `docker-prep.sh` + tarball workflow with Gitea's built-in npm package registry for `@bytelyst/*` shared packages.
|
||||
>
|
||||
> **Phase policy:** use **this Mac as the future single VM** and validate the entire flow locally first. No Azure rollout and no publishing to any external or corporate registry in this phase.
|
||||
|
||||
---
|
||||
|
||||
## 1. Decision
|
||||
|
||||
We should adopt the local Gitea npm registry as the long-term package-distribution path for `@bytelyst/*` packages.
|
||||
|
||||
We should **not** switch directly in Azure first.
|
||||
|
||||
Instead, we should validate the exact end-to-end flow locally on this Mac:
|
||||
|
||||
- local Gitea git hosting
|
||||
- local Gitea Actions
|
||||
- local Gitea npm package registry
|
||||
- local Docker / Docker Compose builds
|
||||
- local product repos consuming `@bytelyst/*` from local Gitea
|
||||
|
||||
Once that is stable, we replicate the same validated shape on the Azure single VM.
|
||||
|
||||
---
|
||||
|
||||
## 2. Why We Want This
|
||||
|
||||
### Current pain
|
||||
|
||||
Today many repos still depend on one of these Docker-time workarounds:
|
||||
|
||||
- `file:` references to sibling `learning_ai_common_plat/packages/*`
|
||||
- `docker-prep.sh`
|
||||
- `.docker-deps` or older `.tarballs` copy steps
|
||||
- temporary `package.json` rewriting before builds
|
||||
|
||||
This works, but it creates repeated operational drag:
|
||||
|
||||
- every build may need prep
|
||||
- Dockerfiles become repo-specific and fragile
|
||||
- CI, local development, and Docker are not using one clean dependency source
|
||||
- path and build-context mismatches keep surfacing during ecosystem validation
|
||||
|
||||
### Target outcome
|
||||
|
||||
We want one dependency source for shared packages:
|
||||
|
||||
- local development
|
||||
- local CI via Gitea Actions
|
||||
- local Docker builds
|
||||
- later Azure single-VM builds
|
||||
|
||||
That dependency source should be the **local Gitea npm registry**.
|
||||
|
||||
---
|
||||
|
||||
## 3. Local-First Target Architecture
|
||||
|
||||
### 3.1 This Mac as the rehearsal VM
|
||||
|
||||
```text
|
||||
This Mac
|
||||
├── local Gitea (git + actions + npm registry)
|
||||
├── local Docker / Docker Compose
|
||||
├── learning_ai_common_plat
|
||||
└── product repos
|
||||
```
|
||||
|
||||
Flow:
|
||||
|
||||
```text
|
||||
learning_ai_common_plat/packages/*
|
||||
↓ build locally
|
||||
↓ publish to local Gitea npm registry
|
||||
local Gitea npm registry (localhost:3300)
|
||||
↓ install via semver refs
|
||||
product repos / Docker builds / local Gitea CI
|
||||
```
|
||||
|
||||
### 3.2 Future Azure single VM
|
||||
|
||||
After the local rehearsal is proven, Azure should mirror the same shape:
|
||||
|
||||
```text
|
||||
Azure VM
|
||||
├── Gitea
|
||||
├── Gitea Actions runner
|
||||
├── Gitea npm registry
|
||||
├── Docker / Compose
|
||||
└── same repo + build flow
|
||||
```
|
||||
|
||||
The Azure plan should reuse the validated local approach, not invent a second design.
|
||||
|
||||
---
|
||||
|
||||
## 4. What Is In Scope Right Now
|
||||
|
||||
### Included
|
||||
|
||||
- validate local Gitea package-registry access
|
||||
- validate local Gitea Actions availability
|
||||
- define local-only package publish flow to local Gitea
|
||||
- define local-only consumer install flow from local Gitea
|
||||
- define local-only Docker build flow from local Gitea
|
||||
- define the later Azure single-VM replication steps
|
||||
|
||||
### Excluded
|
||||
|
||||
- Azure changes right now
|
||||
- external npm registry publishing
|
||||
- corporate registry publishing
|
||||
- non-local deployment actions
|
||||
- K3s / Helm work
|
||||
|
||||
---
|
||||
|
||||
## 5. Current Local Readiness Snapshot
|
||||
|
||||
Validated locally on this Mac:
|
||||
|
||||
- local Gitea health endpoint responds
|
||||
- local Gitea version endpoint responds
|
||||
- local `bytelyst` user auth works
|
||||
- local Gitea Actions workflow exists for `learning_ai_common_plat`
|
||||
- local Gitea package registry endpoint is reachable
|
||||
- local pilot packages have been published to local Gitea successfully
|
||||
- a clean scratch consumer install from local Gitea works on the host
|
||||
|
||||
That is enough to treat this Mac as the single-VM rehearsal environment.
|
||||
|
||||
---
|
||||
|
||||
## 5.1 Validation Results So Far
|
||||
|
||||
### Proven locally on this Mac
|
||||
|
||||
- shared packages build cleanly in `learning_ai_common_plat`
|
||||
- a local-only Gitea package token was created and used successfully
|
||||
- pilot packages were published to local Gitea npm registry successfully
|
||||
- a clean scratch `pnpm install` from the local Gitea registry worked on the host
|
||||
- `learning_ai_flowmonk` was migrated on the host from tarball refs to semver `@bytelyst/*` refs
|
||||
- `learning_ai_flowmonk` host-side install, typecheck, and tests passed against the local Gitea registry
|
||||
|
||||
### Important implementation finding
|
||||
|
||||
Publishing `@bytelyst/*` workspace packages with raw `npm publish` is not sufficient for this migration.
|
||||
|
||||
Why:
|
||||
|
||||
- packages with internal `workspace:*` dependencies can be published with unresolved workspace specifiers
|
||||
- those published artifacts break downstream consumers
|
||||
|
||||
What worked:
|
||||
|
||||
- `pnpm pack` produces normalized package metadata for workspace dependencies
|
||||
- publishing the normalized tarball to Gitea works
|
||||
|
||||
This is now codified in the local-only helper script:
|
||||
|
||||
`scripts/publish-local-gitea-packages.sh`
|
||||
|
||||
That script is currently the authoritative local rehearsal path for publishing `@bytelyst/*` packages to local Gitea.
|
||||
|
||||
### Current blocker
|
||||
|
||||
The remaining blocker is not host-side package publishing or host-side package consumption.
|
||||
|
||||
The blocker is specifically:
|
||||
|
||||
- Docker / BuildKit installs against the local Homebrew Gitea instance on this Mac
|
||||
- registry metadata and tarball fetch behavior across the Mac host ↔ Docker build boundary
|
||||
|
||||
At the time of writing:
|
||||
|
||||
- host-side registry usage is validated
|
||||
- FlowMonk host-side pilot migration is validated
|
||||
- Docker-side local-registry validation is still blocked
|
||||
|
||||
That blocker must be resolved before we can claim full local E2E completion.
|
||||
|
||||
---
|
||||
|
||||
## 6. Migration Strategy
|
||||
|
||||
### Stage A — Local registry rehearsal
|
||||
|
||||
Validate the package-registry path locally before changing every repo.
|
||||
|
||||
Required outcomes:
|
||||
|
||||
1. local package token exists
|
||||
2. a small pilot set of `@bytelyst/*` packages can be published to local Gitea
|
||||
3. a local consumer can install them via semver refs
|
||||
4. a Docker build can install them without `docker-prep.sh`
|
||||
5. local Gitea Actions can run the same package build/publish path
|
||||
|
||||
### Stage B — Pilot repo migration
|
||||
|
||||
Migrate one repo first.
|
||||
|
||||
Recommended pilot:
|
||||
|
||||
- `learning_ai_flowmonk`
|
||||
|
||||
Reason:
|
||||
|
||||
- recent Docker/build work already exposed its edge cases
|
||||
- both backend and web surfaces exist
|
||||
- it is a strong canary for dependency-distribution changes
|
||||
|
||||
### Stage C — Sequential ecosystem rollout
|
||||
|
||||
After the pilot is stable, migrate remaining repos sequentially.
|
||||
|
||||
---
|
||||
|
||||
## 7. Local Implementation Plan
|
||||
|
||||
### 7.1 Verify package metadata in `learning_ai_common_plat`
|
||||
|
||||
Every `packages/*/package.json` should have:
|
||||
|
||||
- a valid `name`
|
||||
- a valid `version`
|
||||
- a valid `files` / `exports` setup
|
||||
- a successful local `build`
|
||||
|
||||
Local validation command:
|
||||
|
||||
```bash
|
||||
cd /Users/sd9235/code/mygh/learning_ai_common_plat
|
||||
pnpm -r --filter './packages/*' build
|
||||
```
|
||||
|
||||
### 7.2 Create a local-only Gitea package token
|
||||
|
||||
Use a token intended only for this Mac rehearsal.
|
||||
|
||||
Required scopes:
|
||||
|
||||
- package read
|
||||
- package write
|
||||
|
||||
Do not use Azure or external registry credentials here.
|
||||
|
||||
### 7.3 Publish only to local Gitea
|
||||
|
||||
In this phase, all `npm publish` activity must point only to:
|
||||
|
||||
`http://localhost:3300/api/packages/bytelyst/npm/`
|
||||
|
||||
No Azure Artifacts.
|
||||
No npmjs.org.
|
||||
No external corporate npm registry.
|
||||
|
||||
Use the local-only helper script for this rehearsal:
|
||||
|
||||
```bash
|
||||
cd /Users/sd9235/code/mygh/learning_ai_common_plat
|
||||
GITEA_NPM_TOKEN='<local-token>' bash ./scripts/publish-local-gitea-packages.sh
|
||||
```
|
||||
|
||||
For a single package:
|
||||
|
||||
```bash
|
||||
cd /Users/sd9235/code/mygh/learning_ai_common_plat
|
||||
GITEA_NPM_TOKEN='<local-token>' bash ./scripts/publish-local-gitea-packages.sh '@bytelyst/errors'
|
||||
```
|
||||
|
||||
### 7.4 Start with a minimal package pilot set
|
||||
|
||||
Publish a small set first:
|
||||
|
||||
- `@bytelyst/errors`
|
||||
- `@bytelyst/config`
|
||||
- `@bytelyst/api-client`
|
||||
|
||||
Then expand once the local install path is proven.
|
||||
|
||||
### 7.5 Consumer dependency model after migration
|
||||
|
||||
Consumers move from local file refs:
|
||||
|
||||
```json
|
||||
"@bytelyst/auth": "file:../../learning_ai_common_plat/packages/auth"
|
||||
```
|
||||
|
||||
to semver refs:
|
||||
|
||||
```json
|
||||
"@bytelyst/auth": "^0.1.0"
|
||||
```
|
||||
|
||||
And resolve via scoped `.npmrc` or `.pnpmrc` config pointing `@bytelyst` to local Gitea.
|
||||
|
||||
### 7.6 Docker model after migration
|
||||
|
||||
After a repo is migrated to the registry model, its Dockerfiles should:
|
||||
|
||||
- stop copying `.docker-deps` or `.tarballs`
|
||||
- stop depending on `docker-prep.sh`
|
||||
- install from local Gitea through scoped registry config
|
||||
|
||||
That is the main operational simplification we want.
|
||||
|
||||
---
|
||||
|
||||
## 8. Local Validation Sequence
|
||||
|
||||
### Step 1 — Validate local Gitea surfaces
|
||||
|
||||
Confirm locally:
|
||||
|
||||
- Gitea UI is reachable
|
||||
- package registry API is reachable with auth
|
||||
- Actions workflow is visible and active
|
||||
|
||||
### Step 2 — Validate package build output
|
||||
|
||||
Run local builds for the shared packages and confirm pack/publish inputs are clean.
|
||||
|
||||
### Step 3 — Publish a minimal package pilot set to local Gitea only
|
||||
|
||||
Success means:
|
||||
|
||||
- packages appear in local Gitea package list
|
||||
- metadata looks correct
|
||||
- no external registry interaction occurred
|
||||
|
||||
### Step 4 — Validate local install from Gitea
|
||||
|
||||
Use a scratch consumer or a pilot repo surface and verify:
|
||||
|
||||
- `pnpm install` resolves `@bytelyst/*` from local Gitea
|
||||
- lockfile updates are clean
|
||||
- no fallback to tarballs is required for the tested packages
|
||||
|
||||
### Step 5 — Validate Docker build from local Gitea
|
||||
|
||||
For the pilot repo:
|
||||
|
||||
- remove the tested surface's dependency on `docker-prep.sh`
|
||||
- add scoped registry config for Docker
|
||||
- build backend and web without tarball prep
|
||||
|
||||
Success means:
|
||||
|
||||
- image build works without `.docker-deps` or `.tarballs`
|
||||
- build does not require local package rewrite logic
|
||||
|
||||
Current status on this Mac:
|
||||
|
||||
- **not yet fully green**
|
||||
- the remaining issue is the local Homebrew Gitea instance serving package metadata/tarball URLs across the Docker build boundary
|
||||
- host-side validation is complete, Docker-side validation is still open
|
||||
|
||||
### Step 6 — Validate local Gitea Actions path
|
||||
|
||||
Add or adapt a local-only workflow in `learning_ai_common_plat` that:
|
||||
|
||||
- builds packages
|
||||
- optionally publishes to local Gitea npm registry
|
||||
- never targets an external registry
|
||||
|
||||
### Step 7 — Validate full local E2E
|
||||
|
||||
At the end of the local rehearsal we should be able to say:
|
||||
|
||||
- shared packages build locally
|
||||
- shared packages publish to local Gitea registry
|
||||
- a pilot consumer installs from local Gitea
|
||||
- a pilot Docker build succeeds from local Gitea
|
||||
- local Gitea Actions can drive the same path
|
||||
|
||||
Only then should we expand repo-by-repo.
|
||||
|
||||
---
|
||||
|
||||
## 9. Recommended Rollout Order
|
||||
|
||||
### Pilot
|
||||
|
||||
1. `learning_ai_flowmonk`
|
||||
|
||||
### Then
|
||||
|
||||
2. `learning_ai_local_memory_gpt`
|
||||
3. `learning_ai_notes`
|
||||
4. `learning_ai_trails`
|
||||
5. `learning_ai_fastgap`
|
||||
6. `learning_ai_clock`
|
||||
7. `learning_ai_jarvis_jr`
|
||||
8. `learning_ai_peakpulse`
|
||||
9. `learning_multimodal_memory_agents`
|
||||
10. `learning_voice_ai_agent`
|
||||
|
||||
The ordering prioritizes recently exercised Docker paths before higher blast-radius repos.
|
||||
|
||||
---
|
||||
|
||||
## 10. CI Model For The Local Rehearsal
|
||||
|
||||
For the local rehearsal phase, Gitea Actions should do only local work:
|
||||
|
||||
- build packages
|
||||
- test packages
|
||||
- typecheck packages
|
||||
- optionally publish to **local Gitea only**
|
||||
|
||||
This is intentionally different from a final enterprise rollout.
|
||||
|
||||
The objective is to prove the local VM pattern first.
|
||||
|
||||
---
|
||||
|
||||
## 11. Risks And Guardrails
|
||||
|
||||
### Risks
|
||||
|
||||
- version drift between local source and published package versions
|
||||
- peer dependency mismatches hidden by current tarball workflow
|
||||
- Docker auth/config differences from shell installs
|
||||
- accidental publishing to the wrong registry
|
||||
- switching too many repos before the pilot is stable
|
||||
|
||||
### Guardrails
|
||||
|
||||
- use local-only credentials for this phase
|
||||
- keep `@bytelyst` scoped registry config explicit
|
||||
- do not remove tarball fallback globally until the pilot is green
|
||||
- migrate one consumer repo at a time
|
||||
- keep rollback steps documented per repo
|
||||
|
||||
---
|
||||
|
||||
## 12. Rollback Plan
|
||||
|
||||
If the registry-based flow fails during pilot migration:
|
||||
|
||||
1. revert the tested consumer back to its current working dependency mode
|
||||
2. restore its `docker-prep.sh` path if needed
|
||||
3. keep the Gitea registry work isolated to the local rehearsal branch/state
|
||||
4. fix the issue locally before retrying
|
||||
|
||||
No repo should lose its known-good build path until the registry model is proven.
|
||||
|
||||
---
|
||||
|
||||
## 13. Azure Single-VM Follow-Through
|
||||
|
||||
After the local rehearsal is green, Azure should follow the same validated recipe:
|
||||
|
||||
1. provision one VM
|
||||
2. install Docker and Gitea
|
||||
3. enable Gitea Actions runner
|
||||
4. enable Gitea packages
|
||||
5. clone the same repos onto the VM
|
||||
6. apply the same local registry/token/package flow
|
||||
7. re-run the pilot repo first
|
||||
8. then expand sequentially across the ecosystem
|
||||
|
||||
Azure should be a replication step, not a redesign step.
|
||||
|
||||
---
|
||||
|
||||
## 14. Definition Of Done
|
||||
|
||||
This migration plan is locally validated only when all are true:
|
||||
|
||||
- [x] local Gitea package-registry auth verified
|
||||
- [x] local package publish path verified for a pilot package set
|
||||
- [x] local consumer install path verified
|
||||
- [ ] local Docker build path verified without `docker-prep.sh`
|
||||
- [ ] local Gitea Actions path verified
|
||||
- [ ] pilot repo migrated successfully end-to-end including Docker
|
||||
- [ ] rollback path documented and tested conceptually
|
||||
- [ ] Azure single-VM reproduction steps documented from the validated local process
|
||||
|
||||
---
|
||||
|
||||
## 15. Immediate Next Actions
|
||||
|
||||
1. create or verify a local-only Gitea package token
|
||||
2. publish a minimal pilot `@bytelyst/*` package set to local Gitea only
|
||||
3. validate install from local Gitea in one pilot consumer
|
||||
4. validate Docker build from local Gitea in that pilot repo
|
||||
5. validate the same path via local Gitea Actions
|
||||
6. only then expand to broader ecosystem migration
|
||||
@ -27,17 +27,19 @@
|
||||
|
||||
### Shared Infrastructure (common-plat)
|
||||
|
||||
| Service | Port | Image | RAM Est. |
|
||||
| ------------------------ | ---------- | ---------------------------------------------------------------------- | -------- |
|
||||
| **platform-service** | 4003 | Fastify 5 + TS | ~200 MB |
|
||||
| **extraction-service** | 4005 | Fastify 5 + Python sidecar | ~350 MB |
|
||||
| **mcp-server** | 4007 | Fastify 5 + TS | ~150 MB |
|
||||
| **Cosmos DB Emulator** | 8081, 1234 | `mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview` | ~2 GB |
|
||||
| **Azurite** (blob) | 10000 | `mcr.microsoft.com/azure-storage/azurite` | ~100 MB |
|
||||
| **Mailpit** (SMTP) | 1025, 8025 | `axllent/mailpit` | ~50 MB |
|
||||
| **Traefik** (gateway) | 80, 8080 | `traefik:v3.3` | ~100 MB |
|
||||
| **Loki** (logs) | 3100 | `grafana/loki` | ~200 MB |
|
||||
| **Grafana** (dashboards) | 3000 | `grafana/grafana` | ~200 MB |
|
||||
| Service | Port | Image | RAM Est. |
|
||||
| ----------------------------------- | ---------- | ---------------------------------------------------------------------- | -------- |
|
||||
| **platform-service** | 4003 | Fastify 5 + TS | ~200 MB |
|
||||
| **extraction-service** | 4005 | Fastify 5 + Python sidecar | ~350 MB |
|
||||
| **mcp-server** | 4007 | Fastify 5 + TS | ~150 MB |
|
||||
| **Cosmos DB Emulator** | 8081, 1234 | `mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview` | ~2 GB |
|
||||
| **Azurite** (blob) | 10000 | `mcr.microsoft.com/azure-storage/azurite` | ~100 MB |
|
||||
| **Mailpit** (SMTP) | 1025, 8025 | `axllent/mailpit` | ~50 MB |
|
||||
| **Traefik** (gateway) | 80, 8080 | `traefik:v3.3` | ~100 MB |
|
||||
| **Loki** (logs) | 3100 | `grafana/loki` | ~200 MB |
|
||||
| **Grafana** (dashboards) | 3000 | `grafana/grafana` | ~200 MB |
|
||||
| **Gitea** (git + CI + pkg registry) | 3300, 2222 | `gitea/gitea` | ~150 MB |
|
||||
| **act_runner** (CI runner) | — | `gitea/act_runner` (or host-mode binary) | ~100 MB |
|
||||
|
||||
### Product Backends (Fastify 5 + TypeScript)
|
||||
|
||||
@ -98,8 +100,9 @@
|
||||
- 3 shared services × 250 MB = ~0.75 GB
|
||||
- 11 Next.js webs × 200 MB = ~2.2 GB
|
||||
- Infra (Traefik, Loki, Grafana, Azurite, Mailpit) = ~0.65 GB
|
||||
- Gitea + act_runner = ~0.25 GB
|
||||
- K3s overhead = ~0.5 GB
|
||||
- **Subtotal: ~7.4 GB** → headroom for spikes + build cache = **32 GB**
|
||||
- **Subtotal: ~7.65 GB** → headroom for spikes + build cache = **32 GB**
|
||||
|
||||
### Recommended (with Ollama, small models)
|
||||
|
||||
@ -241,6 +244,32 @@ services:
|
||||
volumes: [grafana-data:/var/lib/grafana]
|
||||
restart: unless-stopped
|
||||
|
||||
# ══════════════════════════════════════════════════════
|
||||
# GIT + CI + PACKAGE REGISTRY
|
||||
# ══════════════════════════════════════════════════════
|
||||
gitea:
|
||||
image: gitea/gitea:1.23
|
||||
ports: ['3300:3300', '2222:22']
|
||||
environment:
|
||||
GITEA__server__HTTP_PORT: 3300
|
||||
GITEA__server__ROOT_URL: http://gitea:3300
|
||||
GITEA__packages__ENABLED: 'true' # Built-in npm/container registry
|
||||
GITEA__service__DISABLE_REGISTRATION: 'true'
|
||||
volumes:
|
||||
- gitea-data:/data
|
||||
restart: unless-stopped
|
||||
|
||||
act-runner:
|
||||
image: gitea/act_runner:latest
|
||||
environment:
|
||||
GITEA_INSTANCE_URL: http://gitea:3300
|
||||
GITEA_RUNNER_REGISTRATION_TOKEN: '<generate-from-gitea-admin>'
|
||||
GITEA_RUNNER_LABELS: 'ubuntu-latest:host'
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock
|
||||
depends_on: [gitea]
|
||||
restart: unless-stopped
|
||||
|
||||
# ══════════════════════════════════════════════════════
|
||||
# SHARED SERVICES (common-plat — no file: deps, pnpm workspace handles it)
|
||||
# ══════════════════════════════════════════════════════
|
||||
@ -479,6 +508,7 @@ volumes:
|
||||
azurite-data:
|
||||
loki-data:
|
||||
grafana-data:
|
||||
gitea-data:
|
||||
localmemgpt-data:
|
||||
```
|
||||
|
||||
@ -663,8 +693,8 @@ kubectl scale deploy/platform-service --replicas=3 -n bytelyst-platform
|
||||
|
||||
```bash
|
||||
kubectl create secret generic bytelyst-secrets \
|
||||
--from-literal=JWT_SECRET=new-secret \
|
||||
--from-literal=COSMOS_KEY=new-key \
|
||||
--from-literal=JWT_SECRET='<replace-with-real-jwt-secret>' \
|
||||
--from-literal=COSMOS_KEY='<replace-with-real-cosmos-key>' \
|
||||
-n bytelyst-platform --dry-run=client -o yaml | kubectl apply -f -
|
||||
kubectl rollout restart deploy -n bytelyst-platform
|
||||
```
|
||||
@ -817,20 +847,21 @@ kubectl get pods -A
|
||||
|
||||
## 10. Dockerization Status (all complete)
|
||||
|
||||
| Repo | Backend Dockerfile | Web Dockerfile | `docker-prep.sh` | `output:'standalone'` | Package manager state | Lockfile state | Docker template type | Status |
|
||||
| --------------- | ------------------ | ------------------- | ---------------- | --------------------- | ---------------------------------------- | ---------------------------------- | ------------------------------- | ------------------------------------ |
|
||||
| **LysnrAI** | ✅ | ✅ user-dashboard | ✅ | ✅ (conditional) | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready |
|
||||
| **MindLyst** | ✅ | ✅ | ✅ | ✅ (conditional) | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready |
|
||||
| **ChronoMind** | ✅ | ✅ | ✅ | ✅ (conditional) | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready |
|
||||
| **JarvisJr** | ✅ | ✅ | ✅ | ✅ (conditional) | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready |
|
||||
| **PeakPulse** | ✅ | — (no web) | ✅ | — | No Node web surface in this repo | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready |
|
||||
| **FlowMonk** | ✅ | ✅ | ✅ | ✅ (conditional) | **Pilot candidate** for `pnpm` migration | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready |
|
||||
| **NomGap** | ✅ | ✅ | ✅ | ✅ | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Fixed (added `.tarballs/` COPY) |
|
||||
| **NoteLett** | ✅ | ✅ | ✅ | ✅ | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Fixed (explicit COPY, not `.`) |
|
||||
| **ActionTrail** | ✅ | ✅ | ✅ | ✅ | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready (uses `.tarballs/` pattern) |
|
||||
| **LocalMemGPT** | ✅ | ✅ | ✅ | ✅ | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready (repo-root build context) |
|
||||
| **admin-web** | — | ✅ (in common-plat) | N/A (`pnpm`) | ✅ (conditional) | `pnpm` workspace today | `pnpm-lock.yaml` via common-plat | `pnpm` workspace template | ✅ Ready |
|
||||
| **tracker-web** | — | ✅ (in common-plat) | N/A (`pnpm`) | ✅ (conditional) | `pnpm` workspace today | `pnpm-lock.yaml` via common-plat | `pnpm` workspace template | ✅ Ready |
|
||||
| Repo | Backend Dockerfile | Web Dockerfile | `docker-prep.sh` | `output:'standalone'` | Package manager state | Lockfile state | Docker template type | Status |
|
||||
| ------------------ | ------------------ | ------------------- | ---------------- | --------------------- | ---------------------------------------- | ---------------------------------- | ------------------------------- | ------------------------------------ |
|
||||
| **LysnrAI** | ✅ | ✅ user-dashboard | ✅ | ✅ (conditional) | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready |
|
||||
| **MindLyst** | ✅ | ✅ | ✅ | ✅ (conditional) | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready |
|
||||
| **ChronoMind** | ✅ | ✅ | ✅ | ✅ (conditional) | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready |
|
||||
| **JarvisJr** | ✅ | ✅ | ✅ | ✅ (conditional) | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready |
|
||||
| **PeakPulse** | ✅ | — (no web) | ✅ | — | No Node web surface in this repo | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready |
|
||||
| **FlowMonk** | ✅ | ✅ | ✅ | ✅ (conditional) | **Pilot candidate** for `pnpm` migration | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready |
|
||||
| **NomGap** | ✅ | ✅ | ✅ | ✅ | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Fixed (added `.tarballs/` COPY) |
|
||||
| **NoteLett** | ✅ | ✅ | ✅ | ✅ | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Fixed (explicit COPY, not `.`) |
|
||||
| **ActionTrail** | ✅ | ✅ | ✅ | ✅ | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready (uses `.tarballs/` pattern) |
|
||||
| **LocalMemGPT** | ✅ | ✅ | ✅ | ✅ | Transitioning toward `pnpm` target | Follow repo-local current lockfile | Repo-specific during transition | ✅ Ready (repo-root build context) |
|
||||
| **admin-web** | — | ✅ (in common-plat) | N/A (`pnpm`) | ✅ (conditional) | `pnpm` workspace today | `pnpm-lock.yaml` via common-plat | `pnpm` workspace template | ✅ Ready |
|
||||
| **tracker-web** | — | ✅ (in common-plat) | N/A (`pnpm`) | ✅ (conditional) | `pnpm` workspace today | `pnpm-lock.yaml` via common-plat | `pnpm` workspace template | ✅ Ready |
|
||||
| **user-dashboard** | — | ✅ (in common-plat) | N/A (`pnpm`) | ✅ (conditional) | `pnpm` workspace today | `pnpm-lock.yaml` via common-plat | `pnpm` workspace template | ✅ Ready |
|
||||
|
||||
**All 10 product repos now have Dockerfiles, `docker-prep.sh`, and `output:'standalone'`.** Created 2026-03-22.
|
||||
|
||||
@ -1058,6 +1089,181 @@ A repo is only considered migrated when all of the following are aligned and pas
|
||||
|
||||
---
|
||||
|
||||
## 11.2 Gitea — Self-Hosted Git + CI + Package Registry
|
||||
|
||||
### Why Gitea on the VM?
|
||||
|
||||
Gitea is a lightweight, self-hosted Git forge (~150 MB RAM) that provides three capabilities in one container:
|
||||
|
||||
1. **Git hosting** — mirror or primary for all 13+ repos
|
||||
2. **CI/CD** — GitHub Actions–compatible workflows via `act_runner`
|
||||
3. **npm package registry** — built-in, eliminates the `docker-prep.sh` tarball workflow
|
||||
|
||||
### Current pain: `docker-prep.sh` + `.tgz` tarballs
|
||||
|
||||
Every product repo has `file:../../learning_ai_common_plat/packages/*` dependencies. These don't resolve inside Docker build contexts, so each repo needs a `docker-prep.sh` that:
|
||||
|
||||
1. Builds all `@bytelyst/*` packages in common-plat
|
||||
2. Packs each into a `.tarballs/*.tgz` file
|
||||
3. Rewrites package.json `file:` refs → `file:.tarballs/bytelyst-*.tgz`
|
||||
4. Preserves the repo's active package-manager semantics during the rewrite
|
||||
|
||||
This is fragile, slow, and must be repeated for every image rebuild.
|
||||
|
||||
### Solution: Gitea npm Package Registry
|
||||
|
||||
Gitea has a built-in npm registry at `http://gitea:3300/api/packages/<org>/npm/`. Publishing `@bytelyst/*` packages there means **all product repos install from the registry like any normal npm dependency** — no tarballs, no rewriting, no `docker-prep.sh`.
|
||||
|
||||
> **Current local rehearsal status on this Mac:** host-side package publishing and host-side consumer installs are validated. Full Docker-side validation against the local Homebrew Gitea instance is still blocked by host/Docker registry URL reachability and tarball URL generation behavior. Treat the Docker portion of this section as the target architecture, not yet a fully cleared local result.
|
||||
|
||||
#### Setup (one-time)
|
||||
|
||||
```bash
|
||||
# 1. Create a Gitea API token (Settings → Applications → Generate Token)
|
||||
# Scopes: package:write, package:read
|
||||
|
||||
# 2. Configure npm/pnpm to use Gitea registry for @bytelyst scope
|
||||
# In ~/.npmrc or project .npmrc:
|
||||
@bytelyst:registry=http://localhost:3300/api/packages/bytelyst/npm/
|
||||
//localhost:3300/api/packages/bytelyst/npm/:_authToken=<gitea-token>
|
||||
|
||||
# For Docker builds, use the compose service name:
|
||||
@bytelyst:registry=http://gitea:3300/api/packages/bytelyst/npm/
|
||||
```
|
||||
|
||||
#### Publish workflow (CI or manual)
|
||||
|
||||
```bash
|
||||
# In learning_ai_common_plat — publish all packages after build
|
||||
cd /path/to/learning_ai_common_plat
|
||||
|
||||
# Build all packages
|
||||
pnpm -r --filter './packages/**' build
|
||||
|
||||
# Publish via the local helper that uses the proven pnpm-pack-based flow
|
||||
GITEA_NPM_TOKEN='<local-token>' bash ./scripts/publish-local-gitea-packages.sh
|
||||
```
|
||||
|
||||
This helper is important because raw `npm publish` is not sufficient for packages that contain internal `workspace:*` references. The validated local path uses `pnpm pack` first so workspace dependencies are normalized before the tarball is published to Gitea.
|
||||
|
||||
Or add a CI step in `learning_ai_common_plat/.gitea/workflows/ci.yml`:
|
||||
|
||||
```yaml
|
||||
publish-packages:
|
||||
name: Publish @bytelyst/* to Gitea npm registry
|
||||
runs-on: ubuntu-latest
|
||||
needs: [build-test-typecheck]
|
||||
steps:
|
||||
- name: Pull latest
|
||||
working-directory: /path/to/learning_ai_common_plat
|
||||
run: git pull --ff-only origin main || true
|
||||
|
||||
- name: Build packages
|
||||
working-directory: /path/to/learning_ai_common_plat
|
||||
run: pnpm -r --filter './packages/**' build
|
||||
|
||||
- name: Publish to Gitea registry
|
||||
working-directory: /path/to/learning_ai_common_plat
|
||||
env:
|
||||
NPM_TOKEN: ${{ secrets.GITEA_NPM_TOKEN }}
|
||||
run: |
|
||||
echo "@bytelyst:registry=http://gitea:3300/api/packages/bytelyst/npm/" > .npmrc
|
||||
echo "//gitea:3300/api/packages/bytelyst/npm/:_authToken=${NPM_TOKEN}" >> .npmrc
|
||||
for pkg in packages/*/; do
|
||||
(cd "$pkg" && npm publish --registry http://gitea:3300/api/packages/bytelyst/npm/ 2>/dev/null || true)
|
||||
done
|
||||
```
|
||||
|
||||
#### Product repo migration
|
||||
|
||||
For each product repo, replace `file:` refs with versioned (or `latest`) registry refs:
|
||||
|
||||
```jsonc
|
||||
// Before (package.json)
|
||||
{
|
||||
"@bytelyst/auth": "file:../../learning_ai_common_plat/packages/auth",
|
||||
"@bytelyst/config": "file:../../learning_ai_common_plat/packages/config",
|
||||
"@bytelyst/cosmos": "file:../../learning_ai_common_plat/packages/cosmos"
|
||||
}
|
||||
|
||||
// After
|
||||
{
|
||||
"@bytelyst/auth": "^0.1.0",
|
||||
"@bytelyst/config": "^0.1.0",
|
||||
"@bytelyst/cosmos": "^0.1.0"
|
||||
}
|
||||
```
|
||||
|
||||
With `.npmrc` pointing `@bytelyst` scope to Gitea, host-side `pnpm install` / `npm install` resolves from the local registry successfully in the current local rehearsal.
|
||||
|
||||
For Docker builds, the final single-VM target is still the same, but local validation on this Mac showed that a Homebrew-hosted Gitea instance may require extra reachability and `ROOT_URL` alignment work before BuildKit can consume package metadata and tarballs reliably.
|
||||
|
||||
#### Dockerfile simplification (after registry migration)
|
||||
|
||||
```dockerfile
|
||||
# NO MORE docker-prep.sh or .tarballs/ needed!
|
||||
FROM node:22-alpine AS builder
|
||||
WORKDIR /app
|
||||
|
||||
# .npmrc baked in (or passed as build arg)
|
||||
COPY .npmrc ./
|
||||
COPY package.json pnpm-lock.yaml ./
|
||||
RUN corepack enable && pnpm install --frozen-lockfile
|
||||
|
||||
COPY tsconfig.json ./
|
||||
COPY src/ ./src/
|
||||
RUN pnpm run build
|
||||
|
||||
FROM node:22-alpine
|
||||
WORKDIR /app
|
||||
ENV NODE_ENV=production
|
||||
COPY .npmrc ./
|
||||
COPY package.json pnpm-lock.yaml ./
|
||||
RUN corepack enable && pnpm install --frozen-lockfile --prod
|
||||
COPY --from=builder /app/dist ./dist
|
||||
EXPOSE ${PORT:-4010}
|
||||
CMD ["node", "dist/server.js"]
|
||||
```
|
||||
|
||||
#### Comparison: before vs after
|
||||
|
||||
| Aspect | Current (`docker-prep.sh`) | With Gitea npm registry |
|
||||
| ------------------- | ----------------------------------- | ---------------------------------------- |
|
||||
| **Pre-build step** | Run `docker-prep.sh` per repo | None (registry resolves deps) |
|
||||
| **Package format** | `.tgz` tarballs in `.tarballs/` | Standard npm registry install |
|
||||
| **`package.json`** | Rewritten with `file:.tarballs/...` | Normal `^0.1.0` semver refs |
|
||||
| **Docker build** | Must COPY `.tarballs/` into context | Standard `pnpm install` |
|
||||
| **Version pinning** | Whatever was packed at build time | Explicit semver in lockfile |
|
||||
| **Local dev** | `file:` symlinks (fast, direct) | Can keep `file:` for dev OR use registry |
|
||||
| **CI publish** | N/A | Auto-publish on common-plat push |
|
||||
| **Disk overhead** | ~50 MB tarballs per repo | Centralized in Gitea storage |
|
||||
|
||||
#### Migration path (incremental)
|
||||
|
||||
1. **Phase 0:** Gitea running locally with packages enabled
|
||||
2. **Phase 1:** Publish `@bytelyst/*` packages to local Gitea using the `pnpm pack`-based helper flow
|
||||
3. **Phase 2:** Migrate one pilot repo on the host: replace tarball/file refs with semver refs and validate install/typecheck/test
|
||||
4. **Phase 3:** Validate Docker builds against the local registry
|
||||
5. **Phase 4:** Only after local Docker validation is green, remove `docker-prep.sh` patterns repo-by-repo
|
||||
6. **Keep `file:` refs available** for local dev via overrides or link-based workflows if needed
|
||||
|
||||
### Gitea Container Registry (bonus)
|
||||
|
||||
Gitea also provides a built-in **Docker container registry** at `http://gitea:3300/v2/`. Instead of building images and importing them into K3s, you can:
|
||||
|
||||
```bash
|
||||
# Tag and push to Gitea container registry
|
||||
docker tag bytelyst/platform-service:latest gitea:3300/bytelyst/platform-service:latest
|
||||
docker push gitea:3300/bytelyst/platform-service:latest
|
||||
|
||||
# K3s / K8s pulls from Gitea registry
|
||||
# In deployment YAML: image: gitea:3300/bytelyst/platform-service:latest
|
||||
```
|
||||
|
||||
This gives you a **complete self-hosted DevOps stack**: Git → CI → npm registry → container registry → deploy.
|
||||
|
||||
---
|
||||
|
||||
## 12. Audit Findings (Review 2026-03-22)
|
||||
|
||||
Systematic code review of all claims in this document against the actual codebase.
|
||||
@ -1084,7 +1290,7 @@ Systematic code review of all claims in this document against the actual codebas
|
||||
1. Runs `pnpm build` in common-plat
|
||||
2. Packs each `@bytelyst/*` package into a `.tarballs/*.tgz`
|
||||
3. Rewrites package.json `file:` refs → `file:.tarballs/bytelyst-*.tgz`
|
||||
4. Preserves the product repo's active package-manager semantics during the rewrite
|
||||
4. Preserves the repo's active package-manager semantics during the rewrite
|
||||
|
||||
**All 10 repos now have `docker-prep.sh`** (created 2026-03-22). Previously only ActionTrail, LocalMemGPT, NoteLett, NomGap had them.
|
||||
|
||||
@ -1092,11 +1298,15 @@ Systematic code review of all claims in this document against the actual codebas
|
||||
|
||||
### F3. NomGap Backend Dockerfile Ignores `file:` Deps (BUG)
|
||||
|
||||
`@/learning_ai_fastgap/backend/Dockerfile` does `COPY package.json → npm ci` but doesn't copy `.tarballs/`. The `file:` refs will fail. Needs the `.tarballs/` COPY step added.
|
||||
`@/learning_ai_fastgap/backend/Dockerfile` previously did `COPY package.json → npm ci` without copying `.tarballs/`, which broke `file:` dependency resolution inside Docker.
|
||||
|
||||
**Status:** ✅ Fixed. The backend Dockerfile was updated to include the `.tarballs/` pattern.
|
||||
|
||||
### F4. NoteLett Backend Dockerfile Copies Everything (BUG)
|
||||
|
||||
`@/learning_ai_notes/backend/Dockerfile` does `COPY . .` in the build stage, which includes broken `node_modules` symlinks from `file:` deps. Should use explicit `COPY` of `src/`, `tsconfig.json`, and `.tarballs/` instead.
|
||||
`@/learning_ai_notes/backend/Dockerfile` previously used `COPY . .` in the build stage, which pulled in broken `node_modules` symlinks from `file:` dependencies.
|
||||
|
||||
**Status:** ✅ Fixed. The backend Dockerfile now uses explicit `COPY` steps instead of broad context copy.
|
||||
|
||||
### F5. Missing `output: 'standalone'` in next.config.ts (CRITICAL)
|
||||
|
||||
@ -1137,29 +1347,36 @@ Several web apps require `--webpack` flag for builds (Serwist PWA incompatible w
|
||||
|
||||
### F9. Missing `.env.ecosystem` Template
|
||||
|
||||
The compose references `.env.ecosystem` but the doc doesn't define its contents. Key vars needed:
|
||||
The compose references `.env.ecosystem`. A working template now exists at the workspace root. Current key vars:
|
||||
|
||||
```env
|
||||
# .env.ecosystem — shared env for all services
|
||||
COSMOS_ENDPOINT=https://cosmos-emulator:8081
|
||||
COSMOS_KEY=<emulator-key>
|
||||
COSMOS_ENDPOINT=http://cosmos-emulator:8081
|
||||
COSMOS_KEY=<cosmos-emulator-key-or-local-placeholder>
|
||||
COSMOS_DATABASE=bytelyst
|
||||
JWT_SECRET=dev-ecosystem-secret-change-me
|
||||
DB_PROVIDER=cosmos
|
||||
JWT_SECRET=dev-ecosystem-secret-do-not-use-in-production
|
||||
AZURE_BLOB_CONNECTION_STRING=DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=...;BlobEndpoint=http://azurite:10000/devstoreaccount1;
|
||||
PLATFORM_SERVICE_URL=http://platform-service:4003
|
||||
EXTRACTION_SERVICE_URL=http://extraction-service:4005
|
||||
DB_PROVIDER=memory
|
||||
NODE_ENV=production
|
||||
CORS_ORIGIN=*
|
||||
SMTP_HOST=mailpit
|
||||
SMTP_PORT=1025
|
||||
SMTP_FROM=noreply@bytelyst.local
|
||||
FIELD_ENCRYPT_KEY_PROVIDER=env
|
||||
FIELD_ENCRYPT_KEY=<64-char-hex-dev-key>
|
||||
FIELD_ENCRYPT_MEK_NAME=dev-ecosystem-mek
|
||||
NODE_ENV=production
|
||||
CORS_ORIGIN=*
|
||||
LOG_LEVEL=info
|
||||
```
|
||||
|
||||
**Status:** ✅ Implemented as `.env.ecosystem` in the workspace root.
|
||||
|
||||
### F10. `host.docker.internal` Only Works on Docker Desktop (Mac/Windows)
|
||||
|
||||
LocalMemGPT uses `OLLAMA_URL: 'http://host.docker.internal:11434'` — this works on Docker Desktop but **not on Linux VMs** (which is the likely deployment target).
|
||||
|
||||
**Fix on Linux:** Add `extra_hosts: ['host.docker.internal:host-gateway']` to the service, or use `network_mode: host`.
|
||||
**Status:** ✅ Addressed in `docker-compose.ecosystem.yml` via `extra_hosts: ['host.docker.internal:host-gateway']` on `localmemgpt-backend`.
|
||||
|
||||
### Summary of Required Work Before Compose Works
|
||||
|
||||
@ -1171,311 +1388,33 @@ LocalMemGPT uses `OLLAMA_URL: 'http://host.docker.internal:11434'` — this work
|
||||
| **P0** | Add `output: 'standalone'` to next.config.ts | 3 webs | ✅ Done (4 webs: ChronoMind, JarvisJr, FlowMonk, MindLyst) |
|
||||
| **P1** | Fix NomGap backend Dockerfile (add `.tarballs/` COPY) | 1 file | ✅ Done |
|
||||
| **P1** | Fix NoteLett backend Dockerfile (explicit COPY, not `.`) | 1 file | ✅ Done |
|
||||
| **P1** | Create `.env.ecosystem` template | 1 file | Pending |
|
||||
| **P1** | Create `.env.ecosystem` template | 1 file | ✅ Done |
|
||||
| **P2** | Standardize Node.js version to 22-alpine | 4 Dockerfiles | ✅ Done (all new Dockerfiles use 22-alpine) |
|
||||
| **P2** | Add `extra_hosts` for Linux VM Ollama access | 1 service | Pending |
|
||||
| **P2** | Add `extra_hosts` for Linux VM Ollama access | 1 service | ✅ Done (`localmemgpt-backend`) |
|
||||
|
||||
---
|
||||
|
||||
## 13. K8s & Docker Best Practices (from Production Comparisons)
|
||||
## 13. Kubernetes Roadmap Reference
|
||||
|
||||
> Derived from comparing three production K8s deployments: a Go-based Call Controller (Paladin), a Python/FastAPI streaming agent platform (NetBond), and a Python/FastAPI voice agent (Welcome Agent). These patterns should be adopted when ByteLyst moves from Docker Compose → K3s → managed K8s.
|
||||
Kubernetes planning has been split into a standalone roadmap:
|
||||
|
||||
### 13.1 Deployment — Zero-Downtime Rolling Updates
|
||||
- `docs/devops/KUBERNETES_ROADMAP.md`
|
||||
|
||||
**Do this (NetBond pattern):**
|
||||
Use that document for:
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
strategy:
|
||||
type: RollingUpdate
|
||||
rollingUpdate:
|
||||
maxUnavailable: 0 # Never kill a pod before its replacement is ready
|
||||
maxSurge: 1 # Only 1 extra pod during rollout
|
||||
template:
|
||||
spec:
|
||||
terminationGracePeriodSeconds: 45 # Match your app's drain timeout
|
||||
containers:
|
||||
- lifecycle:
|
||||
preStop:
|
||||
exec:
|
||||
command: ['sleep', '5'] # Let load balancer deregister before SIGTERM
|
||||
```
|
||||
- Docker Compose → Docker Desktop K8s / K3s transition planning
|
||||
- K8s best practices and production comparison takeaways
|
||||
- Helm values layering and namespace strategy
|
||||
- secrets progression
|
||||
- CI/CD guidance for chart/image promotion
|
||||
- local K8s deployment workflow shape
|
||||
|
||||
**Don't do this (Paladin anti-pattern):**
|
||||
`SINGLE_VM_DEPLOYMENT.md` remains the source of truth for:
|
||||
|
||||
```yaml
|
||||
maxUnavailable: 50% # Half your pods die instantly — users get errors
|
||||
maxSurge: 50% # Wastes resources by doubling pod count
|
||||
```
|
||||
|
||||
**ByteLyst action:** Every deployment template should use `maxUnavailable: 0` + preStop sleep + explicit `terminationGracePeriodSeconds` matching the Fastify graceful shutdown timeout.
|
||||
|
||||
### 13.2 Pod Security Context
|
||||
|
||||
**Always set (NetBond pattern):**
|
||||
|
||||
```yaml
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
```
|
||||
|
||||
If the app needs writable paths (e.g., `/tmp`, cache dirs), use `emptyDir` volumes:
|
||||
|
||||
```yaml
|
||||
volumes:
|
||||
- name: tmp
|
||||
emptyDir: {}
|
||||
- name: cache
|
||||
emptyDir: {}
|
||||
volumeMounts:
|
||||
- name: tmp
|
||||
mountPath: /tmp
|
||||
- name: cache
|
||||
mountPath: /home/node/.cache
|
||||
```
|
||||
|
||||
**ByteLyst action:** All Fastify backends are stateless — `readOnlyRootFilesystem: true` works. Next.js standalone servers may need `/tmp` writable.
|
||||
|
||||
### 13.3 Health Probes — Dedicated Endpoints
|
||||
|
||||
**Do this:**
|
||||
|
||||
```yaml
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health # Dedicated lightweight endpoint
|
||||
port: 4003
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5 # Fast fail — 5s max
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 4003
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
timeoutSeconds: 5
|
||||
```
|
||||
|
||||
**Don't do this (Welcome Agent anti-pattern):**
|
||||
|
||||
```yaml
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /openapi.json # Heavy endpoint, not a health check
|
||||
timeoutSeconds: 60 # Masks real failures for a full minute
|
||||
```
|
||||
|
||||
**ByteLyst action:** All backends already expose `GET /health` → `{ status: "ok" }`. Use it. Set timeout to 5s.
|
||||
|
||||
### 13.4 Ingress — WebSocket Support
|
||||
|
||||
If any service uses WebSocket or SSE (FlowMonk SSE, LocalMemGPT streaming, future real-time features):
|
||||
|
||||
```yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
annotations:
|
||||
nginx.ingress.kubernetes.io/proxy-read-timeout: '1800'
|
||||
nginx.ingress.kubernetes.io/proxy-send-timeout: '1800'
|
||||
nginx.ingress.kubernetes.io/proxy-buffering: 'off'
|
||||
nginx.ingress.kubernetes.io/proxy-http-version: '1.1'
|
||||
nginx.ingress.kubernetes.io/configuration-snippet: |
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
```
|
||||
|
||||
Missing WebSocket headers is a silent failure — connections drop after 60s with no error.
|
||||
|
||||
### 13.5 HPA — Use `autoscaling/v2`
|
||||
|
||||
**Do this:**
|
||||
|
||||
```yaml
|
||||
apiVersion: autoscaling/v2 # Current API, supports multiple metrics
|
||||
```
|
||||
|
||||
**Don't do this:**
|
||||
|
||||
```yaml
|
||||
apiVersion: autoscaling/v1 # Deprecated, CPU-only, will be removed
|
||||
```
|
||||
|
||||
### 13.6 Dockerfile Best Practices
|
||||
|
||||
| Practice | Do | Don't |
|
||||
| ------------------- | ---------------------------------------------------------- | ---------------------------------------------------------------------------------- |
|
||||
| **ENTRYPOINT form** | `ENTRYPOINT ["node", "dist/server.js"]` (exec form) | `ENTRYPOINT node dist/server.js` (shell form — PID 1 is `/bin/sh`, signals broken) |
|
||||
| **COPY scope** | `COPY package.json ./` then `COPY src/ ./src/` (selective) | `COPY . .` (copies node_modules, .git, tests, everything) |
|
||||
| **Layer count** | Combine related `RUN` steps | 3 separate `RUN pip install` / `RUN npm install` steps |
|
||||
| **Non-root** | `USER node` (Node.js images have a `node` user) | Running as root in production |
|
||||
| **Local variant** | Provide `local.Dockerfile` without corp proxy/JFrog deps | Single Dockerfile that only works behind corporate proxy |
|
||||
| **Build args** | `ARG NODE_ENV=production` for conditional behavior | Hardcoded env in Dockerfile |
|
||||
|
||||
### 13.7 Helm Values Layering
|
||||
|
||||
Use 3 layers for environment management:
|
||||
|
||||
```
|
||||
values.yaml # Base defaults (image, port, probes, resources)
|
||||
├── env/local.yaml # Local K3s overrides (lower resources, NodePort, no TLS)
|
||||
├── env/dev.yaml # Dev cluster overrides (replicas, hostnames, secrets)
|
||||
└── env/prod.yaml # Prod overrides (more replicas, real TLS, HPA limits)
|
||||
```
|
||||
|
||||
Deploy with layered `-f` flags:
|
||||
|
||||
```bash
|
||||
# Local
|
||||
helm upgrade --install myapp ./charts -f charts/values.yaml -f charts/env/local.yaml
|
||||
|
||||
# Dev
|
||||
helm upgrade --install myapp ./charts -f charts/values.yaml -f charts/env/dev.yaml
|
||||
|
||||
# Prod
|
||||
helm upgrade --install myapp ./charts -f charts/values.yaml -f charts/env/prod.yaml
|
||||
```
|
||||
|
||||
### 13.8 Namespace Strategy
|
||||
|
||||
Use Helm `_helpers.tpl` for namespace — never hardcode:
|
||||
|
||||
```yaml
|
||||
# ✅ Standard pattern — respects --namespace flag
|
||||
{{ include "myapp.namespace" . }}
|
||||
|
||||
# ❌ Anti-pattern — ignores helm --namespace, causes confusion
|
||||
{{ .Values.namespace }}
|
||||
```
|
||||
|
||||
### 13.9 Secrets Management Progression
|
||||
|
||||
| Phase | Strategy | Complexity |
|
||||
| ------------------------ | ----------------------------------------------------- | ---------- |
|
||||
| **Phase 1** (Compose) | `.env.ecosystem` file (gitignored) | Trivial |
|
||||
| **Phase 2** (K3s) | Native K8s `Secret` objects + `kubectl create secret` | Low |
|
||||
| **Phase 3** (Production) | Azure Key Vault via `SecretProviderClass` CSI driver | Medium |
|
||||
| **Phase 4** (Enterprise) | AKV + `AzureKeyVaultSecret` CRD with auto-sync | High |
|
||||
|
||||
ByteLyst already uses AKV in production (platform-service) — the CSI driver pattern is the natural next step.
|
||||
|
||||
### 13.10 CI/CD Best Practices (Lessons from Production Pipelines)
|
||||
|
||||
| Practice | Description |
|
||||
| ---------------------- | ------------------------------------------------------------------------------------------------------------ |
|
||||
| **Semantic release** | Auto-version from commit messages (`feat:` → minor, `fix:` → patch). ByteLyst already uses this convention. |
|
||||
| **Image promotion** | Build once → push to staging repo → promote to gold/prod repo (never rebuild for prod). |
|
||||
| **Branch pipelines** | Different CI stages per branch: feature (lint+test), develop (build+deploy-dev), main (promote+deploy-prod). |
|
||||
| **Security gates** | SAST + SCA scans on every build. Block merges on critical findings. |
|
||||
| **Quality gates** | Unit tests + coverage + SonarQube. Fail pipeline if coverage drops. |
|
||||
| **Auto-deploy to dev** | Pipeline trigger: when build completes → auto-deploy to dev. Manual gate for prod. |
|
||||
| **Chart versioning** | Publish Helm chart to OCI registry (ACR) with semantic version. Pull by version during deploy. |
|
||||
|
||||
### 13.11 Local K8s Development Script Template
|
||||
|
||||
A good local K8s deploy script should handle both Docker Desktop K8s (kind) and K3s:
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
# deploy-local-k8s.sh — Full local K8s deployment for ByteLyst ecosystem
|
||||
# Works with both Docker Desktop Kubernetes and K3s.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
NAMESPACE="bytelyst"
|
||||
ACTION="${1:-deploy}" # deploy | teardown
|
||||
|
||||
# Detect K8s runtime
|
||||
detect_runtime() {
|
||||
local ctx
|
||||
ctx=$(kubectl config current-context 2>/dev/null || echo "")
|
||||
if [[ "$ctx" == "docker-desktop" ]]; then
|
||||
echo "docker-desktop" # kind cluster inside Docker Desktop
|
||||
elif command -v k3s &>/dev/null; then
|
||||
echo "k3s"
|
||||
else
|
||||
echo "unknown"
|
||||
fi
|
||||
}
|
||||
|
||||
case "$ACTION" in
|
||||
deploy)
|
||||
RUNTIME=$(detect_runtime)
|
||||
echo "Detected K8s runtime: $RUNTIME"
|
||||
|
||||
# 1. Build all Docker images
|
||||
echo "Building images..."
|
||||
for svc in platform-service extraction-service mcp-server; do
|
||||
docker build -t bytelyst/$svc:local ./learning_ai_common_plat/services/$svc
|
||||
done
|
||||
|
||||
# 2. Load images into K8s runtime
|
||||
if [[ "$RUNTIME" == "docker-desktop" ]]; then
|
||||
echo "Docker Desktop: images are already available to K8s (shared daemon)."
|
||||
elif [[ "$RUNTIME" == "k3s" ]]; then
|
||||
echo "K3s: importing images into containerd..."
|
||||
for img in $(docker images --format '{{.Repository}}:{{.Tag}}' | grep bytelyst); do
|
||||
sudo k3s ctr images import <(docker save "$img")
|
||||
done
|
||||
else
|
||||
echo "WARNING: Unknown K8s runtime. You may need to load images manually."
|
||||
fi
|
||||
|
||||
# 3. Create namespace + secrets
|
||||
kubectl create namespace "$NAMESPACE" --dry-run=client -o yaml | kubectl apply -f -
|
||||
kubectl create secret generic bytelyst-secrets \
|
||||
--from-env-file=.env.ecosystem \
|
||||
-n "$NAMESPACE" --dry-run=client -o yaml | kubectl apply -f -
|
||||
|
||||
# 4. Deploy via Helm with local overlay
|
||||
helm upgrade --install bytelyst ./helm/bytelyst-ecosystem \
|
||||
-f helm/bytelyst-ecosystem/values.yaml \
|
||||
-f helm/bytelyst-ecosystem/env/local.yaml \
|
||||
-n "$NAMESPACE"
|
||||
|
||||
# 5. Wait + verify
|
||||
kubectl rollout status deploy -n "$NAMESPACE" --timeout=120s
|
||||
echo ""
|
||||
echo "All pods:"
|
||||
kubectl get pods -n "$NAMESPACE"
|
||||
echo ""
|
||||
if [[ "$RUNTIME" == "docker-desktop" ]]; then
|
||||
echo "View in Docker Desktop: Kubernetes tab → namespace: $NAMESPACE"
|
||||
fi
|
||||
echo "Port-forward: kubectl port-forward svc/platform-service 4003:4003 -n $NAMESPACE"
|
||||
;;
|
||||
|
||||
teardown)
|
||||
helm uninstall bytelyst -n "$NAMESPACE" 2>/dev/null || true
|
||||
kubectl delete namespace "$NAMESPACE" 2>/dev/null || true
|
||||
echo "Teardown complete."
|
||||
;;
|
||||
esac
|
||||
```
|
||||
|
||||
### 13.12 Quick Reference — What to Apply at Each Phase
|
||||
|
||||
| Best Practice | Phase 1 (Compose) | Phase 2 (K3s) | Phase 3 (Prod K8s) |
|
||||
| ---------------------------- | ------------------------ | ------------------ | ------------------ |
|
||||
| Zero-downtime rolling update | N/A | ✅ Apply | ✅ Apply |
|
||||
| Pod security context | N/A | ✅ Apply | ✅ Apply |
|
||||
| Health probes | N/A (use `healthcheck:`) | ✅ Apply | ✅ Apply |
|
||||
| WebSocket ingress headers | N/A | ✅ If using SSE/WS | ✅ Apply |
|
||||
| HPA v2 | N/A | Optional | ✅ Apply |
|
||||
| Exec-form ENTRYPOINT | ✅ Apply now | ✅ | ✅ |
|
||||
| Selective COPY | ✅ Apply now | ✅ | ✅ |
|
||||
| Non-root user | ✅ Apply now | ✅ | ✅ |
|
||||
| Values layering | N/A | ✅ Apply | ✅ Apply |
|
||||
| Secrets via AKV CSI | N/A | N/A | ✅ Apply |
|
||||
| Semantic release | ✅ Apply now | ✅ | ✅ |
|
||||
| Image promotion | N/A | N/A | ✅ Apply |
|
||||
| Local deploy script | N/A | ✅ Apply | ✅ Adapt |
|
||||
- single-VM deployment scope
|
||||
- Docker Compose ecosystem architecture
|
||||
- Dockerization/package-manager deployment guidance
|
||||
- current implementation status and audit findings
|
||||
|
||||
---
|
||||
|
||||
@ -1483,9 +1422,11 @@ esac
|
||||
|
||||
| Question | Answer |
|
||||
| ------------------------------ | -------------------------------------------------------------------------------------------------------------- |
|
||||
| **Can deploy on single VM?** | **Yes.** All ~25 services fit in 32 GB RAM. |
|
||||
| **Can deploy on single VM?** | **Yes.** All ~27 services fit in 32 GB RAM. |
|
||||
| **All Dockerized?** | **Yes.** All 10 product repos now have Dockerfiles + docker-prep.sh. |
|
||||
| **Package-manager direction?** | **`pnpm` is the long-term standard** for Node/TS repos, but migration is phased repo-by-repo, not big-bang. |
|
||||
| **Self-hosted CI?** | **Yes.** Gitea + act_runner (~250 MB combined). GitHub Actions–compatible workflows. |
|
||||
| **Package registry?** | **Gitea npm registry** replaces `docker-prep.sh` + `.tgz` tarballs. Also has Docker container registry. |
|
||||
| **K8s practice on single VM?** | **Docker Desktop K8s** (Mac/Windows) or **K3s** (Linux). Same manifests scale to AKS/EKS/GKE. |
|
||||
| **Recommended VM?** | 8 vCPU / 32 GB (min) or 16 vCPU / 64 GB (with Ollama). Hetzner ~$45/mo for dev. |
|
||||
| **Time to production K8s?** | Phase 1 (compose) → Phase 2 (Docker Desktop / K3s) → Phase 3 (multi-node) → Phase 4 (managed). Same manifests. |
|
||||
|
||||
77
scripts/publish-local-gitea-packages.sh
Normal file
77
scripts/publish-local-gitea-packages.sh
Normal file
@ -0,0 +1,77 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
PACKAGES_DIR="$REPO_ROOT/packages"
|
||||
TMP_DIR="${TMPDIR:-/tmp}/bytelyst-gitea-publish"
|
||||
REGISTRY_URL="${GITEA_NPM_REGISTRY_URL:-http://localhost:3300/api/packages/bytelyst/npm/}"
|
||||
TOKEN="${GITEA_NPM_TOKEN:-}"
|
||||
PACKAGE_FILTER="${1:-}"
|
||||
|
||||
if [ -z "$TOKEN" ]; then
|
||||
echo "❌ GITEA_NPM_TOKEN is required"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
rm -rf "$TMP_DIR"
|
||||
mkdir -p "$TMP_DIR"
|
||||
|
||||
publish_package() {
|
||||
local pkg_dir="$1"
|
||||
local package_name
|
||||
local package_version
|
||||
package_name="$(node -e "const fs=require('fs'); const pkg=JSON.parse(fs.readFileSync(process.argv[1], 'utf8')); process.stdout.write(pkg.name);" "$pkg_dir/package.json")"
|
||||
package_version="$(node -e "const fs=require('fs'); const pkg=JSON.parse(fs.readFileSync(process.argv[1], 'utf8')); process.stdout.write(pkg.version);" "$pkg_dir/package.json")"
|
||||
local safe_name
|
||||
safe_name="${package_name//@/}"
|
||||
safe_name="${safe_name//\//-}"
|
||||
local work_dir="$TMP_DIR/$safe_name-$package_version"
|
||||
local packed_tgz
|
||||
local final_tgz
|
||||
|
||||
rm -rf "$work_dir"
|
||||
mkdir -p "$work_dir"
|
||||
|
||||
echo "📦 Packing $package_name@$package_version"
|
||||
(
|
||||
cd "$pkg_dir"
|
||||
pnpm pack --pack-destination "$work_dir" >/dev/null
|
||||
)
|
||||
|
||||
packed_tgz="$(find "$work_dir" -maxdepth 1 -name '*.tgz' | head -1)"
|
||||
if [ -z "$packed_tgz" ]; then
|
||||
echo "❌ Failed to pack $package_name@$package_version"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mkdir -p "$work_dir/unpacked"
|
||||
tar -xzf "$packed_tgz" -C "$work_dir/unpacked"
|
||||
(
|
||||
cd "$work_dir/unpacked/package"
|
||||
npm pack --pack-destination "$work_dir" >/dev/null
|
||||
)
|
||||
|
||||
final_tgz="$(find "$work_dir" -maxdepth 1 -name '*.tgz' | sort | tail -1)"
|
||||
if [ -z "$final_tgz" ]; then
|
||||
echo "❌ Failed to repack $package_name@$package_version"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "🚀 Publishing $package_name@$package_version to $REGISTRY_URL"
|
||||
if ! npm publish "$final_tgz" \
|
||||
--registry "$REGISTRY_URL" \
|
||||
--//localhost:3300/api/packages/bytelyst/npm/:_authToken="$TOKEN"; then
|
||||
echo "⚠️ Publish failed for $package_name@$package_version (possibly already published)"
|
||||
fi
|
||||
}
|
||||
|
||||
while IFS= read -r -d '' pkg_json; do
|
||||
pkg_dir="$(dirname "$pkg_json")"
|
||||
pkg_name="$(node -e "const fs=require('fs'); const pkg=JSON.parse(fs.readFileSync(process.argv[1], 'utf8')); process.stdout.write(pkg.name);" "$pkg_json")"
|
||||
if [ -n "$PACKAGE_FILTER" ] && [ "$pkg_name" != "$PACKAGE_FILTER" ]; then
|
||||
continue
|
||||
fi
|
||||
publish_package "$pkg_dir"
|
||||
done < <(find "$PACKAGES_DIR" -mindepth 2 -maxdepth 2 -name package.json -print0 | sort -z)
|
||||
|
||||
echo "✅ Local Gitea package publish complete"
|
||||
Loading…
Reference in New Issue
Block a user