bytelyst-devops-tools/docs/deployment-optimization-roadmap.md

Deployment Optimization Roadmap

Status: v2 (PROPOSED — analysis complete, no changes applied yet) · Owner: Platform DevOps · Created: 2026-05-31 · Revised: 2026-05-31

v2 review pass: corrected findings after auditing all deploy scripts + compose files (not just deploy-invttrdg.sh). Key fixes: memory limits already exist (D6 rewritten), down/--force-recreate is invttrdg-only not universal (D2 re-scoped), BuildKit gap spans all docker compose build paths (D4 broadened), and added D8 — deploy-script drift across the per-product scripts.

Optimize the deployment orchestration layer for the single-Azure-VM MVP: reduce deploy wall-clock time, eliminate deploy-time downtime, and stop the production VM from running out of RAM/disk during builds.

Scope boundary — read this first. This roadmap is about how we ship and run images on the VM (the deploy-*.sh scripts, docker compose lifecycle, registry strategy, VM resource limits). The complementary image-build concerns (pnpm install speed, BuildKit cache mounts, .npmrc.docker, Gitea registry correctness, "build green / app broken" silent failures) live in docker-build-optimization-roadmap.md and are out of scope here — this doc references that work, never duplicates it.

---

0. Current state (audited 2026-05-31)

The production deployment model is Docker Compose on a single Azure VM, fronted by Caddy on 80/443 (see DEPLOYMENT_GUIDE.md and learning_ai_common_plat/docs/devops/single_azure_vm/docker/DEPLOYMENT_STATUS_2026-03-29.md). Four production repos deploy here: learning_ai_invt_trdg, learning_ai_common_plat, learning_ai_clock, learning_ai_notes.

Deploy surfaces audited (2026-05-31): deploy-invttrdg.sh, deploy-clock.sh, deploy-notes.sh, deploy-all.sh, dashboard/deploy.sh, plus the production docker-compose.yml of clock and the docker-compose.ecosystem.yml of common_plat.

What is already good (do not redo): the per-repo Dockerfiles are multi-stage, use node:22-alpine, mount a BuildKit pnpm-store cache, inject the Gitea token via a BuildKit secret, and emit Next.js standalone output. The image-build layer is in good shape — credit to the build-optimization roadmap.

The bottlenecks are in the deployment flow, not the Dockerfiles.

#	Finding	Location	Symptom it causes	Severity
D1	Images are built on the production VM. docker build runs on the box while ~31 containers (Cosmos emulator + Azurite are RAM-heavy) are live. The VM has less RAM than the deploy scripts assumed (per the deployment-status doc), so builds thrash/swap.	deploy-invttrdg.sh build step; deploy-all.sh docker compose build	Slow builds and memory pressure and risk of OOM-killing live services	High (keystone)
D2	deploy-invttrdg.sh does a blanket docker compose down → up -d --force-recreate — stops all its services and restarts them cold, even for a one-line change. (Note: deploy-clock.sh/deploy-notes.sh do not — they already run docker compose build + docker compose up -d, i.e. differential recreate. So this is a per-script inconsistency, not a universal pattern — see D8.)	deploy-invttrdg.sh (down then --force-recreate)	Deploy-time downtime + cold caches on invttrdg releases	Medium
D3	deploy-all.sh rebuilds every service in every repo, sequentially. No change-detection, no parallelism; loops repos one-by-one and runs docker compose build (all services).	deploy-all.sh deploy loop	Multi-repo "deploy all" takes minutes even when one file changed	High
D4	The docker compose build paths don't pin BuildKit. deploy-clock.sh, deploy-notes.sh, and deploy-all.sh all call plain docker compose build with no DOCKER_BUILDKIT=1 / COMPOSE_DOCKER_CLI_BUILD=1. Modern Compose v2 defaults to BuildKit (and the compose files use build-time secrets:, which require BuildKit — so on a stale/old toolchain the build hard-fails rather than silently slows). Pinning it explicitly removes the version-dependent ambiguity and guarantees the Dockerfile --mount=type=cache pnpm-store wins.	deploy-clock.sh, deploy-notes.sh, deploy-all.sh build steps	Toolchain-dependent build behavior; risk of losing warm-cache wins	Low–Medium
D5	Whole node_modules (dev deps included) copied into the runtime stage. Backend runtime COPY --from=builder .../node_modules is the full install (verified in learning_ai_clock/backend/Dockerfile).	*/backend/Dockerfile runtime stage	Larger images → more disk, slower pull/start	Medium
D6	Memory limits exist but are likely untuned vs the real VM, and log/image-disk guardrails are absent. Limits are set via deploy.resources.limits.memory (cosmos-emulator 1g, azurite 256m, most services 128m–512m). The gaps: (a) the sum of limits hasn't been reconciled against the VM's actual (lower-than-assumed) RAM; (b) only limits, no reservations; (c) no scheduled docker image prune and no container-log rotation, so disk creeps unbounded.	docker-compose.ecosystem.yml, product docker-compose.yml, VM cron	Disk creeps to full; limits may over- or under-commit RAM	Medium
D7	Rollback requires a rebuild. DEPLOYMENT_GUIDE.md rollback path is git revert + docker compose up -d --build — i.e. rebuild on the VM to roll back. Production images are tagged :latest (e.g. invttrdg-backend:latest) or auto-named by Compose, so there is no immutable per-release artifact to re-point to.	DEPLOYMENT_GUIDE.md rollback section; image tags	Slow, risky rollbacks; no clean "known-good" artifact	Medium
D8	Deploy-script drift — no single source of truth. Three near-duplicate per-product scripts (deploy-invttrdg.sh, deploy-clock.sh, deploy-notes.sh) plus deploy-all.sh and dashboard/deploy.sh have diverged: invttrdg builds via an explicit docker build loop then down + --force-recreate; clock/notes use docker compose build + up -d; common_plat has no dedicated script (goes through deploy-all.sh). Any fix in this roadmap must be applied N times and risks further drift.	all deploy-*.sh + dashboard/deploy.sh	Inconsistent behavior; fixes don't propagate; maintenance burden	Medium

Implications

• D1 is the keystone. Moving builds off the VM removes the build/runtime resource contention that drives slow builds, memory pressure, and (with SHA-tagged images) enables fast rollback. Most other items compound on top of it.
• D8 is a force-multiplier risk. Because the per-product scripts have already drifted, every other fix (D2, D4, change-detection, rollback) must either be applied 3+ times or — better — the scripts should first be unified behind one parameterized deploy library. Prefer consolidating early so later phases land once.
• D2 removes invttrdg's deploy-time downtime and is low-risk; clock/notes already do the right thing here, which is exactly why unifying (D8) matters.
• D3/D4 are the "deploy all is slow" story; D5/D6 are the disk/memory story. Note D6's headline gap is disk hygiene + RAM right-sizing, not "limits missing" — limits already exist.

---

1. Goals & non-goals

Goals

• Cut warm deploy wall-clock time to seconds for a single changed service.
• Zero (or near-zero) downtime for routine deploys.
• Keep the production VM's RAM/disk predictable and bounded during deploys.
• Make rollback an artifact re-point, not a rebuild.

Non-goals

• ❌ Adopting Kubernetes / Swarm / Nomad. Compose-on-one-VM is the correct model at MVP; revisit orchestration only when we outgrow a single host.
• ❌ Re-doing image-build internals (pnpm, BuildKit cache, Gitea path) — owned by docker-build-optimization-roadmap.md.
• ❌ Full blue/green infra. Tier 1 already removes most downtime; multi-replica comes only when traffic justifies it.

Measurement targets

Metric	Baseline (observed/estimated)	Target
Warm deploy, 1 service changed	~2–3 min (build-on-VM)	< 30 s (pull + recreate one service)
Deploy-time downtime per service	full stop/start cycle	~0 (recreate-in-place, old stays up until new is ready)
Peak VM RAM during deploy	build spike on top of live stack	no build spike (build is off-VM)
Rollback to previous release	rebuild on VM (minutes)	< 30 s (re-point SHA tag + up -d)

Fill in actuals during Phase 3.

---

2. Phased roadmap (why / what / how)

Phases are ordered by leverage. Phase 0 (unify the scripts) is a prerequisite enabler; Phase 1 is the keystone — every later phase should land in the unified script once, not be copy-pasted across the per-product scripts.

Phase 0 — Unify the deploy scripts (prerequisite)

Why. D8: deploy-invttrdg.sh, deploy-clock.sh, and deploy-notes.sh are near-duplicate copies that have already drifted (invttrdg uses an explicit docker build loop + down + --force-recreate; clock/notes use docker compose build + up -d). Every fix below would otherwise have to be written 3+ times. Consolidating first means Phases 1–5 are implemented once.

What. A single parameterized deploy entrypoint (one script or a sourced library) that takes the repo/product as input and encodes the build + lifecycle

• health-check steps once. The per-product scripts become thin wrappers (or are replaced by deploy.sh <product>).

How (checklist).

• 0.1 Inventory the divergence across deploy-invttrdg.sh, deploy-clock.sh, deploy-notes.sh, deploy-all.sh, dashboard/deploy.sh (build method, lifecycle command, health endpoints, package-publication check).
• 0.2 Extract the shared steps (dirty-check, fetch/rebase, smoke test, build, deploy, health check) into one library; express per-product differences (ports, endpoints, image names) as config/data.
• 0.3 Replace the per-product scripts with thin wrappers calling the library; keep the old filenames as shims so existing muscle memory + the DevOps dashboard's deploy buttons keep working.
• 0.4 Point deploy-all.sh and dashboard/deploy.sh at the same library.
• 0.5 Add a drift guard (lint/CI) so the scripts can't silently diverge again — mirror the check-*-drift.sh pattern already used for .npmrc and docker-prep.sh in learning_ai_common_plat.

Done when: one code path drives all production deploys; per-product files are config or thin shims; a drift check guards against regression.

If Phase 0 is deferred, treat every checklist item in Phases 1–5 as "apply to invttrdg and clock and notes" — the drift tax is real.

---

Phase 1 — Build off the VM, ship images, VM pulls ⟵ keystone

Why. D1 is the single biggest cause of slow builds, memory pressure, and risky rollback. The production VM should run containers, not compile them. Removing build work from the box frees its scarce RAM/CPU and turns deploys into a fast pull + recreate. Tagging images by commit SHA gives an immutable, re-pointable artifact (fixes D7).

What.

• Build images in CI (GitHub Actions / Gitea Actions) or on a dedicated builder.
• Push to the Gitea container/image registry, tagged :<commit-sha> and :latest.
• The VM deploy step becomes docker compose pull && docker compose up -d — no docker build on the box.

How (checklist).

• 1.1 Stand up / confirm an image registry (reuse Gitea on the VM, or a hosted registry). Decide auth: reuse the existing GITEA_NPM_TOKEN pattern from deploy-invttrdg.sh for docker login.
• 1.2 Add a CI build job per production repo: build backend + web images with BuildKit, tag :<git-sha> + :latest, push to the registry. Reuse the existing BYTELYST_COMMIT_* build args already collected in deploy-invttrdg.sh.
• 1.3 Parameterize each docker-compose.yml service to use image: <registry>/<svc>:${IMAGE_TAG:-latest} instead of a local build: context for production. (Keep build: for local dev via an override file.)
• 1.4 Rewrite the VM-side deploy path to docker compose pull then docker compose up -d (no build). Pass IMAGE_TAG=<sha> to deploy a specific release.
• 1.5 Keep a thin "emergency build-on-VM" fallback flag for when the registry/CI is unavailable, but make pull-based the default.
• 1.6 Verify: a clean deploy uses zero tsc/next build CPU on the VM.

Done when: deploys to the VM perform no compilation; images are addressable by commit SHA in the registry.

---

Phase 2 — Stop-the-world → recreate-in-place

Why. D2: deploy-invttrdg.sh does docker compose down + --force-recreate, taking everything down on every deploy. Plain up -d already recreates only the containers whose image/config changed, leaving the rest running — which is exactly what deploy-clock.sh/deploy-notes.sh already do. This phase is mostly about bringing invttrdg in line with clock/notes (and, post-Phase 0, deleting the divergence entirely).

What. Remove the blanket down; rely on Compose's differential recreate. Target individual services where possible.

How (checklist).

• 2.1 Remove docker compose down from deploy-invttrdg.sh (the only script that has it). Replace up -d --force-recreate (all services) with up -d (differential) — --force-recreate only when config didn't change but image did and you're not using SHA tags (after Phase 1, the SHA tag change makes Compose recreate automatically). Adopt the clock/notes pattern.
• 2.2 Support per-service deploys: docker compose up -d --no-deps <svc> so deploying the backend doesn't bounce unrelated services.
• 2.3 Confirm every service has a correct healthcheck so up -d waits for healthy before considering the deploy done. Clock already handles the IPv6/localhost pitfall (F12 in the build roadmap) by forcing 127.0.0.1 in its healthcheck — verify the other production repos do the same.
• 2.4 (Later / optional) For true zero-downtime on a hot service, run two replicas behind Caddy and recreate one at a time. Defer until traffic needs it.

Done when: a routine single-service deploy does not interrupt the other running services.

---

Phase 3 — Deploy only what changed; guarantee the fast path

Why. D3/D4: deploy-all.sh rebuilds everything sequentially, and none of the docker compose build paths pin BuildKit. After Phase 1 most of this moves to CI, but the VM-side and any remaining build paths should still skip untouched services and use a deterministic, warm-cache builder.

What. Change-detection on what to deploy + an explicit BuildKit guarantee for any path that still builds.

How (checklist).

• 3.1 In deploy-all.sh (or the Phase-0 unified library), compute changed services via git diff --name-only <last-deployed-sha>..HEAD and skip services with no changes. Record the last-deployed SHA per repo (e.g. a small state file or the registry tag).
• 3.2 Pin DOCKER_BUILDKIT=1 and COMPOSE_DOCKER_CLI_BUILD=1 (or switch to docker buildx bake) in every path that still builds — deploy-clock.sh, deploy-notes.sh, deploy-all.sh (D4) — so behavior is not toolchain-version-dependent and the Dockerfile --mount=type=cache pnpm-store wins are guaranteed.
• 3.3 Where multiple independent images must build, build them in parallel (buildx bake, or docker compose build --parallel) instead of the current sequential loop.
• 3.4 Capture before/after timings into the Measurement targets table above.

Done when: "deploy all" only touches changed services and always uses the warm-cache build path.

---

Phase 4 — Image size & VM resource guardrails

Why. D5/D6: bloated images and unbounded disk are the slow-burn causes of "the VM filled up / a build OOM'd the box." Memory limits already exist via deploy.resources.limits.memory — the remaining work is right-sizing them against the real (lower-than-assumed) VM RAM, adding reservations, and adding disk hygiene.

What. Prune runtime deps, reconcile + extend memory caps, rotate logs, prune images.

How (checklist).

• 4.1 In each backend runtime stage, install production-only deps (pnpm install --prod / pnpm deploy --prod) instead of copying the full builder node_modules (D5). Verify the app still starts.
• 4.2 Reconcile the existing deploy.resources.limits.memory values (cosmos-emulator 1g, azurite 256m, services 128m–512m) against the VM's actual RAM — confirm the sum fits with headroom. Add reservations (not just limits) so the scheduler protects critical services, and add cpus where a service is CPU-bursty.
• 4.3 Add Docker daemon log rotation (json-file with max-size + max-file, or ship logs to Loki only) so container logs can't fill disk.
• 4.4 Add a scheduled docker image prune -f (and builder prune) on the VM to reclaim dangling layers left by rebuilds.
• 4.5 Add a small swap file on the VM as an OOM safety net for any residual on-box work; alert when disk > 80%.

Done when: runtime images are prod-only, memory limits are reconciled with VM RAM (+ reservations), and disk usage is bounded by rotation + prune.

---

Phase 5 — Fast, artifact-based rollback

Why. D7: rollback today means rebuild-on-VM. With SHA-tagged images from Phase 1, rollback becomes re-pointing to a known-good tag.

What. A rollback command that redeploys a previous image tag.

How (checklist).

• 5.1 Keep the last N image SHAs in the registry (don't prune the most recent good tags).
• 5.2 Add IMAGE_TAG=<previous-sha> docker compose up -d rollback path (and a deploy-*.sh --rollback [sha] wrapper).
• 5.3 Update DEPLOYMENT_GUIDE.md rollback section to use tag re-point instead of git revert + rebuild.
• 5.4 Document how to find the currently-deployed SHA (the /api/devops/version endpoint already exposed and checked by deploy-invttrdg.sh).

Done when: rolling back is a sub-30s tag re-point with no rebuild.

---

3. Quick-reference summary

Phase	Theme	Fixes	Primary symptom addressed
0	Unify deploy scripts	D8	Stops fixes from being copy-pasted/drifting
1	Build off VM, pull images	D1, D7	Slow builds + memory pressure + rollback
2	Recreate-in-place (align invttrdg)	D2	Downtime
3	Deploy only changed + BuildKit guarantee	D3, D4	Slow "deploy all"
4	Image slimming + RAM right-sizing + disk hygiene	D5, D6	Disk/memory
5	Artifact rollback	D7	Rollback speed/safety

Suggested order: Phase 0 (unify, so later fixes land once) → Phase 1 → 2 (≈80% of the benefit across all three symptoms), then 3 → 4 → 5. If Phase 0 is skipped, apply Phases 2–4 to invttrdg, clock, and notes individually.

4. Explicitly out of scope

• Image-build internals (pnpm/BuildKit/Gitea/.npmrc.docker/silent-break correctness) — see docker-build-optimization-roadmap.md.
• Migration to Kubernetes/Swarm or a managed cloud runtime.
• Multi-platform image builds.

5. Related docs

• ../DEPLOYMENT_GUIDE.md — current production deploy procedure
• docker-build-optimization-roadmap.md — image-build layer
• VM_OBSERVABILITY_ROADMAP.md — metrics/monitoring for the VM
• vm-security-blind-spots-roadmap.md — VM hardening
• ../../learning_ai_common_plat/docs/devops/single_azure_vm/docker/DEPLOYMENT_STATUS_2026-03-29.md — VM deployment status snapshot