# Azure VM Deployment Status — 2026-03-29

> Status snapshot for the single-Azure-VM Docker deployment described in this folder.
> This document records what was completed on the VM, what was manually fixed during validation, what is currently healthy, and what still remains.

---

## Executive Summary

The Azure VM deployment is **partially successful**.

### 2026-03-31 follow-up

The VM moved forward after the original snapshot:

- `prometheus`, `node-exporter`, `cadvisor`, and `valkey` are now running
- `admin-web` and `tracker-web` are healthy again
- `extraction-service` product rate limiting now uses Valkey-backed shared counters
- the admin ops page now includes VM inventory and Valkey inspection panels
- the admin ops page now includes allowlisted container restart actions and guarded Valkey delete actions
- live Mission Control and inventory now cover the VM-hosted product web apps and other browser-facing VM surfaces
- `llmlab-dashboard` is treated as internal VM tooling, while `localmemgpt-web` is intended to be Vercel-hosted rather than part of the VM web surface
- live verification confirmed a request for `productId=valkey-check-2` created the key `extraction:product-rate-limit:29582406:valkey-check-2` in Valkey with value `1`
- GoDaddy DNS now points `api`, `gitea`, `admin`, `tracker`, `llmlab`, and `ollama` at `187.124.159.82`
- Caddy is the active VM ingress on `80/443` with valid Let's Encrypt certificates for all six hostnames
- `ollama.bytelyst.com/api/version` returns `200` and serves Ollama `0.18.0` over HTTPS

### What is working

- all platform and product **backend services** are running and healthy
- core infrastructure needed by the backend stack is up:
  - Cosmos emulator
  - Azurite
  - Mailpit
  - Loki
  - Grafana
  - Gitea
  - Traefik
- the generated host-side health script confirms all backend APIs are reachable
- the package registry path used by Docker builds was repaired, and the backend fleet was recovered after build failures

### What is not fully complete

- `setup.sh --status` still reports **Phase 7 pending**
- `admin-web` and `tracker-web` remain unresolved
- some product web apps were brought up but remain out of scope for a backend-only completion target
- some prompt-level validation checks still fail because they depend on web/UI or local-LLM surfaces rather than backend health

### Current bottom line

If the definition of success is:

- "all backends are running and healthy"

then the deployment is in a usable state.

If the definition of success is:

- "all 31 services are green and the original prompt is fully complete"

then the deployment is **not yet complete**.

---

## Environment Reality vs. Prompt Assumptions

The prompt assumed:

- Ubuntu 24.04 LTS
- `Standard_D8s_v5`
- 8 vCPU
- 32 GB RAM
- 128 GB disk

The actual VM observed during execution differed:

- Ubuntu was newer than the prompt assumption
- available RAM was lower than the prompt expectation

This matters because the deployment and dry-run checks were written around the original VM assumptions.

---

## Phase Status

From `sudo ./setup.sh --status` on the deployed VM copy:

- Phase 1: DONE (`2026-03-29T07:48:24+00:00`)
- Phase 2: DONE (`2026-03-29T17:26:38+00:00`)
- Phase 3: DONE (`2026-03-29T07:48:37+00:00`)
- Phase 4: DONE (`2026-03-29T07:52:46+00:00`)
- Phase 5: DONE (`2026-03-29T07:53:33+00:00`)
- Phase 6: DONE (`2026-03-29T07:53:33+00:00`)
- Phase 7: **pending**
- Phase 8: DONE (`2026-03-29T08:33:11+00:00`)

### Interpretation

The setup state is inconsistent with the actual recovered runtime:

- several services were manually repaired and rebuilt after the original deploy attempt
- backends are healthy now
- but the phase marker for Phase 7 was never cleanly advanced to done after the manual recovery work

---

## Verified Healthy Backend Services

Host-side health verification confirmed these backend ports returned success:

- `4003` platform-service
- `4005` extraction-service
- `4007` mcp-server
- `4010` peakpulse
- `4011` chronomind
- `4012` jarvisjr
- `4013` nomgap
- `4014` mindlyst
- `4015` lysnrai
- `4016` notelett
- `4017` flowmonk
- `4018` actiontrail
- `4019` localmemgpt

This was verified with host-context curls against `http://127.0.0.1:<port>/health`.

---

## Current Container State Summary

### Healthy infrastructure / service containers

- cosmos-emulator
- azurite
- mailpit
- loki
- grafana
- caddy
- platform-service
- extraction-service
- mcp-server
- all 10 product backends

### Healthy internal web and ingress surfaces

- `admin.bytelyst.com`
- `tracker.bytelyst.com`
- `gitea.bytelyst.com`
- `llmlab.bytelyst.com`
- `ollama.bytelyst.com`
- `api.bytelyst.com`

### Present but not yet added to the active VM stack

- Prometheus
- Alertmanager
- node-exporter
- cadvisor
- Valkey / Redis-compatible cache
- PostgreSQL + pgvector

These are not currently running containers on the VM snapshot described by this document.

### Not fully resolved

- `admin-web`
- `tracker-web`

### Deliberately deprioritized during the final pass

These were not needed for the backend-only goal:

- product web apps
- LLM Lab dashboard

Some of those web containers were partially repaired and some became reachable, but the final backend-only objective explicitly skipped them.

---

## Manual Fixes Applied During Recovery

The deployment required several manual fixes on the **deployed VM copies under `/opt/bytelyst/`**, not in this repo working tree.

### 1. Gitea package metadata host fix

Problem:

- package tarballs were advertised with `http://localhost:3300/...`
- Docker builds inside containers could not fetch those packages

Fix:

- changed the deployed Gitea setup to advertise package URLs via the Docker-reachable host IP
- recreated the Gitea registry container with the corrected host settings

Impact:

- allowed product Docker builds to fetch private `@bytelyst/*` packages successfully

### 2. Re-published shared packages needed by downstream builds

Published corrected package versions in the local Gitea registry:

- `@bytelyst/ui@0.1.1`
- `@bytelyst/llm-router@0.1.1`

Impact:

- fixed broken downstream web and dashboard builds that depended on those packages

### 3. Extraction-service healthcheck fix

Problem:

- extraction runtime container healthcheck used `wget`
- runtime image did not include `wget`

Fix:

- updated the deployed extraction-service runtime image to include `wget`

Impact:

- extraction-service became healthy

### 4. Product web build repairs

Examples:

- added missing `next-env.d.ts` to ChronoMind web
- moved several product web apps off `file:` package references and onto published Gitea package versions
- corrected some runtime `CMD` entrypoints for Next standalone images

Impact:

- many product web apps became buildable and at least partially runnable

### 5. Local LLM dashboard Dockerfile repair

Problem:

- dashboard Dockerfile had an invalid `COPY ... 2>/dev/null || true` pattern

Fix:

- replaced that with a valid Docker instruction and corrected runtime pathing

Impact:

- dashboard image became buildable

### 6. Dashboard standalone runtime fix

Problem:

- `admin-web` and `tracker-web` were built with Next.js standalone output enabled
- their runtime images still attempted to start from a deployed package layout that did not contain the expected standalone entrypoint

Fix:

- changed both dashboard Dockerfiles to copy `.next/standalone` and `.next/static`
- switched the runtime command to `node server.js`

Impact:

- aligns both images with the Next.js standalone artifact layout
- removes the immediate startup failure seen in the container logs

---

## Validation Results

### `/opt/bytelyst/check-health.sh`

At the time of the final backend-focused validation:

- backend and core service health checks were green
- `admin-web` and `tracker-web` were still red
- host-side probes for some optional/non-backend surfaces still failed

### `sudo ./setup.sh --dry-run`

Dry-run reported `13/17` checks passed.

Failures included:

- disk/RAM checks reported `0 GB`
- Ollama service running
- GitHub reachable
- Phase 7 pending

### Important dry-run caveat

The RAM and disk failures were not trustworthy as-is because the dry-run parsing logic itself emitted `awk` errors and produced `0 GB` values.

That means the dry-run output should be treated as:

---

## Recommended Phase 2 Additions

The next infrastructure additions for this VM should be introduced in this order:

1. Prometheus + node-exporter + cadvisor
2. Valkey
3. PostgreSQL + pgvector only when a concrete service requires it

### Why this order

- Grafana and Loki are already running, so metrics collection is the biggest missing observability gap
- Valkey is low-overhead and unlocks shared cache, pub/sub, rate-limits, and SSE fan-out
- PostgreSQL + pgvector adds more operational weight and should be installed only for an actual relational/vector workload

### Internal-only hosting policy

These services should remain VM-hosted and internal-only:

- Grafana
- Loki
- Prometheus
- Alertmanager
- admin-web
- tracker-web
- any future logs, traces, metrics, or ops dashboards

Do not expose raw service ports publicly. Front any browser-accessed internal tool through Caddy with authentication and preferably IP allowlisting, VPN, or SSO.

- useful for broad status
- not authoritative for RAM/disk capacity on this VM

---

## What Is Completed So Far

### Completed deployment capabilities

- base system dependencies installed
- local registry installed and working
- repos cloned on the VM
- shared packages built and published
- environment generated
- backend fleet deployed and healthy
- host-side health script generated

### Completed operational outcome

The VM can currently serve the backend API surface required by:

- platform clients
- product backends
- backend-to-backend traffic
- backend health validation

This makes the VM usable for backend integration testing and further deployment hardening.

---

## What Remains

### High priority

1. Resolve `admin-web`
2. Resolve `tracker-web`
3. Bring Phase 7 state into a truthful completed state

### Medium priority

4. Normalize the web runtime images so all product web containers use the correct Next standalone entrypoint
5. Re-run the full end-to-end prompt validation after web surfaces are repaired

### Security / hardening priority

6. Stop documenting raw `http://<vm-ip>:<port>` exposure as the recommended client model
7. Move client-facing access behind a single HTTPS API hostname
8. Reduce public NSG exposure to `80/443` for app traffic
9. Lock down or remove public access to operational ports:
   - `3000`
   - `3300`
   - `8025`
   - `8080`
   - `10000`
   - `11434`

### Nice to have

10. Correct the dry-run disk/RAM check parsing so it reflects real system capacity

---

## Recommended Next Steps

### If the goal is backend-only readiness

The VM is already in acceptable shape.

Recommended action:

- treat the backend deployment as operational
- document the two remaining dashboard gaps
- move on to HTTPS and gateway hardening

### If the goal is full prompt completion

Do this next:

1. finish `admin-web`
2. finish `tracker-web`
3. re-run full health validation
4. update phase markers or complete Phase 7 cleanly

### If the goal is client-facing production readiness

Do not expose raw backend ports directly.

Instead:

1. front the APIs behind one HTTPS domain
2. route by path prefix
3. keep backend ports private on the Docker network

That recommendation is documented separately in:

- [`SECURE_API_EXPOSURE.md`](./SECURE_API_EXPOSURE.md)

---

## Status Classification

### Backend deployment

- **Status:** complete enough for backend use

### Full single-VM ecosystem deployment

- **Status:** partially complete

### Client-facing production-ready deployment

- **Status:** not complete

---

## Notes on Scope

This document records deployment status as observed during an execution and repair session on the VM.

It is intentionally:

- factual rather than aspirational
- focused on what was verified
- separate from the secure API exposure decision doc

It should be updated again after:

- `admin-web` is fixed
- `tracker-web` is fixed
- the final phase-state inconsistency is resolved
