# ByteLyst Single-VM Kubernetes Deployment (k3s)

> Deploy the **entire ByteLyst ecosystem** (30 services, 10 products) on Kubernetes
> using **k3s** — a lightweight, CNCF-certified K8s distribution.
> Production-grade for ~50 beta users on a single Azure VM.

---

## Quick Start

```bash
# Step 1: Run Docker setup phases 1-5 (system deps, Gitea, repos, packages)
cd /opt/bytelyst/learning_ai_common_plat/docs/devops/single_azure_vm
sudo ./docker/setup.sh --resume      # Runs phases 1-5 (skip 6-8)

# Step 2: Deploy to Kubernetes
sudo ./k8s/setup-k8s.sh              # 6 phases: preflight → k3s → images → config → deploy → health

# Step 3: Verify
/opt/bytelyst/check-health-k8s.sh    # 32 health checks
kubectl get pods -A                   # All pods
```

## Prerequisites

- **Azure VM:** Ubuntu 24.04 LTS, **Standard_D8s_v5** (8 vCPU, 32 GB RAM, 128 GB disk)
- **Docker setup phases 1-5 completed** (system deps, Gitea, repos, packages built + published)

## Why k3s?

| Feature | k3s | minikube | kind | microk8s |
|---------|-----|----------|------|----------|
| RAM overhead | ~512 MB | ~2 GB | ~1 GB | ~800 MB |
| Production-grade | Yes (CNCF certified) | No | No | Yes |
| Built-in Traefik | Yes | No | No | Optional |
| Single binary | Yes | No | No | No (snap) |
| SQLite backend | Yes (no etcd needed) | N/A | N/A | Dqlite |

## Architecture

```
Ubuntu 24.04 VM
├── k3s (single-node cluster)
│   ├── kube-system namespace
│   │   ├── CoreDNS
│   │   ├── Traefik Ingress Controller
│   │   ├── Local Path Provisioner
│   │   └── Metrics Server
│   │
│   ├── bytelyst-infra namespace
│   │   ├── cosmos-emulator (StatefulSet + PVC)
│   │   ├── azurite (StatefulSet + PVC)
│   │   ├── mailpit (Deployment)
│   │   ├── loki (StatefulSet + PVC)
│   │   └── grafana (Deployment + PVC)
│   │
│   ├── bytelyst-platform namespace
│   │   ├── platform-service (Deployment, replicas: 1)
│   │   ├── extraction-service (Deployment, replicas: 1)
│   │   └── mcp-server (Deployment, replicas: 1)
│   │
│   ├── bytelyst-dashboards namespace
│   │   ├── admin-web (Deployment, replicas: 1)
│   │   └── tracker-web (Deployment, replicas: 1)
│   │
│   └── bytelyst-products namespace
│       ├── *-backend (10 Deployments)
│       └── *-web (9 Deployments)
│
├── Ollama (systemd, host network — :11434)
└── Gitea (Docker container — :3300, used for build-time only)
```

## File Structure

```
k8s/
├── README.md                    # This file
├── setup-k8s.sh                 # Bootstrap script (6 phases)
├── namespaces.yaml              # 4 namespaces
├── config/
│   ├── configmap.yaml           # Shared env vars (replaces .env.ecosystem)
│   └── secrets.yaml             # JWT_SECRET template (generated at deploy)
├── infra/
│   ├── cosmos-emulator.yaml     # StatefulSet + Service + PVC + NodePort
│   ├── azurite.yaml             # StatefulSet + Service + PVC + NodePort
│   ├── mailpit.yaml             # Deployment + Service + NodePort
│   ├── loki.yaml                # StatefulSet + Service + PVC + NodePort
│   ├── grafana.yaml             # Deployment + Service + PVC + NodePort
│   └── ollama-external.yaml     # Service + Endpoints → host Ollama
├── platform/
│   ├── platform-service.yaml    # Deployment + Service + NodePort (:4003)
│   ├── extraction-service.yaml  # Deployment + Service + NodePort (:4005)
│   └── mcp-server.yaml          # Deployment + Service + NodePort (:4007)
├── dashboards/
│   ├── admin-web.yaml           # Deployment + Service + NodePort (:3001)
│   └── tracker-web.yaml         # Deployment + Service + NodePort (:3003)
└── products/
    ├── backends.yaml            # 10 backend Deployments + Services + NodePorts
    └── webs.yaml                # 9 web Deployments + Services + NodePorts
```

## Setup Phases

| Phase | Duration | What happens |
|-------|----------|--------------|
| 1. Pre-flight | ~10s | Verify Docker phases 1-5 completed, check disk/RAM |
| 2. Install k3s | ~2 min | k3s with Docker runtime, NodePort range 1024-32767 |
| 3. Build images | ~15 min | Docker compose build + tag as `bytelyst/<service>:latest` |
| 4. Generate config | ~30s | Namespaces, ConfigMap (3 copies), Secrets (JWT), Ollama endpoint |
| 5. Deploy | ~5 min | Apply manifests: infra → platform → dashboards → products |
| 6. Health check | ~1 min | 32 endpoint checks + kubectl pod status |

## Key Design Decisions

### k3s with Docker Runtime
k3s installed with `--docker` flag — reuses existing Docker daemon and images.
No `containerd` import step needed. Same images used by Docker Compose work directly.

### 4-Namespace Isolation
- **bytelyst-infra** — Cosmos emulator, Azurite, Mailpit, Loki, Grafana
- **bytelyst-platform** — platform-service, extraction-service, mcp-server
- **bytelyst-dashboards** — admin-web, tracker-web
- **bytelyst-products** — 10 backends + 9 web apps

ConfigMap + Secrets are copied to all 3 app namespaces by the setup script.

### Cross-Namespace DNS
K8s DNS: `<service>.<namespace>.svc.cluster.local`
- Backends reach Cosmos: `cosmos-emulator.bytelyst-infra.svc:8081`
- Webs reach backends: `flowmonk-backend.bytelyst-products.svc:4017`
- Everything reaches platform: `platform-service.bytelyst-platform.svc:4003`

### Ollama as External Service
Ollama stays on the host (systemd). A headless Service + Endpoints in `bytelyst-infra`
points to the node's internal IP. Pods reach it as `ollama.bytelyst-infra.svc:11434`.
Setup script auto-detects the node IP.

### NodePort for External Access
All services use the **same ports** as Docker Compose (e.g., `:4003`, `:3002`, `:3030`).
k3s is configured with `--kube-apiserver-arg=service-node-port-range=1024-32767`.

### Resource Limits (tuned for 32 GB VM, 50 beta users)

| Service type | CPU request | CPU limit | Memory request | Memory limit |
|-------------|------------|-----------|---------------|-------------|
| Backend (×10) | 100m | 500m | 256Mi | 512Mi |
| Web app (×9) | 100m | 500m | 256Mi | 512Mi |
| Platform (×3) | 200m | 1000m | 384Mi | 768Mi |
| Cosmos emulator | 500m | 2000m | 2Gi | 3Gi |
| Grafana | 100m | 500m | 128Mi | 256Mi |
| Mailpit / Loki | 50-100m | 500m | 64-128Mi | 512Mi |
| k3s overhead | — | — | — | ~512Mi |
| Ollama (host) | — | — | — | ~3Gi |
| **Total** | | | **~10 Gi** | **~19 Gi** |

Fits comfortably in 32 GB with ~13 GB headroom.

### Readiness + Liveness Probes
Every service gets both:
- **Readiness:** `GET /health` every 10s (traffic only when ready)
- **Liveness:** `GET /health` every 30s (auto-restart on failure)
- Backends: `initialDelaySeconds: 15`, Web apps: `initialDelaySeconds: 15`
- Cosmos emulator: `initialDelaySeconds: 60` (slow startup)

## Operations Cheat Sheet

```bash
# ── Cluster status ─────────────────────────────────
kubectl get nodes                              # Node health
kubectl get pods -A                            # All pods
kubectl top pods -A                            # Resource usage (CPU/memory)

# ── Deploy / update ────────────────────────────────
kubectl apply -f k8s/products/                 # Re-apply product manifests
kubectl rollout restart deploy/flowmonk-backend -n bytelyst-products  # Rolling restart

# ── Scaling (for load testing) ─────────────────────
kubectl scale deploy/platform-service --replicas=2 -n bytelyst-platform
kubectl autoscale deploy/flowmonk-backend --min=1 --max=3 --cpu-percent=70 -n bytelyst-products

# ── Debugging ──────────────────────────────────────
kubectl logs deploy/platform-service -n bytelyst-platform -f        # Stream logs
kubectl describe pod <name> -n bytelyst-platform                    # Pod events
kubectl exec -it deploy/platform-service -n bytelyst-platform -- sh # Shell into pod

# ── Teardown ───────────────────────────────────────
sudo ./setup-k8s.sh --teardown                 # Delete all namespaces (keep k3s)
/usr/local/bin/k3s-uninstall.sh                # Uninstall k3s completely
```

## Port Map (same as Docker Compose)

| Service | Port | Health check |
|---------|------|-------------|
| Gitea (npm) | 3300 | `http://localhost:3300/api/v1/version` |
| Ollama (LLM) | 11434 | `http://localhost:11434/api/version` |
| Cosmos Explorer | 1234 | `http://localhost:1234` |
| Azurite (Blob) | 10000 | `http://localhost:10000/devstoreaccount1?comp=list` |
| Mailpit UI | 8025 | `http://localhost:8025` |
| Loki | 3100 | `http://localhost:3100/ready` |
| Grafana | 3000 | `http://localhost:3000/api/health` |
| platform-service | 4003 | `/health` |
| extraction-service | 4005 | `/health` |
| mcp-server | 4007 | `/health` |
| admin-web | 3001 | `/` |
| tracker-web | 3003 | `/` |
| Backends | 4010-4019 | `/health` |
| Web apps | 3002, 3030, 3035, 3040, 3045, 3050, 3055, 3060, 3070 | `/` |

## Switching Between Docker Compose and K8s

Both approaches coexist on the same VM:

```bash
# Docker → K8s
cd /opt/bytelyst/learning_ai_common_plat
docker compose -f docker-compose.ecosystem.yml down   # Stop compose stack
sudo ../docs/devops/single_azure_vm/k8s/setup-k8s.sh  # Deploy to k3s

# K8s → Docker
sudo ./setup-k8s.sh --teardown                        # Remove k8s resources
sudo ../docker/setup.sh --phase=7                      # Re-deploy via compose
```

Both share: Gitea registry (Docker container), Ollama (systemd), and built Docker images.
