# ByteLyst DevOps Tools - Repository Context

> **Purpose**: This file provides quick context for AI coding agents about what this repository contains and how to use it effectively.

## What This Repository Provides

This is the **operational tools repository** for the ByteLyst ecosystem. It contains:

- **Deployment scripts** for all products (deploy-{project}.sh pattern)
- **GitHub administration tools** and multi-repo safety helpers
- **Operational automation** scripts
- **Internal DevOps dashboard** (dashboard/ - full product)
- **Cross-repo utilities** and maintenance scripts

## Key Locations

### Deployment Scripts (MOST IMPORTANT FOR PRODUCT DEPLOYMENTS)
- **Location**: Root level `deploy-*.sh` files
- **Pattern**: `deploy-{project}.sh`
- **Available Scripts**:
  - `deploy-invttrdg.sh` - Investment Trading deployment
  - `deploy-notes.sh` - Notes deployment
  - `deploy-clock.sh` - Clock deployment
  - `deploy-all.sh` - Deploy all products
- **Always check here first** before attempting custom deployment solutions

### Deployment Script Pattern
All deployment scripts follow this standard pattern:
1. Dirty checks (uncommitted changes, unpushed commits)
2. Pull and rebase from origin/main
3. Run smoke tests (if available)
4. Build Docker images with `--network host` (critical for Gitea registry access)
5. Deploy containers via docker-compose
6. Health checks and endpoint verification

### Critical Deployment Configuration
- **Docker Build**: Uses `--network host` flag for Gitea registry access
- **Gitea Token**: Required for @bytelyst/* package installation
- **Token Location**: 
  - Environment variable: `GITEA_NPM_TOKEN`
  - Fallback 1: `/opt/bytelyst/.gitea_token`
  - Fallback 2: `$HOME/.gitea_npm_token`
- **Build Metadata**: Automatically includes commit SHA, branch, author, build time

### GitHub Administration Tools
- **bytelyst-cli.sh**: Main CLI for common GitHub operations
- **remove_user_interactive.sh**: Interactive collaborator removal
- **remove_user_guided.sh**: Guided removal workflow
- **remove_user_from_repos.sh**: Scripted removal flow

### Multi-Repo Git Safety Tools
- **Location**: `git-work-safety-tools/`
- **Purpose**: Safe bulk git workflows across multiple repositories
- **Key Scripts**:
  - `git_repos_status.sh` - Scan many repos for dirty state
  - `git_repos_rebase_commit_push.sh` - Safe rebase + commit + push
  - `multi_repo_safe_push.sh` - Safe multi-repo pushing

### Scripts Directory
- **Location**: `scripts/`
- **Purpose**: Self-contained operational scripts
- **Pattern**: More focused than root-level helpers

### Documentation
- **Location**: `docs/`
- **Key Files**:
  - `getting-started.md` - Onboarding guide
  - `repo-map.md` - Repository structure map
  - `tooling-status.md` - Tool availability status
  - `operations.md` - Operational patterns

## Common Usage Patterns

### Deployment (Most Common Use Case)
```bash
# Deploy specific product
./deploy-invttrdg.sh

# Force deployment (skip dirty checks)
./deploy-invttrdg.sh --force

# Skip health checks
./deploy-invttrdg.sh --skip-health-check

# Deploy all products
./deploy-all.sh
```

### GitHub Operations
```bash
# List public repos for a user
./bytelyst-cli.sh list-public-repos --user <username>

# List private repos for an org
./bytelyst-cli.sh list-private-repos --org <orgname>

# Interactive user removal
./remove_user_interactive.sh
```

### Multi-Repo Safety
```bash
# Check status of multiple repos
./git-work-safety-tools/git_repos_status.sh

# Safe rebase + commit + push across repos
./git-work-safety-tools/git_repos_rebase_commit_push.sh
```

## Deployment Script Details

### Standard Deployment Flow
1. **Prerequisites Check**: Verify repo directory exists
2. **Dirty Checks**: 
   - Uncommitted changes (fail unless --force)
   - Untracked files (fail unless --force)
   - Unpushed commits (fail unless --force)
3. **Git Sync**: Pull and rebase origin/main
4. **Smoke Tests**: Run `scripts/smoke-release.sh` if available
5. **Docker Build**:
   - Build backend and web images
   - Use `--network host` for Gitea registry access
   - Pass GITEA_NPM_TOKEN via BuildKit secrets
   - Include build metadata (commit SHA, branch, etc.)
6. **Deployment**: `docker compose up -d --no-build`
7. **Health Checks**: Verify local and production endpoints

### Docker Build Configuration
```bash
docker build --network host \
  --secret id=gitea_npm_token,env=GITEA_NPM_TOKEN \
  --build-arg "BYTELYST_COMMIT_SHA=${COMMIT_SHA}" \
  --build-arg "BYTELYST_BRANCH=${BRANCH}" \
  -f Dockerfile -t image:tag .
```

### Health Check Endpoints
- **Backend**: `http://localhost:4025/health/live`
- **Web**: `http://localhost:3085`
- **Production API**: `https://api.bytelyst.com/invttrdg`
- **Production Web**: `https://invttrdg.bytelyst.com`

## Important Notes

### Before Deployment
1. **Always check deployment scripts here first** - don't build custom deployment solutions
2. Ensure GITEA_NPM_TOKEN is set correctly
3. Run smoke tests if available
4. Check git status for uncommitted changes

### Common Deployment Issues
- **"Unauthorized - 401" from Gitea**: Wrong token, check GITEA_NPM_TOKEN
- **"Cannot find module @bytelyst/..."**: Package not built, check learning_ai_common_plat
- **Docker build network errors**: Missing --network host flag
- **Port conflicts**: Check if containers are already running

### Gitea Registry Access
- **Registry URL**: `http://localhost:3300/api/packages/bytelyst/npm/`
- **Access Method**: --network host flag in Docker build
- **Why this works**: Allows build container to access host's localhost:3300
- **Alternative approaches** (network bridges, container names) typically fail

### Build Metadata
Deployment scripts automatically inject:
- `BYTELYST_COMMIT_SHA`: Short commit hash
- `BYTELYST_COMMIT_SHA_FULL`: Full commit hash
- `BYTELYST_BRANCH`: Git branch name
- `BYTELYST_BUILT_AT`: ISO timestamp
- `BYTELYST_COMMIT_AUTHOR`: Commit author
- `BYTELYST_COMMIT_MESSAGE`: Commit message (truncated)
- `BYTELYST_DOCKER_IMAGE`: Docker image tag

## Internal DevOps Dashboard

### Dashboard Product
- **Location**: `dashboard/`
- **Purpose**: Internal deployment orchestration and service monitoring
- **Architecture**: Full ByteLyst product (backend + web)
- **Backend**: Fastify 5 (port 4004)
- **Web**: Next.js 16 (port 3000)
- **Integration**: Uses @bytelyst/* packages, links to admin-web

### Dashboard Setup
See `dashboard/README.md` for setup and usage instructions.

## Safety and Operational Guidelines

### Sensitive Files
- `accounts.json` - GitHub account credentials
- `.env` files - Environment configuration
- Generated contributor/output data - Operational snapshots
- **Never echo these files** unless task requires it

### Destructive Operations
- Many scripts are operational and potentially destructive
- Preserve prompts, dry-run behavior, and confirmations
- Review scripts before automation use
- Some encode assumptions about ByteLyst org structure

### Pre-commit Validation
```bash
pre-commit run --all-files
```

## Quick Reference for Common Tasks

| Task | Command |
|------|---------|
| Deploy invttrdg | `./deploy-invttrdg.sh` |
| Force deployment | `./deploy-invttrdg.sh --force` |
| Deploy all products | `./deploy-all.sh` |
| Check repo status | `./git-work-safety-tools/git_repos_status.sh` |
| List user repos | `./bytelyst-cli.sh list-public-repos --user <user>` |
| Remove user interactively | `./remove_user_interactive.sh` |
| Run pre-commit checks | `pre-commit run --all-files` |

## Related Repositories

- **learning_ai_common_plat**: Shared platform packages consumed by products
- **Product repos**: learning_ai_invt_trdg, learning_ai_notes, etc. (deployed via scripts here)

## First Steps for Any Deployment Task

1. **Check deployment scripts first** - look for `deploy-{project}.sh`
2. **Read the script** - understand its flow and requirements
3. **Check GITEA_NPM_TOKEN** - ensure it's set correctly
4. **Verify companion repos** - ensure learning_ai_common_plat is accessible
5. **Run the deployment script** - use appropriate flags (--force, --skip-health-check)
6. **Never build custom deployment** unless script doesn't exist for your use case

## New Deployment Pattern for @bytelyst/* Packages (Learned 2026-05)

### Background
Products that consume @bytelyst/* packages from learning_ai_common_plat require a special build preparation step because:
- Gitea registry is not accessible from Docker build context
- Direct npm install would fail during Docker build
- Dependencies need to be pre-packaged as tarballs

### Required Pre-Build Step
**Products using @bytelyst/* packages MUST have a `scripts/docker-prep.sh` script:**

```bash
# Before building Docker images
bash scripts/docker-prep.sh

# After building Docker images
bash scripts/docker-prep.sh --restore
```

### Dockerfile Changes Required
**Dockerfiles MUST be updated to remove Gitea secret mounting:**

**Before (incorrect):**
```dockerfile
RUN --mount=type=secret,id=gitea_npm_token \
  TOKEN=$(cat /run/secrets/gitea_npm_token) && \
  echo "//localhost:3300/api/packages/bytelyst/npm/:_authToken=$TOKEN" >> .npmrc && \
  pnpm install --ignore-scripts --lockfile=false
```

**After (correct):**
```dockerfile
COPY .docker-deps/ ../.docker-deps/
RUN pnpm install --ignore-scripts --lockfile=false
```

### Environment Variable Configuration
**NEXT_PUBLIC_ variables MUST use public API URLs:**

**Incorrect:**
```yaml
NEXT_PUBLIC_BACKEND_URL: http://backend:4011
```

**Correct:**
```yaml
NEXT_PUBLIC_BACKEND_URL: https://api.bytelyst.com/{product}
```

### Docker Compose Format
**Use YAML mapping syntax, not array syntax:**

**Incorrect:**
```yaml
environment:
  - NODE_ENV=production
  - PORT:4011
```

**Correct:**
```yaml
environment:
  NODE_ENV: production
  PORT: 4011
```

### Container Naming
**Container names MUST match Caddyfile configuration:**

```yaml
services:
  backend:
    container_name: {product}-backend
  web:
    container_name: {product}-web
```

### DNS Setup
**Add DNS records via GoDaddy API for new subdomains:**

```bash
curl -X PUT "https://api.godaddy.com/v1/domains/bytelyst.com/records/A/{subdomain}" \
  -H "Authorization: sso-key $GODADDY_API_KEY:$GODADDY_API_SECRET" \
  -H "Content-Type: application/json" \
  -d '[{"data": "187.124.159.82", "name": "{subdomain}", "ttl": 600, "type": "A"}]'
```

### Caddy Configuration
**Add subdomain routing to /opt/bytelyst/Caddyfile:**

```caddy
{subdomain}.bytelyst.com {
  encode gzip
  reverse_proxy {product}-web:{port}
}
```

### Products Requiring This Pattern
- learning_ai_notes (NoteLett)
- learning_ai_clock (ChronoMind)
- learning_ai_invttrdg (Investment Trading)
- Any product consuming @bytelyst/* packages

### Deployment Script Updates
**Deployment scripts should be updated to:**
1. Check for docker-prep.sh existence
2. Run docker-prep.sh before Docker build
3. Run docker-prep.sh --restore after successful deployment
4. Provide option to skip docker-prep if already run

### Health Check Requirements
**Backend Dockerfiles MUST include wget for health checks:**

```dockerfile
RUN apk add --no-cache wget
```

Health check in docker-compose.yml:
```yaml
healthcheck:
  test: ["CMD", "wget", "--spider", "-q", "http://localhost:{port}/health"]
  interval: 30s
  timeout: 5s
  retries: 3
```