diff --git a/REPO_CONTEXT.md b/REPO_CONTEXT.md new file mode 100644 index 0000000..6bfbd77 --- /dev/null +++ b/REPO_CONTEXT.md @@ -0,0 +1,355 @@ +# ByteLyst DevOps Tools - Repository Context + +> **Purpose**: This file provides quick context for AI coding agents about what this repository contains and how to use it effectively. + +## What This Repository Provides + +This is the **operational tools repository** for the ByteLyst ecosystem. It contains: + +- **Deployment scripts** for all products (deploy-{project}.sh pattern) +- **GitHub administration tools** and multi-repo safety helpers +- **Operational automation** scripts +- **Internal DevOps dashboard** (dashboard/ - full product) +- **Cross-repo utilities** and maintenance scripts + +## Key Locations + +### Deployment Scripts (MOST IMPORTANT FOR PRODUCT DEPLOYMENTS) +- **Location**: Root level `deploy-*.sh` files +- **Pattern**: `deploy-{project}.sh` +- **Available Scripts**: + - `deploy-invttrdg.sh` - Investment Trading deployment + - `deploy-notes.sh` - Notes deployment + - `deploy-clock.sh` - Clock deployment + - `deploy-all.sh` - Deploy all products +- **Always check here first** before attempting custom deployment solutions + +### Deployment Script Pattern +All deployment scripts follow this standard pattern: +1. Dirty checks (uncommitted changes, unpushed commits) +2. Pull and rebase from origin/main +3. Run smoke tests (if available) +4. Build Docker images with `--network host` (critical for Gitea registry access) +5. Deploy containers via docker-compose +6. Health checks and endpoint verification + +### Critical Deployment Configuration +- **Docker Build**: Uses `--network host` flag for Gitea registry access +- **Gitea Token**: Required for @bytelyst/* package installation +- **Token Location**: + - Environment variable: `GITEA_NPM_TOKEN` + - Fallback 1: `/opt/bytelyst/.gitea_token` + - Fallback 2: `$HOME/.gitea_npm_token` +- **Build Metadata**: Automatically includes commit SHA, branch, author, build time + +### GitHub Administration Tools +- **bytelyst-cli.sh**: Main CLI for common GitHub operations +- **remove_user_interactive.sh**: Interactive collaborator removal +- **remove_user_guided.sh**: Guided removal workflow +- **remove_user_from_repos.sh**: Scripted removal flow + +### Multi-Repo Git Safety Tools +- **Location**: `git-work-safety-tools/` +- **Purpose**: Safe bulk git workflows across multiple repositories +- **Key Scripts**: + - `git_repos_status.sh` - Scan many repos for dirty state + - `git_repos_rebase_commit_push.sh` - Safe rebase + commit + push + - `multi_repo_safe_push.sh` - Safe multi-repo pushing + +### Scripts Directory +- **Location**: `scripts/` +- **Purpose**: Self-contained operational scripts +- **Pattern**: More focused than root-level helpers + +### Documentation +- **Location**: `docs/` +- **Key Files**: + - `getting-started.md` - Onboarding guide + - `repo-map.md` - Repository structure map + - `tooling-status.md` - Tool availability status + - `operations.md` - Operational patterns + +## Common Usage Patterns + +### Deployment (Most Common Use Case) +```bash +# Deploy specific product +./deploy-invttrdg.sh + +# Force deployment (skip dirty checks) +./deploy-invttrdg.sh --force + +# Skip health checks +./deploy-invttrdg.sh --skip-health-check + +# Deploy all products +./deploy-all.sh +``` + +### GitHub Operations +```bash +# List public repos for a user +./bytelyst-cli.sh list-public-repos --user + +# List private repos for an org +./bytelyst-cli.sh list-private-repos --org + +# Interactive user removal +./remove_user_interactive.sh +``` + +### Multi-Repo Safety +```bash +# Check status of multiple repos +./git-work-safety-tools/git_repos_status.sh + +# Safe rebase + commit + push across repos +./git-work-safety-tools/git_repos_rebase_commit_push.sh +``` + +## Deployment Script Details + +### Standard Deployment Flow +1. **Prerequisites Check**: Verify repo directory exists +2. **Dirty Checks**: + - Uncommitted changes (fail unless --force) + - Untracked files (fail unless --force) + - Unpushed commits (fail unless --force) +3. **Git Sync**: Pull and rebase origin/main +4. **Smoke Tests**: Run `scripts/smoke-release.sh` if available +5. **Docker Build**: + - Build backend and web images + - Use `--network host` for Gitea registry access + - Pass GITEA_NPM_TOKEN via BuildKit secrets + - Include build metadata (commit SHA, branch, etc.) +6. **Deployment**: `docker compose up -d --no-build` +7. **Health Checks**: Verify local and production endpoints + +### Docker Build Configuration +```bash +docker build --network host \ + --secret id=gitea_npm_token,env=GITEA_NPM_TOKEN \ + --build-arg "BYTELYST_COMMIT_SHA=${COMMIT_SHA}" \ + --build-arg "BYTELYST_BRANCH=${BRANCH}" \ + -f Dockerfile -t image:tag . +``` + +### Health Check Endpoints +- **Backend**: `http://localhost:4025/health/live` +- **Web**: `http://localhost:3085` +- **Production API**: `https://api.bytelyst.com/invttrdg` +- **Production Web**: `https://invttrdg.bytelyst.com` + +## Important Notes + +### Before Deployment +1. **Always check deployment scripts here first** - don't build custom deployment solutions +2. Ensure GITEA_NPM_TOKEN is set correctly +3. Run smoke tests if available +4. Check git status for uncommitted changes + +### Common Deployment Issues +- **"Unauthorized - 401" from Gitea**: Wrong token, check GITEA_NPM_TOKEN +- **"Cannot find module @bytelyst/..."**: Package not built, check learning_ai_common_plat +- **Docker build network errors**: Missing --network host flag +- **Port conflicts**: Check if containers are already running + +### Gitea Registry Access +- **Registry URL**: `http://localhost:3300/api/packages/bytelyst/npm/` +- **Access Method**: --network host flag in Docker build +- **Why this works**: Allows build container to access host's localhost:3300 +- **Alternative approaches** (network bridges, container names) typically fail + +### Build Metadata +Deployment scripts automatically inject: +- `BYTELYST_COMMIT_SHA`: Short commit hash +- `BYTELYST_COMMIT_SHA_FULL`: Full commit hash +- `BYTELYST_BRANCH`: Git branch name +- `BYTELYST_BUILT_AT`: ISO timestamp +- `BYTELYST_COMMIT_AUTHOR`: Commit author +- `BYTELYST_COMMIT_MESSAGE`: Commit message (truncated) +- `BYTELYST_DOCKER_IMAGE`: Docker image tag + +## Internal DevOps Dashboard + +### Dashboard Product +- **Location**: `dashboard/` +- **Purpose**: Internal deployment orchestration and service monitoring +- **Architecture**: Full ByteLyst product (backend + web) +- **Backend**: Fastify 5 (port 4004) +- **Web**: Next.js 16 (port 3000) +- **Integration**: Uses @bytelyst/* packages, links to admin-web + +### Dashboard Setup +See `dashboard/README.md` for setup and usage instructions. + +## Safety and Operational Guidelines + +### Sensitive Files +- `accounts.json` - GitHub account credentials +- `.env` files - Environment configuration +- Generated contributor/output data - Operational snapshots +- **Never echo these files** unless task requires it + +### Destructive Operations +- Many scripts are operational and potentially destructive +- Preserve prompts, dry-run behavior, and confirmations +- Review scripts before automation use +- Some encode assumptions about ByteLyst org structure + +### Pre-commit Validation +```bash +pre-commit run --all-files +``` + +## Quick Reference for Common Tasks + +| Task | Command | +|------|---------| +| Deploy invttrdg | `./deploy-invttrdg.sh` | +| Force deployment | `./deploy-invttrdg.sh --force` | +| Deploy all products | `./deploy-all.sh` | +| Check repo status | `./git-work-safety-tools/git_repos_status.sh` | +| List user repos | `./bytelyst-cli.sh list-public-repos --user ` | +| Remove user interactively | `./remove_user_interactive.sh` | +| Run pre-commit checks | `pre-commit run --all-files` | + +## Related Repositories + +- **learning_ai_common_plat**: Shared platform packages consumed by products +- **Product repos**: learning_ai_invt_trdg, learning_ai_notes, etc. (deployed via scripts here) + +## First Steps for Any Deployment Task + +1. **Check deployment scripts first** - look for `deploy-{project}.sh` +2. **Read the script** - understand its flow and requirements +3. **Check GITEA_NPM_TOKEN** - ensure it's set correctly +4. **Verify companion repos** - ensure learning_ai_common_plat is accessible +5. **Run the deployment script** - use appropriate flags (--force, --skip-health-check) +6. **Never build custom deployment** unless script doesn't exist for your use case + +## New Deployment Pattern for @bytelyst/* Packages (Learned 2026-05) + +### Background +Products that consume @bytelyst/* packages from learning_ai_common_plat require a special build preparation step because: +- Gitea registry is not accessible from Docker build context +- Direct npm install would fail during Docker build +- Dependencies need to be pre-packaged as tarballs + +### Required Pre-Build Step +**Products using @bytelyst/* packages MUST have a `scripts/docker-prep.sh` script:** + +```bash +# Before building Docker images +bash scripts/docker-prep.sh + +# After building Docker images +bash scripts/docker-prep.sh --restore +``` + +### Dockerfile Changes Required +**Dockerfiles MUST be updated to remove Gitea secret mounting:** + +**Before (incorrect):** +```dockerfile +RUN --mount=type=secret,id=gitea_npm_token \ + TOKEN=$(cat /run/secrets/gitea_npm_token) && \ + echo "//localhost:3300/api/packages/bytelyst/npm/:_authToken=$TOKEN" >> .npmrc && \ + pnpm install --ignore-scripts --lockfile=false +``` + +**After (correct):** +```dockerfile +COPY .docker-deps/ ../.docker-deps/ +RUN pnpm install --ignore-scripts --lockfile=false +``` + +### Environment Variable Configuration +**NEXT_PUBLIC_ variables MUST use public API URLs:** + +**Incorrect:** +```yaml +NEXT_PUBLIC_BACKEND_URL: http://backend:4011 +``` + +**Correct:** +```yaml +NEXT_PUBLIC_BACKEND_URL: https://api.bytelyst.com/{product} +``` + +### Docker Compose Format +**Use YAML mapping syntax, not array syntax:** + +**Incorrect:** +```yaml +environment: + - NODE_ENV=production + - PORT:4011 +``` + +**Correct:** +```yaml +environment: + NODE_ENV: production + PORT: 4011 +``` + +### Container Naming +**Container names MUST match Caddyfile configuration:** + +```yaml +services: + backend: + container_name: {product}-backend + web: + container_name: {product}-web +``` + +### DNS Setup +**Add DNS records via GoDaddy API for new subdomains:** + +```bash +curl -X PUT "https://api.godaddy.com/v1/domains/bytelyst.com/records/A/{subdomain}" \ + -H "Authorization: sso-key $GODADDY_API_KEY:$GODADDY_API_SECRET" \ + -H "Content-Type: application/json" \ + -d '[{"data": "187.124.159.82", "name": "{subdomain}", "ttl": 600, "type": "A"}]' +``` + +### Caddy Configuration +**Add subdomain routing to /opt/bytelyst/Caddyfile:** + +```caddy +{subdomain}.bytelyst.com { + encode gzip + reverse_proxy {product}-web:{port} +} +``` + +### Products Requiring This Pattern +- learning_ai_notes (NoteLett) +- learning_ai_clock (ChronoMind) +- learning_ai_invttrdg (Investment Trading) +- Any product consuming @bytelyst/* packages + +### Deployment Script Updates +**Deployment scripts should be updated to:** +1. Check for docker-prep.sh existence +2. Run docker-prep.sh before Docker build +3. Run docker-prep.sh --restore after successful deployment +4. Provide option to skip docker-prep if already run + +### Health Check Requirements +**Backend Dockerfiles MUST include wget for health checks:** + +```dockerfile +RUN apk add --no-cache wget +``` + +Health check in docker-compose.yml: +```yaml +healthcheck: + test: ["CMD", "wget", "--spider", "-q", "http://localhost:{port}/health"] + interval: 30s + timeout: 5s + retries: 3 +``` \ No newline at end of file