bytelyst/bytelyst-devops-tools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
416f25794c
bytelyst-devops-tools/REPO_CONTEXT.md
root 47c96d5db4 docs(devops): add repo context note
2026-05-18 09:01:09 +00:00

11 KiB
Raw Blame History

ByteLyst DevOps Tools - Repository Context

Purpose: This file provides quick context for AI coding agents about what this repository contains and how to use it effectively.

What This Repository Provides

This is the operational tools repository for the ByteLyst ecosystem. It contains:

  • Deployment scripts for all products (deploy-{project}.sh pattern)
  • GitHub administration tools and multi-repo safety helpers
  • Operational automation scripts
  • Internal DevOps dashboard (dashboard/ - full product)
  • Cross-repo utilities and maintenance scripts

Key Locations

Deployment Scripts (MOST IMPORTANT FOR PRODUCT DEPLOYMENTS)

  • Location: Root level deploy-*.sh files
  • Pattern: deploy-{project}.sh
  • Available Scripts:
    • deploy-invttrdg.sh - Investment Trading deployment
    • deploy-notes.sh - Notes deployment
    • deploy-clock.sh - Clock deployment
    • deploy-all.sh - Deploy all products
  • Always check here first before attempting custom deployment solutions

Deployment Script Pattern

All deployment scripts follow this standard pattern:

  1. Dirty checks (uncommitted changes, unpushed commits)
  2. Pull and rebase from origin/main
  3. Run smoke tests (if available)
  4. Build Docker images with --network host (critical for Gitea registry access)
  5. Deploy containers via docker-compose
  6. Health checks and endpoint verification

Critical Deployment Configuration

  • Docker Build: Uses --network host flag for Gitea registry access
  • Gitea Token: Required for @bytelyst/* package installation
  • Token Location:
    • Environment variable: GITEA_NPM_TOKEN
    • Fallback 1: /opt/bytelyst/.gitea_token
    • Fallback 2: $HOME/.gitea_npm_token
  • Build Metadata: Automatically includes commit SHA, branch, author, build time

GitHub Administration Tools

  • bytelyst-cli.sh: Main CLI for common GitHub operations
  • remove_user_interactive.sh: Interactive collaborator removal
  • remove_user_guided.sh: Guided removal workflow
  • remove_user_from_repos.sh: Scripted removal flow

Multi-Repo Git Safety Tools

  • Location: git-work-safety-tools/
  • Purpose: Safe bulk git workflows across multiple repositories
  • Key Scripts:
    • git_repos_status.sh - Scan many repos for dirty state
    • git_repos_rebase_commit_push.sh - Safe rebase + commit + push
    • multi_repo_safe_push.sh - Safe multi-repo pushing

Scripts Directory

  • Location: scripts/
  • Purpose: Self-contained operational scripts
  • Pattern: More focused than root-level helpers

Documentation

  • Location: docs/
  • Key Files:
    • getting-started.md - Onboarding guide
    • repo-map.md - Repository structure map
    • tooling-status.md - Tool availability status
    • operations.md - Operational patterns

Common Usage Patterns

Deployment (Most Common Use Case)

# Deploy specific product
./deploy-invttrdg.sh

# Force deployment (skip dirty checks)
./deploy-invttrdg.sh --force

# Skip health checks
./deploy-invttrdg.sh --skip-health-check

# Deploy all products
./deploy-all.sh

GitHub Operations

# List public repos for a user
./bytelyst-cli.sh list-public-repos --user <username>

# List private repos for an org
./bytelyst-cli.sh list-private-repos --org <orgname>

# Interactive user removal
./remove_user_interactive.sh

Multi-Repo Safety

# Check status of multiple repos
./git-work-safety-tools/git_repos_status.sh

# Safe rebase + commit + push across repos
./git-work-safety-tools/git_repos_rebase_commit_push.sh

Deployment Script Details

Standard Deployment Flow

  1. Prerequisites Check: Verify repo directory exists
  2. Dirty Checks:
    • Uncommitted changes (fail unless --force)
    • Untracked files (fail unless --force)
    • Unpushed commits (fail unless --force)
  3. Git Sync: Pull and rebase origin/main
  4. Smoke Tests: Run scripts/smoke-release.sh if available
  5. Docker Build:
    • Build backend and web images
    • Use --network host for Gitea registry access
    • Pass GITEA_NPM_TOKEN via BuildKit secrets
    • Include build metadata (commit SHA, branch, etc.)
  6. Deployment: docker compose up -d --no-build
  7. Health Checks: Verify local and production endpoints

Docker Build Configuration

docker build --network host \
  --secret id=gitea_npm_token,env=GITEA_NPM_TOKEN \
  --build-arg "BYTELYST_COMMIT_SHA=${COMMIT_SHA}" \
  --build-arg "BYTELYST_BRANCH=${BRANCH}" \
  -f Dockerfile -t image:tag .

Health Check Endpoints

  • Backend: http://localhost:4025/health/live
  • Web: http://localhost:3085
  • Production API: https://api.bytelyst.com/invttrdg
  • Production Web: https://invttrdg.bytelyst.com

Important Notes

Before Deployment

  1. Always check deployment scripts here first - don't build custom deployment solutions
  2. Ensure GITEA_NPM_TOKEN is set correctly
  3. Run smoke tests if available
  4. Check git status for uncommitted changes

Common Deployment Issues

  • "Unauthorized - 401" from Gitea: Wrong token, check GITEA_NPM_TOKEN
  • "Cannot find module @bytelyst/...": Package not built, check learning_ai_common_plat
  • Docker build network errors: Missing --network host flag
  • Port conflicts: Check if containers are already running

Gitea Registry Access

  • Registry URL: http://localhost:3300/api/packages/bytelyst/npm/
  • Access Method: --network host flag in Docker build
  • Why this works: Allows build container to access host's localhost:3300
  • Alternative approaches (network bridges, container names) typically fail

Build Metadata

Deployment scripts automatically inject:

  • BYTELYST_COMMIT_SHA: Short commit hash
  • BYTELYST_COMMIT_SHA_FULL: Full commit hash
  • BYTELYST_BRANCH: Git branch name
  • BYTELYST_BUILT_AT: ISO timestamp
  • BYTELYST_COMMIT_AUTHOR: Commit author
  • BYTELYST_COMMIT_MESSAGE: Commit message (truncated)
  • BYTELYST_DOCKER_IMAGE: Docker image tag

Internal DevOps Dashboard

Dashboard Product

  • Location: dashboard/
  • Purpose: Internal deployment orchestration and service monitoring
  • Architecture: Full ByteLyst product (backend + web)
  • Backend: Fastify 5 (port 4004)
  • Web: Next.js 16 (port 3000)
  • Integration: Uses @bytelyst/* packages, links to admin-web

Dashboard Setup

See dashboard/README.md for setup and usage instructions.

Safety and Operational Guidelines

Sensitive Files

  • accounts.json - GitHub account credentials
  • .env files - Environment configuration
  • Generated contributor/output data - Operational snapshots
  • Never echo these files unless task requires it

Destructive Operations

  • Many scripts are operational and potentially destructive
  • Preserve prompts, dry-run behavior, and confirmations
  • Review scripts before automation use
  • Some encode assumptions about ByteLyst org structure

Pre-commit Validation

pre-commit run --all-files

Quick Reference for Common Tasks

Task Command
Deploy invttrdg ./deploy-invttrdg.sh
Force deployment ./deploy-invttrdg.sh --force
Deploy all products ./deploy-all.sh
Check repo status ./git-work-safety-tools/git_repos_status.sh
List user repos ./bytelyst-cli.sh list-public-repos --user <user>
Remove user interactively ./remove_user_interactive.sh
Run pre-commit checks pre-commit run --all-files

Related Repositories

  • learning_ai_common_plat: Shared platform packages consumed by products
  • Product repos: learning_ai_invt_trdg, learning_ai_notes, etc. (deployed via scripts here)

First Steps for Any Deployment Task

  1. Check deployment scripts first - look for deploy-{project}.sh
  2. Read the script - understand its flow and requirements
  3. Check GITEA_NPM_TOKEN - ensure it's set correctly
  4. Verify companion repos - ensure learning_ai_common_plat is accessible
  5. Run the deployment script - use appropriate flags (--force, --skip-health-check)
  6. Never build custom deployment unless script doesn't exist for your use case

New Deployment Pattern for @bytelyst/* Packages (Learned 2026-05)

Background

Products that consume @bytelyst/* packages from learning_ai_common_plat require a special build preparation step because:

  • Gitea registry is not accessible from Docker build context
  • Direct npm install would fail during Docker build
  • Dependencies need to be pre-packaged as tarballs

Required Pre-Build Step

Products using @bytelyst/ packages MUST have a scripts/docker-prep.sh script:*

# Before building Docker images
bash scripts/docker-prep.sh

# After building Docker images
bash scripts/docker-prep.sh --restore

Dockerfile Changes Required

Dockerfiles MUST be updated to remove Gitea secret mounting:

Before (incorrect):

RUN --mount=type=secret,id=gitea_npm_token \
  TOKEN=$(cat /run/secrets/gitea_npm_token) && \
  echo "//localhost:3300/api/packages/bytelyst/npm/:_authToken=$TOKEN" >> .npmrc && \
  pnpm install --ignore-scripts --lockfile=false

After (correct):

COPY .docker-deps/ ../.docker-deps/
RUN pnpm install --ignore-scripts --lockfile=false

Environment Variable Configuration

NEXT_PUBLIC_ variables MUST use public API URLs:

Incorrect:

NEXT_PUBLIC_BACKEND_URL: http://backend:4011

Correct:

NEXT_PUBLIC_BACKEND_URL: https://api.bytelyst.com/{product}

Docker Compose Format

Use YAML mapping syntax, not array syntax:

Incorrect:

environment:
  - NODE_ENV=production
  - PORT:4011

Correct:

environment:
  NODE_ENV: production
  PORT: 4011

Container Naming

Container names MUST match Caddyfile configuration:

services:
  backend:
    container_name: {product}-backend
  web:
    container_name: {product}-web

DNS Setup

Add DNS records via GoDaddy API for new subdomains:

curl -X PUT "https://api.godaddy.com/v1/domains/bytelyst.com/records/A/{subdomain}" \
  -H "Authorization: sso-key $GODADDY_API_KEY:$GODADDY_API_SECRET" \
  -H "Content-Type: application/json" \
  -d '[{"data": "187.124.159.82", "name": "{subdomain}", "ttl": 600, "type": "A"}]'

Caddy Configuration

Add subdomain routing to /opt/bytelyst/Caddyfile:

{subdomain}.bytelyst.com {
  encode gzip
  reverse_proxy {product}-web:{port}
}

Products Requiring This Pattern

  • learning_ai_notes (NoteLett)
  • learning_ai_clock (ChronoMind)
  • learning_ai_invttrdg (Investment Trading)
  • Any product consuming @bytelyst/* packages

Deployment Script Updates

Deployment scripts should be updated to:

  1. Check for docker-prep.sh existence
  2. Run docker-prep.sh before Docker build
  3. Run docker-prep.sh --restore after successful deployment
  4. Provide option to skip docker-prep if already run

Health Check Requirements

Backend Dockerfiles MUST include wget for health checks:

RUN apk add --no-cache wget

Health check in docker-compose.yml:

healthcheck:
  test: ["CMD", "wget", "--spider", "-q", "http://localhost:{port}/health"]
  interval: 30s
  timeout: 5s
  retries: 3