diff --git a/AI.dev/CHEATSHEETS/README.md b/AI.dev/CHEATSHEETS/README.md index 537d5395..a26ac871 100644 --- a/AI.dev/CHEATSHEETS/README.md +++ b/AI.dev/CHEATSHEETS/README.md @@ -12,15 +12,17 @@ ## What's here -Quick, dense reference cards for the three terminal AI agents we use to delegate -coding work. Each sheet is **task-oriented** โ€” commands, modes, session management, -config, and the ByteLyst-specific guardrails โ€” not a marketing overview. +Quick, dense reference cards for the terminal AI agents we use to delegate coding work, +plus an operational guide for running them **non-stop overnight**. Each sheet is +**task-oriented** โ€” commands, modes, session management, config, and the ByteLyst-specific +guardrails โ€” not a marketing overview. | CLI | Cheat sheet | Best for | | ------------------ | -------------------------------------------- | ------------------------------------------------------------------------------- | | ๐Ÿค– **Devin** | [`devin-cli.md`](./devin-cli.md) | Long-running autonomous sessions; delegate a scoped roadmap and walk away | | ๐ŸŸฃ **Claude Code** | [`claude-code-cli.md`](./claude-code-cli.md) | Interactive pair-programming in the terminal; deep multi-file edits with review | | ๐ŸŸข **Codex CLI** | [`codex-cli.md`](./codex-cli.md) | Fast local edits + scriptable `exec` runs in CI / one-shot automation | +| ๐ŸŒ™ **Long-running jobs** | [`long-running-jobs.md`](./long-running-jobs.md) | Running ANY of the above non-stop, unattended overnight (sleep/disconnect survival + best practices) | --- @@ -95,4 +97,4 @@ and the per-repo `AGENTS.md`. Put them in the agent's opening prompt every time: 3. Add it to the table above. 4. Commit: `docs(cheatsheets): update CLI cheat sheet`. -_Last updated: 2026-05-28_ +_Last updated: 2026-05-30_ diff --git a/AI.dev/CHEATSHEETS/long-running-jobs.md b/AI.dev/CHEATSHEETS/long-running-jobs.md new file mode 100644 index 00000000..8f81587e --- /dev/null +++ b/AI.dev/CHEATSHEETS/long-running-jobs.md @@ -0,0 +1,171 @@ +# ๐ŸŒ™ Overnight / Long-Running Agent Runs โ€” Cheat Sheet + +> **What it is:** how to run a multi-hour, unattended AI coding job (Devin / Claude Code / +> Codex / Copilot CLI) **non-stop** without it dying when the machine sleeps or the +> terminal closes โ€” plus the ByteLyst best practices that make an unattended run *safe*. +> **Best for:** delegating a scoped roadmap (e.g. a 10-hour Phase build) and walking away +> overnight. +> **Companion docs:** the per-CLI sheets ([`devin-cli.md`](./devin-cli.md), +> [`claude-code-cli.md`](./claude-code-cli.md), [`codex-cli.md`](./codex-cli.md)) and the +> canonical [`../SKILLS/agent-behavior-guidelines.md`](../SKILLS/agent-behavior-guidelines.md). + +> ๐Ÿง  **The whole problem in one line:** an unattended run has two enemies โ€” the **machine +> sleeping** and the **terminal/session dying**. `caffeinate` kills the first, `tmux` the +> second, and a **checkpoint file** saves you from everything else (reboot, crash, power loss). + +--- + +## The two failure modes (and what fixes each) + +| Failure mode | What happens | Fix | +| ------------ | ------------ | --- | +| **Machine sleeps** (idle timeout / lid closed) | CPU + network suspend โ†’ job freezes mid-step, connections drop | **`caffeinate`** (macOS) / `systemd-inhibit` (Linux) โ€” keep it plugged in, lid open | +| **Terminal/session dies** (window closed, app crash/update, SSH drop, logout) | Job receives `SIGHUP` and is killed | **`tmux`** / `screen` โ€” the job is owned by the tmux server, not the window | +| **Reboot / crash / power loss** | Everything dies, including tmux | **Checkpoint + resume** โ€” the job writes progress to a file; re-running resumes it | + +`caffeinate` and `tmux` are **complementary**, not alternatives โ€” use both for a real overnight run. + +## `caffeinate` (macOS) โ€” stop the Mac from sleeping + +```bash +caffeinate -dimsu # stays awake only while runs +``` + +| Flag | Prevents | +| ---- | -------- | +| `-d` | **display** sleep | +| `-i` | **idle** sleep | +| `-m` | **disk** sleep | +| `-s` | system sleep (on AC power) | +| `-u` | declares **user active** | + +> โš ๏ธ **Lid caveat:** `caffeinate` keeps the Mac awake, but **closing the lid still sleeps +> most Macs** unless on AC power with an external display (clamshell) or the right pmset. +> For overnight: **keep it plugged in and the lid open.** + +## `tmux` โ€” survive a closed terminal / disconnect + +```bash +tmux new -s phase3 # start a named, detachable session +# ... launch your job inside it ... +# Ctrl-b then d # DETACH โ€” job keeps running with no terminal attached +tmux attach -t phase3 # reattach later (after reconnect, or a new window) +tmux ls # list running sessions +``` + +- **Local Mac:** still useful โ€” the job survives **closing the terminal app, an app + crash/auto-update, or logout** (a plain terminal job dies on all of these). It does + **nothing** for sleep โ€” that's `caffeinate`'s job. +- **Over SSH:** essential โ€” the job survives the SSH connection dropping. + +**Linux equivalents:** `systemd-inhibit --what=idle:sleep ` (sleep), `screen` or +`tmux` (session), or `nohup &` + `disown` (cheapest detach, no reattach). + +## Putting it together + +```bash +tmux new -s phase3 +# inside the session: +caffeinate -dimsu "" 2>&1 | tee ~/phase3.log +# Ctrl-b d to detach and walk away; tmux attach -t phase3 in the morning +``` + +`| tee ~/phase3.log` captures the full run to disk so you have the log even if the +scrollback is gone. + +--- + +## Best practices for an unattended agent run + +1. **Green baseline first.** Run the verify gate (`pnpm build && pnpm test`, browsers + installed) **before** launching, so the agent starts from a known-clean state โ€” never + debug a pre-existing red suite at 2am. +2. **Self-contained roadmap, slice-by-slice.** Point the agent at a doc that encodes + scope, per-slice verify commands, and a done-definition (see [`../PROMPTS/`](../PROMPTS/)). + One slice = one commit + push. +3. **Checkpoint + resume.** The job must write progress to a file (e.g. + `docs/-progress.md`: slice, status, commit SHA, gate result). If it dies for any + reason, **re-running the same prompt resumes** from the first not-DONE slice. This is + your safety net for the failures `caffeinate`/`tmux` can't catch. +4. **Failure protocol, not thrash.** Tell it: max N honest attempts per slice, then commit + `wip(...) BLOCKED:` + mark FAILED + move to the next **independent** slice. Order slices + so the independent ones come first. +5. **Never merge unattended.** Push to a feature branch and open **one PR** โ€” a human + reviews + merges in the morning. The agent must **never touch `main`**. +6. **Tests are sacred.** Never weaken/skip a test to go green โ€” fix the code (see the + canonical guidelines). State this explicitly in the prompt. +7. **Scope + push-auth ready.** Cache git credentials / `gh auth login` first so the + overnight `push` never blocks on a prompt. Confirm push rights to the target remote. +8. **All-allowed flag.** Launch with the CLI's auto-approve flag so it never pauses for + permission (`--permission-mode dangerous` for Devin, `--dangerously-skip-permissions` + for Claude Code, `--dangerously-bypass-approvals-and-sandbox` for Codex โ€” confirm with + ` --help`). + +## ByteLyst-specific gotchas (the ones that silently fail overnight) + +- **Corp proxy.** `pnpm install` and Playwright browser downloads will **TLS-fail** behind + the intercepting proxy unless `NODE_EXTRA_CA_CERTS` (proxy CA) is set. **Prefer running + off-corp** for overnight jobs โ€” simplest reliable path. (A proxy-blocked `pnpm install` + is the classic cause of an overnight job that "did nothing".) +- **Playwright e2e.** Slices with browser tests need `npx playwright install --with-deps` + **before** the run, or the e2e gate fails. +- **Workspace, not registry.** `@bytelyst/*` link via `workspace:*` from sibling + `packages/*` โ€” run from the monorepo so they resolve locally; no registry needed for the + build itself. +- **Side-by-side repos.** If the prompt file lives in a sibling repo (e.g. the gigafactory + job lives in `learning_ai_devops_tools` but runs in `learning_ai_common_plat`), clone + **both under the same parent** so the `../` relative path resolves. + +--- + +## Worked example โ€” the gigafactory Phase 3 overnight run + +```bash +# 1) bootstrap a clean baseline (once) +cd ~/code/mygh/learning_ai_common_plat && git pull && corepack enable \ + && pnpm install && pnpm build && pnpm test \ + && (cd dashboards/tracker-web && npx playwright install --with-deps) +cd ~/code/mygh/learning_ai_devops_tools && git pull # prompt + roadmap repo + +# 2) launch non-stop +cd ~/code/mygh/learning_ai_common_plat +tmux new -s phase3 +caffeinate -dimsu \ + "Read ~/code/mygh/learning_ai_devops_tools/agent-queue/docs/jobs/phase3-overnight.md and execute it exactly: branch feat/gigafactory-phase3 off origin/main, implement Phase 3 slice-by-slice, verify each slice green before the next, checkpoint to docs/gigafactory-phase3-progress.md, commit+push per slice, open ONE PR and NEVER merge, never weaken tests (use the 3-attempt failure protocol), end with the consolidated report." \ + 2>&1 | tee ~/phase3.log +# Ctrl-b d to detach +``` + +In the morning: `tmux attach -t phase3`, read the consolidated report, then review + +merge the PR slice-by-slice. + +## Troubleshooting + +| Symptom | Likely cause | Fix | +| ------- | ------------ | --- | +| Job froze partway, no error | Mac slept (idle or **lid closed**) | Use `caffeinate -dimsu`; plug in; lid open | +| Job vanished when I closed Terminal | No tmux โ€” `SIGHUP` killed it | Run inside `tmux`; reattach with `tmux attach` | +| "Did nothing" / 0 commits overnight | `pnpm install` TLS-failed on corp proxy | Run **off-corp**, or set `NODE_EXTRA_CA_CERTS` | +| e2e slice failed immediately | Playwright browsers not installed | `npx playwright install --with-deps` before launch | +| Push at end blocked / hung | git/gh auth not cached | `gh auth login` / cache credentials first | +| Re-ran prompt, it redid finished work | No checkpoint file read | Ensure the job reads `*-progress.md` and resumes | + +## Quick-reference card + +```text +caffeinate -dimsu # macOS: stay awake while runs (lid open + on AC!) +systemd-inhibit --what=idle:sleep # Linux equivalent +tmux new -s # detachable session (Ctrl-b d = detach) +tmux attach -t # reattach (tmux ls = list) +... | tee ~/run.log # capture full output to disk +# launch: tmux -> caffeinate -dimsu "" | tee log +# safety net: checkpoint file + re-run prompt to resume; push to branch, never merge +``` + +--- + +**Related:** [`devin-cli.md`](./devin-cli.md) ยท [`claude-code-cli.md`](./claude-code-cli.md) ยท +[`codex-cli.md`](./codex-cli.md) ยท [`../PROMPTS/`](../PROMPTS/) ยท +[`../SKILLS/agent-behavior-guidelines.md`](../SKILLS/agent-behavior-guidelines.md) + +_Last updated: 2026-05-30 ยท confirm CLI auto-approve flags against your installed version (` --help`)._