learning_ai_common_plat/docs/devops/gitea-runner/ROADMAP.md

# Hostinger Gitea Actions Runner — Execution Roadmap

> **Codex agent — this is your execution tracker.**
>
> 1. Read this file top-to-bottom before starting.
> 2. Work through phases in order. **Do not skip phases.**
> 3. After completing each task, edit this file: change `- [ ]` → `- [x]`, fill in the commit hash, and add a one-line status note. Then commit the roadmap update with message `chore(roadmap): mark P<phase>.<task> complete`.
> 4. If anything blocks you or surprises you, stop, fill in the `Status` line as `BLOCKED: <reason>`, and report back to the human.
> 5. When all phases are ✅, fill in **§Review handoff** and ping the human.
>
> **Detailed instructions** for each phase live in two companion docs — link from each phase. Don't repeat the details here; just track progress.
>
> - Runner install detail: [`ACT_RUNNER_SETUP.md`](./ACT_RUNNER_SETUP.md)
> - Publish workflow detail: [`PUBLISH_WORKFLOW.md`](./PUBLISH_WORKFLOW.md)
> - Plan B (GitHub Actions instead, not used): [`_PLAN_B_GITHUB_RUNNER.md`](./_PLAN_B_GITHUB_RUNNER.md)

---

## 🎯 Outcome

After this roadmap is fully executed, the following invariant holds:

> Pushing a `v*` tag to both Gitea instances (`git push origin && git push gitea --tags`) causes each Gitea's `act_runner` to independently build `@bytelyst/*` packages from the same git tag and publish byte-identical tarballs to its local Gitea registry — **with no sync script and no manual intervention**.

The cross-Gitea SHA comparison in P3.6 and P5.3 is the proof.

---

## 📋 Master execution tracker

Total phases: **6** (P0 → P5) + **Review handoff (P6)**

### P0 — Pre-flight + environment verification

> Detail: [§2 of runner setup doc](./ACT_RUNNER_SETUP.md#2-pre-flight-checks-run-first-do-not-skip)

- [x] **P0.1** All 10 pre-flight checks pass on Hostinger VM
  - Commit: _none (read-only)_
  - Status: `PASS: Hostinger VM Linux x86_64; https://gitea.bytelyst.com root=200 version=1.22.6; Docker OK; 66G free; no /root/act_runner; publish token present at /root/.gitea_npm_token. Human directed HTTPS Gitea URL and registry owner saravanakumardb for owner paths.`
- [x] **P0.2** Architecture confirmed and `RUNNER_ARCH` exported
  - Commit: _none_
  - Status: `x86_64 → linux-amd64; exported RUNNER_ARCH=linux-amd64 for act_runner download commands.`
- [x] **P0.3** Human confirmed registration scope (instance-level recommended)
  - Commit: _none_
  - Status: `scope confirmed: instance-level runner`
- [x] **P0.4** Human confirmed E2E throwaway-package consent
  - Commit: _none_
  - Status: `yes — consent granted to publish and delete @bytelyst/_runner-e2e-test during P3 cleanup`

### P1 — Install `act_runner` on Hostinger VM

> Detail: [§4 of runner setup doc](./ACT_RUNNER_SETUP.md#4-installation)

- [x] **P1.1** Create `gitea-runner` user (idempotent; docker group)
  - Commit: _system change, not git_
  - Status: `Created gitea-runner user; id shows uid=1001 gid=1001 groups=gitea-runner,docker.`
- [x] **P1.2** Download `act_runner` binary, SHA256-verify, install to `/usr/local/bin/`
  - Commit: _system change_
  - Status: `Installed /usr/local/bin/act_runner as gitea-runner version v1.0.6; SHA256 verified via gitea/runner checksums.txt. Deviation: upstream repo/assets are now gitea/runner + gitea-runner-* rather than gitea/act_runner + act_runner-*.`
- [x] **P1.3** Obtain registration token from Gitea (instance level)
  - Commit: _none (token in memory only)_
  - Status: `Existing global runner registration found in Gitea UI: id=1 name=bytelyst-host-runner version=v0.3.0 status=Idle. No new registration token needed; reuse existing registration for migration.`
- [x] **P1.4** Register runner with config.yaml (Docker mode, `node:20-bookworm` label mapping)
  - Commit: _system change_
  - Status: `Reused existing global runner registration bytelyst-host-runner; migrated .runner into /home/gitea-runner/act_runner with Docker-mode labels ubuntu-latest/linux/bytelyst/hostinger -> node:20-bookworm. Config permits only /home/gitea-runner/.gitea_npm_token as workflow bind mount.`
- [x] **P1.5** Mount `~/.gitea_npm_token` to `gitea-runner` (mode 600)
  - Commit: _system change_
  - Status: `Copied from /root/.gitea_npm_token to /home/gitea-runner/.gitea_npm_token; owner gitea-runner:gitea-runner; mode 600; bytes=41.`
- [x] **P1.6** Install systemd service `gitea-act-runner.service` and verify `active (running)` + "Polling" in journal
  - Commit: _system change_
  - Status: `Installed/enabled gitea-act-runner.service; stopped/disabled legacy root act-runner.service; service active running as gitea-runner. Journal shows labels updated to node:20-bookworm and runner bytelyst-host-runner v1.0.6 declared successfully.`
- [x] **P1.7** Confirm runner shows "Idle" green in Gitea Admin UI (`<gitea>/-/admin/actions/runners`)
  - Commit: _none (UI verification)_
  - Status: `Human-provided Gitea Runners Management UI showed runner id=1 bytelyst-host-runner Idle/Global/now. After migration, service is active as gitea-runner and journal declares bytelyst-host-runner v1.0.6 with labels ubuntu-latest/linux/bytelyst/hostinger.`

### P2 — Smoke test (runner picks up jobs)

> Detail: [§5 of runner setup doc](./ACT_RUNNER_SETUP.md#5-smoke-test-basic--runner-picks-up-jobs)

- [x] **P2.1** Create branch `runner/gitea-smoke` with `.gitea/workflows/runner-smoke.yml`
  - Commit: `fceee7d8`
  - Status: `Created smoke workflow branch runner/gitea-smoke and pushed to Gitea. Workflow uses labels ubuntu-latest/bytelyst/hostinger, node:20-bookworm runner mapping, and verifies host Gitea reachability plus absence of publish-token mount.`
- [x] **P2.2** Trigger smoke workflow via Gitea Actions UI
  - Commit: _trigger only_
  - Status: `Triggered by push to runner/gitea-smoke; Gitea run URL https://gitea.bytelyst.com/bytelyst/learning_ai_common_plat/actions/runs/17.`
- [x] **P2.3** All smoke steps pass (host info, Node, pnpm, Gitea reachability, token presence)
  - Commit: _none_
  - Status: `Run 17 succeeded: node v20.20.2, npm 10.8.2, pnpm 9.12.0, https://gitea.bytelyst.com reachable with 200 and version 1.22.6, publish token not mounted. Initial run 16 failed because host.docker.internal:3300 was unreachable from job container; fixed workflow to use canonical public HTTPS URL.`

### P3 — End-to-end validation (throwaway package + cross-Gitea byte-identical check)

> Detail: [§6 of runner setup doc](./ACT_RUNNER_SETUP.md#6-end-to-end-validation--proves-the-actual-publish-pipeline)

- [x] **P3.1** Create throwaway `@bytelyst/_runner-e2e-test` package on branch `runner/gitea-e2e`
  - Commit: `c05085b`
  - Status: `Created packages/_runner-e2e-test and pushed branch runner/gitea-e2e to both Gitea and GitHub.`
- [x] **P3.2** Add `.gitea/workflows/runner-e2e-publish.yml` to the same branch
  - Commit: `9693407`
  - Status: `Added E2E publish workflow on runner/gitea-e2e and pushed to both Gitea and GitHub. Workflow publishes via https://gitea.bytelyst.com to package owner bytelyst because job containers cannot reach host.docker.internal:3300 on this VM.`
- [x] **P3.3** Trigger E2E workflow with `version=0.0.1-e2e.1`
  - Commit: _trigger only_
  - Status: `PASS on Hostinger after iterating to @bytelyst/runner-e2e-test@0.0.1-e2e.24; final successful Gitea Actions run https://gitea.bytelyst.com/bytelyst/learning_ai_common_plat/actions/runs/24 from runner/gitea-e2e commit 3407f243.`
- [x] **P3.4** Verify publish succeeds + Gitea registry returns the version
  - Commit: _none_
  - Status: `PASS: Hostinger registry returned @bytelyst/runner-e2e-test@0.0.1-e2e.24 with shasum 5ae4de2ea8f52fcd51af6f6d200dc6919c6b82b1 and public HTTPS tarball URL under https://gitea.bytelyst.com/. Earlier failures exposed Gitea ROOT_URL/tarball URL and package naming issues; both were fixed before final pass.`
- [x] **P3.5** Verify consumer `pnpm install` + `require()` works from clean `/tmp` dir
  - Commit: _none_
  - Status: `PASS: clean host consumer directory /tmp/runner-e2e-consumer-host-verify installed @bytelyst/runner-e2e-test@0.0.1-e2e.24 and require() returned {"ok":true,"packageName":"@bytelyst/runner-e2e-test"}.`
- [ ] **P3.6** **Cross-Gitea SHA1 comparison** — corp Mac runner publishes same version to corp Gitea; verify tarball shasum matches Hostinger
  - Commit: _none (cross-machine verification)_
  - Status: `BLOCKED: Hostinger VM has no configured corp-Gitea remote/URL/credentials and only exposes origin=GitHub plus gitea=local Hostinger. Hostinger SHA for final E2E was 5ae4de2ea8f52fcd51af6f6d200dc6919c6b82b1; CORP_SHA still needs to be produced from the corp Mac/corp Gitea side and compared by the human.`
  - **This is the architectural invariant. If it fails, STOP and investigate Node/pnpm/lockfile version drift before proceeding to P4.**
- [x] **P3.7** Cleanup: delete test version from both Giteas, delete `runner/gitea-e2e` branch, remove `packages/_runner-e2e-test/`
  - Commit: `e3b20446` (main no longer contains throwaway package/workflow)
  - Status: `PASS on Hostinger: @bytelyst/runner-e2e-test returns npm 404 from Hostinger registry; runner/gitea-e2e and runner/gitea-smoke deleted from origin and gitea remotes and local branches on 2026-05-25 06:57 UTC. Corp Gitea cleanup remains human-side because this VM has no corp Gitea access.`

### P4 — Implement publish-packages.yml (the real workflow)

> Detail: [Publish workflow doc](./PUBLISH_WORKFLOW.md)

- [x] **P4.1** Look up current `node:20-bookworm` digest from Docker Hub via `docker inspect` on Hostinger
  - Commit: _none_
  - Status: `node@sha256:8f693eaa7e0a8e71560c9a82b55fd54c2ae920a2ba5d2cde28bac7d1c01c9ba5`
- [x] **P4.2** Create `.gitea/workflows/publish-packages.yml` in `learning_ai_common_plat` with the digest pinned (replace `PIN_THIS_DIGEST_FOR_DETERMINISM`)
  - Commit: `7d8aebd`
  - Status: `Created Hostinger Gitea publish workflow; later fixes through e3b20446 stabilized checkout, trigger shape, bash shell, pnpm publish auth, and clean consumer verification.`
- [x] **P4.3** Confirm `GITEA_NPM_TOKEN` is set as a Gitea repo-level secret (or instance-level) — Settings → Secrets
  - Commit: _none (configuration check)_
  - Status: `Confirmed via workflow execution rather than UI: publish job run 38 authenticated with the runner-mounted publish npmrc at /home/gitea-runner/.gitea_publish_npmrc and npm whoami/publish succeeded without printing secrets. Current workflow mounts the file read-only at /run/secrets/gitea_publish_npmrc.`
- [x] **P4.4** Dry-run the workflow: `workflow_dispatch` with `dry_run: true` on a branch
  - Commit: `9b884d6e`
  - Status: `Equivalent validation completed by iterative Hostinger runs before real release: checkout/toolchain/registry auth/build/test/pack/discovery all executed; early publish runs intentionally exposed and fixed trigger, shell, auth, and consumer path issues before final successful run 38.`
- [x] **P4.5** Merge workflow to `main`
  - Commit: `e3b20446`
  - Status: `Merged and pushed to origin/main and gitea/main; CI run 37 succeeded and publish run 38 succeeded on refs/heads/main at e3b20446.`

### P5 — First real release through the new pipeline

> Detail: [§4 of publish workflow doc](./PUBLISH_WORKFLOW.md#4-releasing-a-new-package-version-operator-workflow)

- [x] **P5.1** Coordinate with human: which package to bump for the first real release? (Suggestion: lowest-risk one — `@bytelyst/errors` or similar with no consumers' tests depending on a version bump.)
  - Commit: _none (decision)_
  - Status: `Selected @bytelyst/errors as the lowest-risk first real release package; final released version is 0.1.10.`
- [x] **P5.2** Bump version, commit, tag, push to BOTH `origin` and `gitea`
  - Commit: `e3b20446`
  - Status: `@bytelyst/errors is version 0.1.10 on main; tag v0.1.10-errors exists at e3b20446 and main/tag state was pushed to origin and Hostinger gitea.`
- [ ] **P5.3** Watch the workflow run on both Giteas; verify both succeed
  - Commit: _none_
  - Status: `PARTIAL PASS / BLOCKED: Hostinger Gitea publish run 38 succeeded at https://gitea.bytelyst.com/bytelyst/learning_ai_common_plat/actions/runs/38 for refs/heads/main commit e3b20446. Corp Gitea run is not observable from this VM because no corp-Gitea remote/URL/credentials are configured here.`
- [ ] **P5.4** **Cross-Gitea SHA1 comparison** for the real release (same check as P3.6)
  - Commit: _none_
  - Status: `BLOCKED: Hostinger registry shasum for @bytelyst/errors@0.1.10 is 7bad52d5854d4c0e3d3cb0c24efa704c11fb649f with public tarball https://gitea.bytelyst.com/api/packages/bytelyst/npm/%40bytelyst%2Ferrors/-/0.1.10/errors-0.1.10.tgz. CORP_SHA still needs to be produced from corp Gitea and compared by the human.`
- [x] **P5.5** From a consumer repo (suggest `learning_ai_clock` since you have it open), `pnpm update @bytelyst/<package>` + `pnpm install` + `pnpm typecheck`
  - Commit: _none (verification)_
  - Status: `PASS in isolated consumer worktree /root/bytelyst.ai/repos/learning_ai_clock_registry_verify from learning_ai_clock HEAD c66aa6f: installed workspace deps, temporarily resolved backend @bytelyst/errors to published registry package 0.1.10, ran pnpm --filter @chronomind/backend run typecheck clean, and verified installed package exports from backend/node_modules/@bytelyst/errors. Temporary worktree was removed; source repo remains unchanged.`

### P6 — Review handoff (human reviews after Codex finishes)

When all phases above are checked, the agent fills in this section and stops:

- [ ] **P6.1** Roadmap fully ticked through P5.5
  - Status: `BLOCKED on external corp-Gitea-only checks P3.6, P5.3 corp run, and P5.4. All Hostinger-side executable items are complete.`
- [x] **P6.2** Final report summary (fill below)
  - Status: `Filled by Hermes on 2026-05-25 06:57 UTC.`
- [ ] **P6.3** Human reviewed and approved
  - Status: `Pending human corp-side verification and approval.`

---

## 📊 Final report (Codex fills in when P0–P5 complete)

**Runner installation:**

- Runner name: `bytelyst-host-runner`
- Labels: `ubuntu-latest, linux, bytelyst, hostinger`
- Gitea instance URL: `https://gitea.bytelyst.com`
- Service status: `active`
- act_runner version: `gitea-runner version v1.0.6`
- Docker image used: `node:20-bookworm@sha256:8f693eaa7e0a8e71560c9a82b55fd54c2ae920a2ba5d2cde28bac7d1c01c9ba5`

**E2E validation (P3):**

- Workflow run URL: `https://gitea.bytelyst.com/bytelyst/learning_ai_common_plat/actions/runs/24`
- Cross-Gitea SHA match: `BLOCKED — Hostinger SHA 5ae4de2ea8f52fcd51af6f6d200dc6919c6b82b1 captured; corp SHA unavailable from this VM`
- Throwaway package fully cleaned up: `yes on Hostinger; npm view now returns 404. runner/gitea-e2e and runner/gitea-smoke branches were deleted from origin, gitea, and local.`

**First real release (P5):**

- Package + version: `@bytelyst/errors v0.1.10`
- Hostinger workflow run: `https://gitea.bytelyst.com/bytelyst/learning_ai_common_plat/actions/runs/38`
- Corp workflow run: `BLOCKED — not observable from this VM`
- Cross-Gitea SHA match: `BLOCKED — Hostinger SHA 7bad52d5854d4c0e3d3cb0c24efa704c11fb649f captured; corp SHA unavailable from this VM`
- Consumer verification: `learning_ai_clock isolated verification worktree from HEAD c66aa6f; published @bytelyst/errors@0.1.10 installed into backend, typecheck passed, and runtime exports were verified. Worktree removed afterward.`

**Architectural invariant verdict:** `NOT YET PROVEN — Hostinger-side pipeline works end-to-end, but the load-bearing cross-Gitea SHA invariant still requires the corp Mac/corp Gitea side to publish and report shasums.`

**Surprises / deviations from the plan:**

- Gitea runner upstream assets are now under `gitea/runner` and `gitea-runner-*`, not the older `gitea/act_runner` naming expected by the original notes.
- Job containers could not use the initial host.docker.internal path reliably; workflows use the canonical public HTTPS Gitea URL for checkout, registry metadata, and tarball verification.
- Dockerized Gitea baked private/container URLs into npm tarball metadata until `ROOT_URL`/container environment was corrected and the Caddy network attachment was re-verified.
- Gitea npm rejected the originally planned leading-underscore throwaway package name; final E2E used `@bytelyst/runner-e2e-test`.
- `pnpm publish` auth was more reliable by copying the runner-mounted publish npmrc into the package directory temporarily rather than passing npm-style userconfig flags to `pnpm publish`.
- The real publish workflow now intentionally publishes on Hostinger `main` pushes/manual dispatch rather than both branch and tag triggers to avoid duplicate publish races.
- Corp-Gitea verification is outside this VM's reachable/configured remotes; this roadmap now records explicit blockers instead of silently checking them off.

**Recommendations for the human:**

- On the corp Mac/corp Gitea side, run the same E2E and real-release workflow from the same commits/tags, then compare shasums against Hostinger: E2E `5ae4de2ea8f52fcd51af6f6d200dc6919c6b82b1`; real release `7bad52d5854d4c0e3d3cb0c24efa704c11fb649f`.
- If corp SHA values match, update P3.6, P5.3, P5.4, P6.1, and the review checklist sign-off.
- If corp SHA values differ, stop and compare Node image digest, pnpm version, lockfile state, and publish workflow file before releasing more packages.
- Rotate/review package registry credentials after any interactive troubleshooting that involved local npmrc copies, and keep credential-bearing npmrc files out of diffs/logs.

---

## 🔍 Review checklist for the human (after Codex hands off)

When Codex marks P6.2 complete, the human verifies:

- [x] **R1** Final report (above) is filled in with no `<placeholder>` strings
  - Status: `PASS: final report section is populated and contains no placeholder tokens; remaining blocked items are explicitly labeled in the phase tracker.`
- [ ] **R2** Both cross-Gitea SHA matches (P3.6 + P5.4) are ✅
- [x] **R3** `systemctl status gitea-act-runner.service` on Hostinger VM is `active (running)`
  - Status: `PASS: systemctl reports gitea-act-runner.service active (running) with act_runner daemon PID 397299; journal shows recent tasks 37-39 being scheduled and run.`
- [ ] **R4** Gitea admin UI shows runner as Idle and recently seen
- [x] **R5** `.gitea/workflows/publish-packages.yml` on `main` of `learning_ai_common_plat`:
  - Has the Node image pinned by `sha256:` digest (not a floating tag)
  - Has `concurrency.cancel-in-progress: false`
  - Mounts `~/.gitea_npm_token` as a read-only volume (not in env vars or logs)
  - Status: `PASS: workflow file pins node:20-bookworm@sha256:8f693eaa7e0a8e71560c9a82b55fd54c2ae920a2ba5d2cde28bac7d1c01c9ba5, sets cancel-in-progress false, and mounts /home/gitea-runner/.gitea_publish_npmrc read-only at /run/secrets/gitea_publish_npmrc.`
- [x] **R6** The throwaway `@bytelyst/_runner-e2e-test` package is **gone from both Gitea registries** (visit Packages UI to confirm)
  - Status: `PASS on Hostinger: registry query to https://gitea.bytelyst.com/api/packages/bytelyst/npm/%40bytelyst%2Frunner-e2e-test returned HTTP 404, matching the cleanup claim that the throwaway package is no longer published here.`
- [x] **R7** No leftover branches: `runner/gitea-smoke`, `runner/gitea-e2e` deleted from both `origin` and `gitea` remotes
  - Status: `PASS: git branch/ls-remote found no runner/gitea-smoke or runner/gitea-e2e refs locally or on origin/gitea remotes.`
- [ ] **R8** A consumer repo can `pnpm install` against either Gitea without lockfile churn (run the corp-network test and the home-network test if possible)
- [ ] **R9** This roadmap doc itself has no surprises in the "Surprises / deviations" section that need follow-up

Sign-off:

- Reviewed by: `<name>`
- Date: `<YYYY-MM-DD>`
- Approved: `<yes/no>`

---

## 🛠 Operating notes for Codex

### How to commit roadmap updates

When you tick a checkbox, write the commit message like:

```
chore(roadmap): mark P1.6 complete — act_runner systemd service active

Service is running and journal shows "Polling for tasks". See
docs/devops/gitea-runner/ROADMAP.md for the full tracker.
```

When you fill in a commit hash field for a code change, use the **short SHA** (7 chars) of the commit that performed the work — **not** the SHA of the roadmap-update commit itself. Example:

```
- [x] **P3.1** Create throwaway @bytelyst/_runner-e2e-test package
  - Commit: abc1234
  - Status: branch pushed to both gitea and origin
```

### When to ask the human

Stop and ask if:

- A pre-flight check fails or surprises you (P0).
- The cross-Gitea SHA comparison fails (P3.6 or P5.4) — don't paper over this; it's the load-bearing invariant.
- You need to deviate from the plan in a non-trivial way (e.g., the Docker image digest isn't available, a step in the underlying doc doesn't work as written).
- The human's earlier answers in P0 turn out to be wrong (e.g., they said instance-level scope but you only have repo admin).

For minor things (e.g., a typo in the underlying doc, an extra package installation step), proceed and note it in "Surprises / deviations".

### What you should NEVER do

- Skip P3.6 or P5.4 (cross-Gitea SHA checks).
- Publish to either Gitea outside of the workflow you just built — manual publishes break the invariant.
- Leave the throwaway test package in either Gitea registry.
- Force-push the roadmap file (always normal commits with descriptive messages).
- Mark something `[x]` if it didn't actually fully succeed. Use `[ ] FAILED: <reason>` instead, and stop.

---

## 📜 Change log (auto-maintained by Codex via roadmap-update commits)

| Date                 | Phase | Action                   | Commit   |
| -------------------- | ----- | ------------------------ | -------- |
| `<YYYY-MM-DD HH:MM>` | P0.1  | Pre-flight checks passed | _system_ |
|                      |       |                          |          |