# Hostinger VM — GitHub Actions Self-Hosted Runner Setup

> **Delegation prompt for the Codex agent running on the Hostinger VM.**
> Read top-to-bottom before executing. Stop and ask the human if any pre-flight check fails or any deliverable is unclear.

---

## 1. Goal

Set up a GitHub Actions self-hosted runner on the Hostinger VM that can:

1. Receive workflow triggers from **all 20+ `@bytelyst/*` repos** (see §8 for the org-vs-repo registration decision).
2. Build `@bytelyst/*` npm packages from tagged releases.
3. Publish them to **the local Gitea instance on this VM** (`http://localhost:3300/api/packages/bytelyst/npm/`).
4. Upload the same tarballs as **GitHub Release assets** so a corp-network Mac can sync them into its own local Gitea (via the separate `bytelyst-sync` script described in a follow-up prompt).

Self-hosted on Hostinger beats GitHub-hosted runners because:

- No GitHub Actions minute cap.
- Gitea is on `localhost` from this VM → zero-latency publish, no public TLS needed.
- VM is always on; runner is reachable indefinitely.

### Multi-repo registration decision

**Decide before Step 2** whether this runner serves a single repo or all 20+:

| Approach                             | Registration scope        | Use when                                                                                                       |
| ------------------------------------ | ------------------------- | -------------------------------------------------------------------------------------------------------------- |
| **Repo-level** (default in this doc) | Single repo URL           | You're validating the runner first, or you only have 1–2 repos publishing packages                             |
| **Org-level** (recommended at scale) | A GitHub Organization URL | You've migrated 20+ repos under one org — see §8 for migration steps. One registration, all org repos eligible |

The install steps are identical — only the `--url` flag in `config.sh` (Step 3) changes.

---

## 2. Pre-flight checks (run first, do not skip)

```bash
# 1. Confirm Linux VM
hostname && uname -a       # Expected: Linux

# 2. Confirm Gitea is running locally
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:3300/   # Expected: 200 or 302

# 3. Confirm Gitea npm registry endpoint reachable
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:3300/api/packages/bytelyst/npm/    # Expected: 200 or 401

# 4. Confirm Node 20 and pnpm 9 installable
node --version 2>/dev/null || echo "Node not installed (will install in Step 5)"
pnpm --version 2>/dev/null || echo "pnpm not installed (will install in Step 5)"

# 5. Confirm gh CLI exists
gh --version 2>/dev/null || echo "gh CLI not installed (will install in Step 5)"

# 6a. Detect architecture (used for runner tarball selection in Step 3)
ARCH=$(uname -m)
case "$ARCH" in
  x86_64)  export RUNNER_ARCH="linux-x64";;
  aarch64) export RUNNER_ARCH="linux-arm64";;
  *) echo "Unsupported arch: $ARCH — STOP and report"; ;;
esac
echo "Will install runner for: $RUNNER_ARCH"

# 6b. Disk free
df -h /     # Need ~5 GB headroom

# 7. Confirm no existing runner
ls -la ~/actions-runner 2>/dev/null && echo "Runner dir exists — STOP and confirm with human" || echo "No existing runner"

# 8. Confirm github.com reachable (both API and download CDN)
curl -s -o /dev/null -w "api.github.com: %{http_code}\n" https://api.github.com/
curl -s -o /dev/null -w "objects.githubusercontent.com: %{http_code}\n" -L \
  https://objects.githubusercontent.com/   # runner tarball download host
# Expected: 200 / 403 (403 from CDN root is normal; what matters is non-network-error)

# 9. Confirm the Gitea token file exists somewhere on this VM
sudo find /home /root -maxdepth 3 -name ".gitea_npm_token" 2>/dev/null | head -5
# Expected: at least one path. Note the owning user — needed in Step 6.

# 10. Confirm gh CLI is auth'd as saravanakumardb1 (needed for registration token)
gh auth status 2>&1 | grep -E "Logged in to|saravanakumardb1" | head -5
# If not logged in as saravanakumardb1, run: gh auth login (and pick saravanakumardb1)
```

If any check fails or surprises you, **stop and report back** before proceeding.

---

## 3. What you'll create

| Item                       | Path/Identifier                                                                        |
| -------------------------- | -------------------------------------------------------------------------------------- |
| Dedicated Linux user       | `gha-runner`                                                                           |
| Runner installation        | `/home/gha-runner/actions-runner/`                                                     |
| systemd service            | `actions.runner.saravanakumardb1-learning_ai_common_plat.hostinger-bytelyst-1.service` |
| Runner labels              | `self-hosted, linux, x64, hostinger, bytelyst`                                         |
| Gitea publish token (copy) | `/home/gha-runner/.gitea_npm_token` (mode 600)                                         |
| Smoke-test workflow        | `.github/workflows/runner-smoke.yml` (commit on a branch)                              |
| E2E test workflow          | `.github/workflows/runner-e2e-publish.yml` (commit on a branch)                        |

---

## 4. Installation

### Step 1 — Create the dedicated runner user (idempotent)

```bash
if ! getent passwd gha-runner >/dev/null; then
  sudo useradd -m -s /bin/bash gha-runner
  echo "Created gha-runner user"
else
  echo "gha-runner user already exists — skipping useradd"
fi

# Add to docker group only if docker is on this host
if getent group docker >/dev/null; then
  sudo usermod -aG docker gha-runner
  echo "Added gha-runner to docker group"
fi

id gha-runner
```

### Step 2 — Get a GitHub runner registration token (one-time, ~1h TTL)

**Preferred (via `gh` CLI auth'd as `saravanakumardb1`):**

```bash
gh api -X POST /repos/saravanakumardb1/learning_ai_common_plat/actions/runners/registration-token --jq .token
```

**Alternative (browser):** Open `https://github.com/saravanakumardb1/learning_ai_common_plat/settings/actions/runners/new?arch=x64&os=linux` and copy the token from the `./config.sh` command shown.

Hold the token in shell memory only:

```bash
read -s RUNNER_TOKEN   # paste, press Enter (no echo)
```

### Step 3 — Download, verify, configure the runner

```bash
# Query the latest runner version (don't hardcode)
LATEST=$(gh api /repos/actions/runner/releases/latest --jq '.tag_name' | sed 's/^v//')
echo "Latest runner version: $LATEST"
# As of writing this doc: 2.319.1. If LATEST is wildly different, STOP and confirm with human.

sudo -iu gha-runner bash <<EOF
mkdir -p ~/actions-runner && cd ~/actions-runner
RUNNER_VERSION="$LATEST"
RUNNER_ARCH="${RUNNER_ARCH}"   # from pre-flight 5a
TARBALL="actions-runner-\${RUNNER_ARCH}-\${RUNNER_VERSION}.tar.gz"

# Download tarball
curl -fSL -o "\$TARBALL" \
  "https://github.com/actions/runner/releases/download/v\${RUNNER_VERSION}/\$TARBALL"

# Download checksum manifest and verify (GitHub publishes SHA256 alongside each release)
EXPECTED_SHA=\$(gh api /repos/actions/runner/releases/tags/v\${RUNNER_VERSION} \
  --jq ".body" | grep -oE "\b[0-9a-f]{64}\s+\$TARBALL\b" | awk '{print \$1}')
ACTUAL_SHA=\$(sha256sum "\$TARBALL" | awk '{print \$1}')

if [ "\$EXPECTED_SHA" != "\$ACTUAL_SHA" ]; then
  echo "FAIL: SHA mismatch"
  echo "  Expected: \$EXPECTED_SHA"
  echo "  Actual:   \$ACTUAL_SHA"
  exit 1
fi
echo "PASS: tarball SHA verified"

tar xzf "./\$TARBALL"
EOF
```

Note: if `gh api` parsing of the SHA from the release body fails (GitHub sometimes changes release-note formatting), fall back to the official hashes page:
`https://github.com/actions/runner/releases/tag/v<version>`. If you can't verify the SHA, STOP and report — don't run unverified binaries.

Register the runner. **Choose the URL based on your scope decision (§1):**

```bash
# OPTION A — repo-level (default during validation)
REGISTRATION_URL="https://github.com/saravanakumardb1/learning_ai_common_plat"

# OPTION B — org-level (once you've migrated to an org per §8)
# REGISTRATION_URL="https://github.com/<your-org-name>"

sudo -u gha-runner -E -i bash -c "
cd ~/actions-runner && \
./config.sh \
  --url $REGISTRATION_URL \
  --token $RUNNER_TOKEN \
  --name hostinger-bytelyst-1 \
  --labels self-hosted,linux,x64,hostinger,bytelyst \
  --work _work \
  --replace \
  --unattended
"

unset RUNNER_TOKEN
```

Verify in GitHub UI: runner should show "Idle" green under `Settings → Actions → Runners`.

### Step 4 — Install as a systemd service

```bash
sudo bash -c "
cd /home/gha-runner/actions-runner && \
./svc.sh install gha-runner && \
./svc.sh start
"

SVC_NAME='actions.runner.saravanakumardb1-learning_ai_common_plat.hostinger-bytelyst-1.service'
sudo systemctl status "$SVC_NAME"
sudo journalctl -u "$SVC_NAME" -n 30 --no-pager
# Expected: "active (running)" + "Listening for Jobs"
```

### Step 5 — Install Node 20, pnpm 9, gh CLI system-wide

```bash
# Node 20 via Nodesource
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo bash -
sudo apt-get install -y nodejs

# gh CLI (if not already)
sudo apt-get install -y gh   # or follow https://cli.github.com/manual/installation if 'gh' isn't in the apt repos

# pnpm
sudo npm install -g pnpm@9

# Verify all reachable from the runner user
sudo -u gha-runner bash -c 'node --version && pnpm --version && gh --version'
```

### Step 6 — Give the runner access to the local Gitea publish token

```bash
# Identify the source token file (from pre-flight check #9)
SRC_TOKEN=/home/<original-user>/.gitea_npm_token

# Copy and lock down
sudo cp "$SRC_TOKEN" /home/gha-runner/.gitea_npm_token
sudo chown gha-runner:gha-runner /home/gha-runner/.gitea_npm_token
sudo chmod 600 /home/gha-runner/.gitea_npm_token

# Verify
sudo -u gha-runner bash -c 'wc -c < ~/.gitea_npm_token && stat -c "%a %U:%G" ~/.gitea_npm_token'
# Expected: nonzero byte count, mode 600, owner gha-runner:gha-runner
```

---

## 5. Smoke test (basic — runner picks up jobs)

Create branch `runner/smoke` in `learning_ai_common_plat` with the workflow below.

```bash
cd ~/code/mygh/learning_ai_common_plat
git checkout -b runner/smoke
mkdir -p .github/workflows
# (paste workflow below into .github/workflows/runner-smoke.yml)
git add .github/workflows/runner-smoke.yml
git commit -m "ci: add self-hosted runner smoke-test workflow"
git push origin runner/smoke
```

```yaml
# .github/workflows/runner-smoke.yml
name: Runner Smoke Test
on:
  workflow_dispatch:
  push:
    branches: [runner/smoke]
jobs:
  smoke:
    runs-on: [self-hosted, linux, hostinger]
    steps:
      - run: echo "host=$(hostname) user=$(whoami) cwd=$(pwd)"
      - run: node --version && pnpm --version && gh --version
      - run: |
          echo "Gitea health:"
          curl -s -o /dev/null -w "  http://localhost:3300/  → %{http_code}\n" http://localhost:3300/
          curl -s -o /dev/null -w "  /api/packages/bytelyst/npm/ → %{http_code}\n" \
            http://localhost:3300/api/packages/bytelyst/npm/
      - run: |
          if [ -f ~/.gitea_npm_token ]; then
            echo "Gitea token present, $(wc -c < ~/.gitea_npm_token) bytes, mode $(stat -c %a ~/.gitea_npm_token)"
          else
            echo "ERROR: Gitea token missing"
            exit 1
          fi
```

Trigger from GitHub UI (Actions tab → Runner Smoke Test → Run workflow on `runner/smoke`). All steps must pass.

---

## 6. End-to-end validation (CRITICAL — proves the actual use case)

The smoke test only proves the runner can execute. The **E2E test** proves the **whole publish pipeline** works: package builds, publishes to local Gitea, uploads to GitHub Releases, and is installable on a different machine.

### E2E test workflow

Create a **temporary test package** in `learning_ai_common_plat` (will be removed after validation):

```bash
# On Hostinger or from human's machine — does NOT need to run on the runner
mkdir -p packages/_runner-e2e-test
cat > packages/_runner-e2e-test/package.json <<'EOF'
{
  "name": "@bytelyst/_runner-e2e-test",
  "version": "0.0.1",
  "description": "Throwaway package for E2E validating the Hostinger runner. Safe to delete after validation.",
  "main": "index.js",
  "files": ["index.js"]
}
EOF
echo "module.exports = { ok: true, builtAt: new Date().toISOString() };" \
  > packages/_runner-e2e-test/index.js
git checkout -b runner/e2e
git add packages/_runner-e2e-test/
git commit -m "test: add throwaway package for runner E2E validation"
git push origin runner/e2e
```

Create the E2E workflow on the same branch:

```yaml
# .github/workflows/runner-e2e-publish.yml
name: Runner E2E — publish + release

on:
  workflow_dispatch:
    inputs:
      version:
        description: 'Version to publish (semver)'
        required: true
        default: '0.0.1-e2e.1'

jobs:
  publish:
    runs-on: [self-hosted, linux, hostinger]
    permissions:
      contents: write # for GitHub Release creation
    steps:
      - uses: actions/checkout@v4

      - name: Set test version
        working-directory: packages/_runner-e2e-test
        run: |
          npm version "${{ inputs.version }}" --no-git-tag-version --allow-same-version
          cat package.json

      - name: Configure pnpm registry for Gitea
        working-directory: packages/_runner-e2e-test
        run: |
          cat > .npmrc <<NPMRC
          @bytelyst:registry=http://localhost:3300/api/packages/bytelyst/npm/
          //localhost:3300/api/packages/bytelyst/npm/:_authToken=$(cat ~/.gitea_npm_token)
          NPMRC
          echo ".npmrc written:"
          sed 's|_authToken=.*|_authToken=***|' .npmrc

      - name: Publish to local Gitea
        working-directory: packages/_runner-e2e-test
        run: pnpm publish --no-git-checks --registry http://localhost:3300/api/packages/bytelyst/npm/

      - name: Pack tarball for GitHub Release
        working-directory: packages/_runner-e2e-test
        run: |
          pnpm pack --pack-destination /tmp
          ls -la /tmp/bytelyst-_runner-e2e-test-*.tgz

      - name: Create GitHub Release with tarball
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          TAG="e2e-runner-${{ inputs.version }}"
          gh release create "$TAG" \
            /tmp/bytelyst-_runner-e2e-test-*.tgz \
            --title "Runner E2E test $TAG" \
            --notes "Throwaway release from runner E2E validation. Safe to delete." \
            --prerelease

      - name: Verify package queryable from Gitea
        run: |
          AUTH_HEADER="Authorization: Bearer $(cat ~/.gitea_npm_token)"
          curl -s -H "$AUTH_HEADER" \
            "http://localhost:3300/api/packages/bytelyst/npm/@bytelyst%2F_runner-e2e-test" \
            | head -200
          echo ""
          # Assert the version we just published is in the response
          curl -s -H "$AUTH_HEADER" \
            "http://localhost:3300/api/packages/bytelyst/npm/@bytelyst%2F_runner-e2e-test" \
            | grep -q '"${{ inputs.version }}"' || { echo "FAIL: version not found in Gitea registry"; exit 1; }
          echo "PASS: version ${{ inputs.version }} is in Gitea registry"

      - name: Verify pnpm install works from a clean directory
        run: |
          mkdir -p /tmp/runner-e2e-consumer && cd /tmp/runner-e2e-consumer
          cat > .npmrc <<NPMRC
          @bytelyst:registry=http://localhost:3300/api/packages/bytelyst/npm/
          //localhost:3300/api/packages/bytelyst/npm/:_authToken=$(cat ~/.gitea_npm_token)
          NPMRC
          cat > package.json <<JSON
          { "name": "runner-e2e-consumer", "version": "1.0.0", "dependencies": { "@bytelyst/_runner-e2e-test": "${{ inputs.version }}" } }
          JSON
          pnpm install --no-frozen-lockfile
          node -e "const m = require('@bytelyst/_runner-e2e-test'); console.log('Module loaded:', m); process.exit(m.ok ? 0 : 1);"
          echo "PASS: pnpm install + require works end-to-end"

      - name: Verify GitHub Release tarball matches Gitea tarball (byte-identical)
        run: |
          # The released tarball should be byte-identical to what we pushed to Gitea
          RELEASED=$(ls /tmp/bytelyst-_runner-e2e-test-*.tgz)
          GITEA_URL=$(curl -s -H "Authorization: Bearer $(cat ~/.gitea_npm_token)" \
            "http://localhost:3300/api/packages/bytelyst/npm/@bytelyst%2F_runner-e2e-test" \
            | grep -oE '"tarball":"[^"]*"' | head -1 | cut -d'"' -f4)
          curl -s -H "Authorization: Bearer $(cat ~/.gitea_npm_token)" -o /tmp/from-gitea.tgz "$GITEA_URL"
          SHA_RELEASED=$(sha256sum "$RELEASED" | awk '{print $1}')
          SHA_GITEA=$(sha256sum /tmp/from-gitea.tgz | awk '{print $1}')
          echo "Released:  $SHA_RELEASED"
          echo "Gitea:     $SHA_GITEA"
          [ "$SHA_RELEASED" = "$SHA_GITEA" ] || { echo "FAIL: tarball mismatch — bytes-identical guarantee broken"; exit 1; }
          echo "PASS: tarball is byte-identical between Gitea and GitHub Release"
```

### Run the E2E

```bash
# From any machine that has gh auth'd as saravanakumardb1:
gh workflow run runner-e2e-publish.yml --ref runner/e2e -f version=0.0.1-e2e.1 --repo saravanakumardb1/learning_ai_common_plat
gh run watch --repo saravanakumardb1/learning_ai_common_plat
```

### E2E pass criteria — ALL must succeed

| Check                                     | Step that proves it                                                                          |
| ----------------------------------------- | -------------------------------------------------------------------------------------------- |
| Runner picks up the job                   | "Set up job" log in GH Actions UI shows `Runner name: hostinger-bytelyst-1`                  |
| Package publishes to Gitea                | `pnpm publish` step exits 0                                                                  |
| GitHub Release created with tarball asset | `gh release view e2e-runner-0.0.1-e2e.1 --json assets` returns 1 asset                       |
| Gitea reports the version                 | `Verify package queryable from Gitea` step says `PASS: version ... is in Gitea registry`     |
| Consumer install works                    | `Verify pnpm install works` step says `PASS: pnpm install + require works end-to-end`        |
| Release tarball ≡ Gitea tarball (sha256)  | `Verify GitHub Release tarball matches Gitea tarball` says `PASS: tarball is byte-identical` |

The last check is the **key invariant** for the corp-Mac sync flow: same bytes → same SHA512 integrity hash → lockfiles portable across both Gitea instances.

### Cleanup after E2E

```bash
# Delete the test release
gh release delete e2e-runner-0.0.1-e2e.1 --yes --repo saravanakumardb1/learning_ai_common_plat

# Delete the test package from local Gitea (via API or UI under Packages → @bytelyst/_runner-e2e-test → Delete)

# Delete the test branch and package
git checkout main
git push origin --delete runner/e2e
rm -rf packages/_runner-e2e-test
git checkout main
git commit -am "test: remove runner E2E throwaway package" || true
git push origin main
```

Leave both workflow files (`runner-smoke.yml`, `runner-e2e-publish.yml`) **on `main`** — they're idempotent and provide a way to re-validate the runner anytime (just bump the version input).

---

## 7. Hardening (do before relying for prod)

### a. Workflow approval for external PRs

`Settings → Actions → General → Fork pull request workflows` → "Require approval for all outside collaborators". Repo is private so risk is contained, but keep approval-required on.

### b. systemd resource limits

```bash
sudo systemctl edit "$SVC_NAME"
```

```ini
[Service]
CPUQuota=200%
MemoryMax=4G
TasksMax=2048
```

```bash
sudo systemctl daemon-reload
sudo systemctl restart "$SVC_NAME"
```

### c. Log rotation

```bash
sudo tee /etc/logrotate.d/gha-runner > /dev/null <<'EOF'
/home/gha-runner/actions-runner/_diag/*.log {
    weekly
    rotate 4
    compress
    missingok
    notifempty
    copytruncate
}
EOF
```

### d. Update mechanism

GitHub auto-updates the runner agent unless disabled. Keep auto-update on. Pin a minimum version in workflows if you need a specific feature:

```yaml
runs-on: [self-hosted, linux, hostinger]
```

### e. Auth scope of the runner

The runner's `GITHUB_TOKEN` (provided by GitHub Actions automatically) is scoped per-workflow. Verify in `runner-e2e-publish.yml` that `permissions:` is set narrowly (we set `contents: write` only for release creation).

---

## 8. Scaling to all 20+ repos — GitHub Organization migration

A single self-hosted runner can serve all 20+ `@bytelyst/*` repos **only if** registered at **GitHub Organization level**. The personal-account path (repo-level registration) doesn't scale beyond 1–3 repos.

### Why migrate to an org

| Concern                          | Personal account today                     | Under an org                                       |
| -------------------------------- | ------------------------------------------ | -------------------------------------------------- |
| Self-hosted runner reuse         | One registration per repo                  | One registration covers all org repos              |
| Secrets management               | Per-repo (duplicated)                      | Org-level secrets inherited by all repos           |
| Visibility                       | Per-repo Actions tabs (no cross-repo view) | Org-level Actions dashboard across all repos       |
| Permissions / team collaboration | Limited                                    | Teams, code owners, etc.                           |
| Cost                             | Free                                       | Free for unlimited public + private repos          |
| Move cost                        | —                                          | ~1–2 hours total for 20 repos (mostly automatable) |

### Migration steps (do these BEFORE Step 3 if going org-level from day 1)

```bash
# 1. Create the org via the GitHub UI:
#    https://github.com/organizations/plan
#    Choose "Free" plan. Suggested name: bytelyst-platform (or whatever fits).

# 2. Transfer each repo to the org (one-time, preserves all history + issues + stars)
for repo in learning_ai_common_plat learning_ai_clock learning_ai_notes \
            learning_ai_flowmonk learning_ai_trails learning_ai_jarvis_jr \
            learning_ai_fastgap learning_ai_peakpulse learning_ai_efforise \
            learning_ai_auth_app learning_voice_ai_agent learning_multimodal_memory_agents \
            learning_ai_local_memory_gpt learning_ai_local_llms learning_ai_talk2obsidian \
            learning_ai_mac_tooling learning_ai_productivity_web learning_ai_smart_auth; do
  echo "Transferring $repo..."
  gh api -X POST "/repos/saravanakumardb1/$repo/transfer" -f new_owner="<your-org-name>"
done

# 3. Update your local clones to point to the new owner
#    (run on each machine, in each repo dir)
cd ~/code/mygh/<repo>
git remote set-url origin https://github.com/<your-org-name>/<repo>.git
```

GitHub automatically sets up redirects from the old URLs, so external links won't break immediately — but you should update CI references, README badges, and any inter-repo URL references.

### After migration

- Get a runner registration token at the **org level**:
  ```bash
  gh api -X POST /orgs/<your-org-name>/actions/runners/registration-token --jq .token
  ```
- Use the org URL in Step 3's `config.sh` (Option B above).
- The runner now picks up jobs from any repo in the org that targets `runs-on: [self-hosted, hostinger, bytelyst]`.

### Workflow propagation across 20+ repos

Once the runner is org-level, the next problem is propagating the `publish-packages.yml` workflow file to every repo that publishes packages. Two strategies:

1. **Reusable workflow** (preferred) — define `publish-packages.yml` once as a `workflow_call` reusable workflow in `learning_ai_common_plat/.github/workflows/`, then each consuming repo has a tiny stub that calls it.
2. **Per-repo copy maintained by a sync script** — follow the same pattern as the existing `sync-npmrc.sh` in `scripts/`. Less elegant but works fine for a small repo count.

Deliver as a separate follow-up prompt.

---

## 9. Monitoring + observability — how to track this runner

The GitHub Actions tab tracks runner state at three levels:

### a. Per-repo (or per-org) workflow runs

`https://github.com/<owner>/<repo>/actions` (or `/orgs/<org>/actions/` after migration) shows every workflow run with live-streaming logs. The "Set up job" step always logs:

```
Runner name: 'hostinger-bytelyst-1'
Runner group name: 'Default'
Machine name: 'hostinger-vm'
```

This is how you confirm the right runner picked up the job.

### b. Runner pool health

- **Repo level:** `Settings → Actions → Runners`
- **Org level:** `Org settings → Actions → Runners`

Shows: status (`Idle` / `Active` / `Offline`), labels, OS, last connection time. This is where you debug "is my runner alive?".

### c. Scripted monitoring via `gh` CLI

```bash
# Watch a specific run live
gh run watch --repo <owner>/<repo>

# List recent runs
gh run list --repo <owner>/<repo> --limit 10

# View finished run with full logs
gh run view <run-id> --log --repo <owner>/<repo>

# List runners + their status (admin scope required)
gh api /repos/<owner>/<repo>/actions/runners \
  --jq '.runners[] | {name, status, busy, labels: [.labels[].name]}'

# Or at org level:
gh api /orgs/<org>/actions/runners \
  --jq '.runners[] | {name, status, busy}'
```

### d. Host-side observability (on the Hostinger VM)

```bash
SVC_NAME='actions.runner.saravanakumardb1-learning_ai_common_plat.hostinger-bytelyst-1.service'

# Live tail
sudo journalctl -u "$SVC_NAME" -f

# Last 100 lines
sudo journalctl -u "$SVC_NAME" -n 100 --no-pager

# Per-run diagnostic logs
ls -la /home/gha-runner/actions-runner/_diag/

# Current systemd state
sudo systemctl status "$SVC_NAME"
```

Use host-side logs when the runner shows "Offline" in GitHub UI but the VM is reachable — typically a daemon crash, expired registration, or network blip.

---

## 10. Deliverables — report back to the human

When complete:

1. **Service status:**
   ```bash
   sudo systemctl is-active "$SVC_NAME"
   ```
2. **GitHub UI confirmation:** screenshot or text "Runner shows 'Idle' green at github.com/saravanakumardb1/learning_ai_common_plat/settings/actions/runners".
3. **Smoke test run URL** — workflow passed.
4. **E2E test run URL** — workflow passed, all 6 pass criteria green.
5. **Installed versions:**
   ```bash
   sudo -u gha-runner bash -c 'node --version; pnpm --version; gh --version; docker --version 2>/dev/null || echo "no docker"'
   ```
6. **Log paths:**
   - systemd: `journalctl -u <service-name>`
   - runner diag: `/home/gha-runner/actions-runner/_diag/`
7. **Confirmation that cleanup happened:** test release deleted, test package removed from Gitea, throwaway package deleted from repo.

---

## 11. Guardrails

- **Do not** run the runner as root.
- **Do not** persist the GitHub registration token to disk — memory only.
- **Do not** install under the `gitea` user or any other service user — keep concerns separated.
- **Do not** open inbound ports on the VM firewall — the runner is outbound-only long-poll.
- **Do not** skip the E2E test. The smoke test alone does not prove the publish pipeline works.
- **Do not** mark E2E as passed unless all 6 pass criteria succeed, especially the byte-identical tarball check.
- **Do not** leave the throwaway `@bytelyst/_runner-e2e-test` package in the Gitea registry — it pollutes the namespace.

---

## 12. Rollback

```bash
SVC_NAME='actions.runner.saravanakumardb1-learning_ai_common_plat.hostinger-bytelyst-1.service'

# Stop and uninstall systemd service
sudo bash -c "cd /home/gha-runner/actions-runner && ./svc.sh stop && ./svc.sh uninstall"

# Unregister from GitHub (need a fresh removal token)
REMOVAL_TOKEN=$(gh api -X POST /repos/saravanakumardb1/learning_ai_common_plat/actions/runners/remove-token --jq .token)
sudo -u gha-runner bash -c "cd ~/actions-runner && ./config.sh remove --token $REMOVAL_TOKEN"

# Remove the user and all its files
sudo userdel -r gha-runner
```

---

## 13. Follow-up prompts (separate tasks)

Once this runner is verified end-to-end, the next prompts to issue:

1. **`publish-packages.yml`** — the real production workflow in `learning_ai_common_plat`, modeled on the E2E template above, that triggers on `v*` tags and publishes all changed `@bytelyst/*` packages.
2. **`bytelyst-sync` script** — runs on the corp Mac; downloads GitHub Release tarballs and republishes to the corp local Gitea. Verifies sha256 against Gitea before considering sync successful.
3. **SKILL doc** at `AI.dev/SKILLS/gitea-package-sync.md` — describes the full three-system flow for future contributors.

---

## 14. Questions to ask the human BEFORE starting if anything is ambiguous

- "Are we registering this runner at **repo level** (one repo only) or **org level** (after migrating 20+ repos to a GitHub org)? See §1 and §8."
- "If org-level: what is the org name? Has the migration in §8 already happened?"
- "If repo-level: which repo am I registering for? (default: `saravanakumardb1/learning_ai_common_plat`)"
- "Is Docker required on the runner — i.e., does any planned workflow run `docker` commands? (default: no, only Gitea uses Docker)"
- "What user currently owns `~/.gitea_npm_token` on this VM? (pre-flight check #9 will tell us)"
- "Do you have a runner registration token, or should I fetch one via `gh api`?"
- "Are you OK with me creating a throwaway `@bytelyst/_runner-e2e-test` package, publishing it, and then deleting it as part of E2E validation?"

If any of these are unclear, stop and ask before installing anything.