# Operations Guide

Common operational paths for the team.

Use this file as the routing guide. For the exact support boundary, cross-check `docs/supported-scripts.md`.

For app/dashboard bookmarks and deployment URL references, use
[`docs/app-url-bookmarks.md`](app-url-bookmarks.md). Keep that file updated
whenever a new app URL, dashboard URL, API route, or last deploy timestamp
changes.

---

## Hostinger VM Maintenance

See [`docs/hostinger-vm-maintenance.md`](hostinger-vm-maintenance.md) for:
- The May 2026 incident post-mortem (load avg 1305, root cause, full fix log)
- Ongoing disk/memory/load health targets
- Safe cleanup reference commands
- How to deploy `admin-web` when ready

Quick scripts:

```bash
# Health check (read-only, any time)
bash scripts/VMs/HostingerVM/vm-health-check.sh

# Weekly safe cleanup
bash scripts/VMs/HostingerVM/vm-cleanup.sh

# Monthly full cleanup
bash scripts/VMs/HostingerVM/vm-cleanup.sh --full

# Install automated cron schedule
bash scripts/VMs/HostingerVM/vm-cleanup.sh --install-cron
```

Cron setup details: [`scripts/VMs/HostingerVM/CRON_SETUP.md`](../scripts/VMs/HostingerVM/CRON_SETUP.md)

---

## Gitea Registry Backup

The local Gitea npm registry is backed up with a native `gitea dump` job.

Installed VM paths:

- Script: `/opt/bytelyst/scripts/backup-gitea.sh`
- Backup directory: `/opt/bytelyst/backups/gitea`
- Systemd service: `bytelyst-gitea-backup.service`
- Systemd timer: `bytelyst-gitea-backup.timer`

Versioned source files:

- [`scripts/gitea-backup.sh`](../scripts/gitea-backup.sh)
- [`systemd/bytelyst-gitea-backup.service`](../systemd/bytelyst-gitea-backup.service)
- [`systemd/bytelyst-gitea-backup.timer`](../systemd/bytelyst-gitea-backup.timer)

Useful commands:

```bash
# Run a backup immediately
sudo systemctl start bytelyst-gitea-backup.service

# Check last run and next scheduled run
sudo systemctl status bytelyst-gitea-backup.service --no-pager
systemctl list-timers --all --no-pager bytelyst-gitea-backup.timer

# List retained backup dumps
ls -lh /opt/bytelyst/backups/gitea
```

The timer runs daily at `03:15 UTC` and the script deletes dumps older than 14 days by default.

---

## 1. Remove A Collaborator Interactively

Use:

```bash
./remove_user_interactive.sh
```

Use this when:

- you need repository pattern matching
- you want per-repository confirmation
- you want the safest default path for access removal

Requirements:

- `curl`
- `jq`
- GitHub token with the necessary repo and org permissions

## 2. Remove A Collaborator In A More Guided Flow

Use:

```bash
./remove_user_guided.sh
```

Use this when:

- you want a more opinionated step-by-step flow
- you are onboarding someone to the repo tools

## 3. Remove A Collaborator In A Scripted Way

Use:

```bash
./remove_user_from_repos.sh
```

Use this when:

- you already know the target root, user, and repo pattern
- you want a repeatable command invocation

Read first:

- `README_remove_user_script.md`
- `docs/supported-scripts.md`

## 4. Run Common GitHub Admin Commands

Use:

```bash
./bytelyst-cli.sh help
```

Typical commands:

```bash
./bytelyst-cli.sh list-public-repos --user <username>
./bytelyst-cli.sh list-private-repos --org <orgname>
./bytelyst-cli.sh check-collaborators --input github_acc_input.json
```

Use this when:

- you need the main shared CLI entry point
- the task fits the commands already exposed there

## 5. Scan Multi-Repo Git Status

Use:

```bash
./git-work-safety-tools/git_repos_status.sh
./git-work-safety-tools/multi_repo_status.sh
```

Use this when:

- you need visibility across many local repositories
- you want safer batch git operations

## 6. Safely Push Across Multiple Repositories

Use:

```bash
./git-work-safety-tools/multi_repo_safe_push.sh
```

Use this when:

- you are coordinating many repos
- you want a safer batch push helper instead of ad hoc shell loops

## 7. Scan GitHub Access Across Accounts

Use:

```bash
cp accounts.example.json accounts.json
./github_access_scripts/list_user_repos.sh <github_user>
./github_access_scripts/check_repo_access.sh <github_user> <owner/repo>
```

Use this when:

- you need access verification
- you need a focused check outside the main CLI

## 8. Generate Repo Scan Artifacts Locally

Use:

```bash
./github_repo_scanners/create_user_repo_lists.sh
./github_repo_scanners/run_contributor_json_creation.sh accounts.json
```

Generated outputs go to:

- `github_repo_scanners/user_repos/`
- `github_repo_scanners/contributor_repos/`

Do not commit those outputs unless they are intentionally curated fixtures.

## 9. Harden An Ubuntu VM

Use:

```bash
sudo bash scripts/ubuntu-vm-security-update.sh --help
sudo bash scripts/ubuntu-vm-security-update.sh --dry-run
```

Use this when:

- you are preparing or refreshing an Ubuntu VM
- you want unattended upgrades, UFW, and fail2ban configured

Recommended order:

1. Read `scripts/README.md`
2. Run `--help`
3. Run `--dry-run`
4. Run the real command with explicit reboot behavior

After a reboot triggered by the script, verify the resulting host state with:

```bash
test -f /var/run/reboot-required && echo "reboot still required" || echo "reboot clear"
sudo systemctl status unattended-upgrades fail2ban
sudo ufw status verbose
sudo tail -n 50 /var/log/vm-security-update.log
apt list --upgradable
```

If packages are still pending or the services are unhealthy, rerun:

```bash
sudo bash scripts/ubuntu-vm-security-update.sh
```

## 10. Operate Hermes Agent Safely

Use:

```bash
hermes --version
hermes status --all
hermes cron list
systemctl status hermes-gateway --no-pager
```

Read first:

- `docs/hermes-setup-upgrade-roadmap.md`
- `docs/hermes-operations.md`

Use this when:

- you are upgrading or troubleshooting Hermes
- you are checking Telegram gateway health
- you are verifying backup/watchdog cron jobs
- you are evaluating any private-only dashboard/API access pattern

Hard rule: do **not** expose a Hermes dashboard or Hermes API publicly unless S explicitly approves the exact hostname, auth gate, and access path.

## Team Guidance

- Prefer the supported entry points in `docs/tooling-status.md`.
- Prefer `docs/operations.md` and `docs/supported-scripts.md` over scanning the repo root manually.
- Prefer templates such as `accounts.example.json` and `.env.example` over ad hoc local files.
- Do not commit local credentials, generated transcripts, or repo-scan outputs.
