# ByteLyst Open Notebook private pilot evaluation report

Date: 2026-06-28
Task: t_b57eb21e
Deployment: `/opt/bytelyst/learning_ai_devops_tools/deployments/open-notebook-pilot/docker-compose.yml`
Runtime: `/opt/bytelyst/open-notebook-pilot`

## Verdict

Private pilot deployment is operational and hardened for localhost-only evaluation. It is not public and should remain excluded from public Caddy/subdomain routing.

Pilot readiness: 86/100

Main remaining gap: full LLM `/api/search/ask/simple` Q&A timed out with the tiny local `qwen2.5:0.5b` model during first smoke testing. Retrieval, ingestion, embeddings, backup, and restore-readiness passed; source-grounded answer generation needs a larger/faster local model or a sanitized cloud comparison model before relying on generated answers.

## Hardening checks

| Check | Result | Evidence |
|---|---:|---|
| UI localhost-only | PASS | Docker published `127.0.0.1:8502->8502/tcp` only |
| API localhost-only | PASS | Docker published `127.0.0.1:5055->5055/tcp` only |
| SurrealDB public exposure | PASS | No SurrealDB host port published; internal Docker network only |
| Ollama public exposure | PASS | Ollama exposes container port only; no host published port |
| Strong secrets outside git | PASS | `/etc/bytelyst/open-notebook-pilot.env`, mode `0600`, contains generated app/encryption/SurrealDB credentials |
| CORS explicit | PASS | `CORS_ORIGINS=http://127.0.0.1:8502,http://localhost:8502` in protected env |
| Public dashboard exposure | PASS | No Caddy route added; access path is localhost/SSH tunnel only |

## Local model/provider checks

| Item | Result |
|---|---|
| Ollama service | Running in private compose network |
| LLM model pulled | `qwen2.5:0.5b` |
| Embedding model pulled | `nomic-embed-text` |
| Open Notebook model registry | Created `ollama` language and embedding models |
| Default models | Chat/transformation/tools/large-context set to `qwen2.5:0.5b`; embeddings set to `nomic-embed-text` |
| Vector search | PASS: vector smoke query returned `01-access-policy-embedded.md` in 4.764s |
| Generated ask/Q&A | NEEDS FOLLOW-UP: `/api/search/ask/simple` timed out at 300s with `qwen2.5:0.5b` |

## Sanitized docs ingested

Source directory: `/opt/bytelyst/open-notebook-pilot/test-docs`

- `01-access-policy.md`
- `02-backup-policy.md`
- `03-knowledge-scope.md`
- `01-access-policy-embedded.md` (extra embedded source for vector smoke)

Notebook created by smoke script: `notebook:2iyp5pykncuhwul2qc9g`

## Known-answer evaluation

Raw results: `/opt/bytelyst/open-notebook-pilot/reports/known-answer-results.json`

| Question | Retrieval result | Latency | Pass |
|---|---|---:|---:|
| Should the pilot be exposed as a public dashboard? | Top source `01-access-policy-embedded.md` | 0.262s | PASS |
| Which host ports should UI and API use? | Top source `01-access-policy-embedded.md` | 0.064s | PASS |
| Should SurrealDB publish port 8000? | Top source `01-access-policy-embedded.md` | 0.070s | PASS |
| What must backups include? | Top source `02-backup-policy.md` | 0.075s | PASS |
| What remains canonical during the pilot? | Top source `03-knowledge-scope.md` | 0.073s | PASS |

Accuracy: 5/5 for retrieval-backed known-answer source selection.

Citation/source quality: acceptable for this pilot smoke test because every known-answer query returned the expected source title at rank 1. Full citation UX still needs evaluation in the browser and with generated answers once the local LLM path is performant.

Summary quality: not fully scored through Open Notebook transformations because local generated-answer path timed out; manual source retrieval supports the required summaries, but generated summary evaluation remains a follow-up before production use.

Latency: text search was fast (0.064s-0.262s). Vector smoke was acceptable for a first local embedding path (4.764s). Local generated ask path was not acceptable with the tiny model (timeout at 300s).

Operational friction: initial setup was straightforward after using the `/api` prefix; image pulls were large, Ollama model pulls succeeded, and backup scripts briefly stop the pilot to make a consistent RocksDB snapshot.

## Backup/restore validation

Backup command: `scripts/open-notebook-pilot/backup.sh`

Backup artifact:

- `/opt/bytelyst/open-notebook-pilot/backups/open-notebook-pilot-20260628T185205Z.tar.gz`
- `/opt/bytelyst/open-notebook-pilot/backups/open-notebook-pilot-20260628T185205Z.manifest.txt`

Restore-readiness command:

- `scripts/open-notebook-pilot/restore-test.sh /opt/bytelyst/open-notebook-pilot/backups/open-notebook-pilot-20260628T185205Z.tar.gz`

Result: PASS. Restore test verified the archive contains:

- `opt/bytelyst/open-notebook-pilot/surreal_data`
- `opt/bytelyst/open-notebook-pilot/notebook_data`
- `etc/bytelyst/open-notebook-pilot.env`

Post-backup health check: PASS after restart wait, `/health` returned `{"status":"healthy"}`.

## Recommendation

Keep the pilot running privately for additional low-risk browser testing and try one of these next:

1. Pull/use a stronger local model that fits VM headroom, then rerun `/api/search/ask/simple` known-answer tests.
2. Use a sanitized cloud comparison model only on the three test docs to benchmark generated answer/citation quality.
3. If generated answers pass, expand to 10-20 sanitized ByteLyst docs and promote the backup script into the normal VM backup process.

Do not expose this pilot publicly.