103 lines
5.3 KiB
Markdown
103 lines
5.3 KiB
Markdown
# ByteLyst Open Notebook private pilot evaluation report
|
|
|
|
Date: 2026-06-28
|
|
Task: t_b57eb21e
|
|
Deployment: `/opt/bytelyst/learning_ai_devops_tools/deployments/open-notebook-pilot/docker-compose.yml`
|
|
Runtime: `/opt/bytelyst/open-notebook-pilot`
|
|
|
|
## Verdict
|
|
|
|
Private pilot deployment is operational and hardened for localhost-only evaluation. It is not public and should remain excluded from public Caddy/subdomain routing.
|
|
|
|
Pilot readiness: 86/100
|
|
|
|
Main remaining gap: full LLM `/api/search/ask/simple` Q&A timed out with the tiny local `qwen2.5:0.5b` model during first smoke testing. Retrieval, ingestion, embeddings, backup, and restore-readiness passed; source-grounded answer generation needs a larger/faster local model or a sanitized cloud comparison model before relying on generated answers.
|
|
|
|
## Hardening checks
|
|
|
|
| Check | Result | Evidence |
|
|
|---|---:|---|
|
|
| UI localhost-only | PASS | Docker published `127.0.0.1:8502->8502/tcp` only |
|
|
| API localhost-only | PASS | Docker published `127.0.0.1:5055->5055/tcp` only |
|
|
| SurrealDB public exposure | PASS | No SurrealDB host port published; internal Docker network only |
|
|
| Ollama public exposure | PASS | Ollama exposes container port only; no host published port |
|
|
| Strong secrets outside git | PASS | `/etc/bytelyst/open-notebook-pilot.env`, mode `0600`, contains generated app/encryption/SurrealDB credentials |
|
|
| CORS explicit | PASS | `CORS_ORIGINS=http://127.0.0.1:8502,http://localhost:8502` in protected env |
|
|
| Public dashboard exposure | PASS | No Caddy route added; access path is localhost/SSH tunnel only |
|
|
|
|
## Local model/provider checks
|
|
|
|
| Item | Result |
|
|
|---|---|
|
|
| Ollama service | Running in private compose network |
|
|
| LLM model pulled | `qwen2.5:0.5b` |
|
|
| Embedding model pulled | `nomic-embed-text` |
|
|
| Open Notebook model registry | Created `ollama` language and embedding models |
|
|
| Default models | Chat/transformation/tools/large-context set to `qwen2.5:0.5b`; embeddings set to `nomic-embed-text` |
|
|
| Vector search | PASS: vector smoke query returned `01-access-policy-embedded.md` in 4.764s |
|
|
| Generated ask/Q&A | NEEDS FOLLOW-UP: `/api/search/ask/simple` timed out at 300s with `qwen2.5:0.5b` |
|
|
|
|
## Sanitized docs ingested
|
|
|
|
Source directory: `/opt/bytelyst/open-notebook-pilot/test-docs`
|
|
|
|
- `01-access-policy.md`
|
|
- `02-backup-policy.md`
|
|
- `03-knowledge-scope.md`
|
|
- `01-access-policy-embedded.md` (extra embedded source for vector smoke)
|
|
|
|
Notebook created by smoke script: `notebook:2iyp5pykncuhwul2qc9g`
|
|
|
|
## Known-answer evaluation
|
|
|
|
Raw results: `/opt/bytelyst/open-notebook-pilot/reports/known-answer-results.json`
|
|
|
|
| Question | Retrieval result | Latency | Pass |
|
|
|---|---|---:|---:|
|
|
| Should the pilot be exposed as a public dashboard? | Top source `01-access-policy-embedded.md` | 0.262s | PASS |
|
|
| Which host ports should UI and API use? | Top source `01-access-policy-embedded.md` | 0.064s | PASS |
|
|
| Should SurrealDB publish port 8000? | Top source `01-access-policy-embedded.md` | 0.070s | PASS |
|
|
| What must backups include? | Top source `02-backup-policy.md` | 0.075s | PASS |
|
|
| What remains canonical during the pilot? | Top source `03-knowledge-scope.md` | 0.073s | PASS |
|
|
|
|
Accuracy: 5/5 for retrieval-backed known-answer source selection.
|
|
|
|
Citation/source quality: acceptable for this pilot smoke test because every known-answer query returned the expected source title at rank 1. Full citation UX still needs evaluation in the browser and with generated answers once the local LLM path is performant.
|
|
|
|
Summary quality: not fully scored through Open Notebook transformations because local generated-answer path timed out; manual source retrieval supports the required summaries, but generated summary evaluation remains a follow-up before production use.
|
|
|
|
Latency: text search was fast (0.064s-0.262s). Vector smoke was acceptable for a first local embedding path (4.764s). Local generated ask path was not acceptable with the tiny model (timeout at 300s).
|
|
|
|
Operational friction: initial setup was straightforward after using the `/api` prefix; image pulls were large, Ollama model pulls succeeded, and backup scripts briefly stop the pilot to make a consistent RocksDB snapshot.
|
|
|
|
## Backup/restore validation
|
|
|
|
Backup command: `scripts/open-notebook-pilot/backup.sh`
|
|
|
|
Backup artifact:
|
|
|
|
- `/opt/bytelyst/open-notebook-pilot/backups/open-notebook-pilot-20260628T185205Z.tar.gz`
|
|
- `/opt/bytelyst/open-notebook-pilot/backups/open-notebook-pilot-20260628T185205Z.manifest.txt`
|
|
|
|
Restore-readiness command:
|
|
|
|
- `scripts/open-notebook-pilot/restore-test.sh /opt/bytelyst/open-notebook-pilot/backups/open-notebook-pilot-20260628T185205Z.tar.gz`
|
|
|
|
Result: PASS. Restore test verified the archive contains:
|
|
|
|
- `opt/bytelyst/open-notebook-pilot/surreal_data`
|
|
- `opt/bytelyst/open-notebook-pilot/notebook_data`
|
|
- `etc/bytelyst/open-notebook-pilot.env`
|
|
|
|
Post-backup health check: PASS after restart wait, `/health` returned `{"status":"healthy"}`.
|
|
|
|
## Recommendation
|
|
|
|
Keep the pilot running privately for additional low-risk browser testing and try one of these next:
|
|
|
|
1. Pull/use a stronger local model that fits VM headroom, then rerun `/api/search/ask/simple` known-answer tests.
|
|
2. Use a sanitized cloud comparison model only on the three test docs to benchmark generated answer/citation quality.
|
|
3. If generated answers pass, expand to 10-20 sanitized ByteLyst docs and promote the backup script into the normal VM backup process.
|
|
|
|
Do not expose this pilot publicly.
|