bytelyst/bytelyst-devops-tools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
8d1c777c65
bytelyst-devops-tools/deployments/open-notebook-pilot/known-answer-evaluation-report.md
Hermes VM e14a46f3e0 Deploy private Open Notebook pilot
2026-06-28 18:56:00 +00:00

5.3 KiB
Raw Blame History

ByteLyst Open Notebook private pilot evaluation report

Date: 2026-06-28 Task: t_b57eb21e Deployment: /opt/bytelyst/learning_ai_devops_tools/deployments/open-notebook-pilot/docker-compose.yml Runtime: /opt/bytelyst/open-notebook-pilot

Verdict

Private pilot deployment is operational and hardened for localhost-only evaluation. It is not public and should remain excluded from public Caddy/subdomain routing.

Pilot readiness: 86/100

Main remaining gap: full LLM /api/search/ask/simple Q&A timed out with the tiny local qwen2.5:0.5b model during first smoke testing. Retrieval, ingestion, embeddings, backup, and restore-readiness passed; source-grounded answer generation needs a larger/faster local model or a sanitized cloud comparison model before relying on generated answers.

Hardening checks

Check Result Evidence
UI localhost-only PASS Docker published 127.0.0.1:8502->8502/tcp only
API localhost-only PASS Docker published 127.0.0.1:5055->5055/tcp only
SurrealDB public exposure PASS No SurrealDB host port published; internal Docker network only
Ollama public exposure PASS Ollama exposes container port only; no host published port
Strong secrets outside git PASS /etc/bytelyst/open-notebook-pilot.env, mode 0600, contains generated app/encryption/SurrealDB credentials
CORS explicit PASS CORS_ORIGINS=http://127.0.0.1:8502,http://localhost:8502 in protected env
Public dashboard exposure PASS No Caddy route added; access path is localhost/SSH tunnel only

Local model/provider checks

Item Result
Ollama service Running in private compose network
LLM model pulled qwen2.5:0.5b
Embedding model pulled nomic-embed-text
Open Notebook model registry Created ollama language and embedding models
Default models Chat/transformation/tools/large-context set to qwen2.5:0.5b; embeddings set to nomic-embed-text
Vector search PASS: vector smoke query returned 01-access-policy-embedded.md in 4.764s
Generated ask/Q&A NEEDS FOLLOW-UP: /api/search/ask/simple timed out at 300s with qwen2.5:0.5b

Sanitized docs ingested

Source directory: /opt/bytelyst/open-notebook-pilot/test-docs

  • 01-access-policy.md
  • 02-backup-policy.md
  • 03-knowledge-scope.md
  • 01-access-policy-embedded.md (extra embedded source for vector smoke)

Notebook created by smoke script: notebook:2iyp5pykncuhwul2qc9g

Known-answer evaluation

Raw results: /opt/bytelyst/open-notebook-pilot/reports/known-answer-results.json

Question Retrieval result Latency Pass
Should the pilot be exposed as a public dashboard? Top source 01-access-policy-embedded.md 0.262s PASS
Which host ports should UI and API use? Top source 01-access-policy-embedded.md 0.064s PASS
Should SurrealDB publish port 8000? Top source 01-access-policy-embedded.md 0.070s PASS
What must backups include? Top source 02-backup-policy.md 0.075s PASS
What remains canonical during the pilot? Top source 03-knowledge-scope.md 0.073s PASS

Accuracy: 5/5 for retrieval-backed known-answer source selection.

Citation/source quality: acceptable for this pilot smoke test because every known-answer query returned the expected source title at rank 1. Full citation UX still needs evaluation in the browser and with generated answers once the local LLM path is performant.

Summary quality: not fully scored through Open Notebook transformations because local generated-answer path timed out; manual source retrieval supports the required summaries, but generated summary evaluation remains a follow-up before production use.

Latency: text search was fast (0.064s-0.262s). Vector smoke was acceptable for a first local embedding path (4.764s). Local generated ask path was not acceptable with the tiny model (timeout at 300s).

Operational friction: initial setup was straightforward after using the /api prefix; image pulls were large, Ollama model pulls succeeded, and backup scripts briefly stop the pilot to make a consistent RocksDB snapshot.

Backup/restore validation

Backup command: scripts/open-notebook-pilot/backup.sh

Backup artifact:

  • /opt/bytelyst/open-notebook-pilot/backups/open-notebook-pilot-20260628T185205Z.tar.gz
  • /opt/bytelyst/open-notebook-pilot/backups/open-notebook-pilot-20260628T185205Z.manifest.txt

Restore-readiness command:

  • scripts/open-notebook-pilot/restore-test.sh /opt/bytelyst/open-notebook-pilot/backups/open-notebook-pilot-20260628T185205Z.tar.gz

Result: PASS. Restore test verified the archive contains:

  • opt/bytelyst/open-notebook-pilot/surreal_data
  • opt/bytelyst/open-notebook-pilot/notebook_data
  • etc/bytelyst/open-notebook-pilot.env

Post-backup health check: PASS after restart wait, /health returned {"status":"healthy"}.

Recommendation

Keep the pilot running privately for additional low-risk browser testing and try one of these next:

  1. Pull/use a stronger local model that fits VM headroom, then rerun /api/search/ask/simple known-answer tests.
  2. Use a sanitized cloud comparison model only on the three test docs to benchmark generated answer/citation quality.
  3. If generated answers pass, expand to 10-20 sanitized ByteLyst docs and promote the backup script into the normal VM backup process.

Do not expose this pilot publicly.