From dfe876d49b765b8f43f057dfbaf5ec653600f413 Mon Sep 17 00:00:00 2001 From: root Date: Thu, 7 May 2026 06:48:29 +0000 Subject: [PATCH] docs(chat): add capability audit --- docs/CHAT_BOT_CAPABILITY_AUDIT.md | 293 ++++++++++++++++++++++++++++++ 1 file changed, 293 insertions(+) create mode 100644 docs/CHAT_BOT_CAPABILITY_AUDIT.md diff --git a/docs/CHAT_BOT_CAPABILITY_AUDIT.md b/docs/CHAT_BOT_CAPABILITY_AUDIT.md new file mode 100644 index 0000000..97a3ae4 --- /dev/null +++ b/docs/CHAT_BOT_CAPABILITY_AUDIT.md @@ -0,0 +1,293 @@ +# Chat / Bot Capability Audit + +Reviewed on `2026-05-07` against the current `main` branch. + +This audit is separate from the completed UI migration roadmap. It assesses the functional maturity of: + +- trading bot/runtime +- realtime socket integration +- Trade Plans automation +- reconciliation / operational tooling +- chat / AI assistant capabilities + +## Executive Summary + +The app has real bot and chat functionality today. It is not a mock product. + +Current maturity estimate: + +| Subsystem | Completion | Notes | +|---|---:|---| +| Trading bot/runtime | `80-85%` | Real execution, state recovery, order lifecycle, and user-scoped broker credentials exist. | +| Realtime web integration | `85-90%` | Socket-driven state and health flows are implemented and tested. | +| Trade Plans automation | `85%` | Buy/exit automation, portfolio linkage, lifecycle toasts, and setup history are in place. | +| Reconciliation / ops | `75-80%` | Strong operational depth exists, but manual-review and quarantine paths are still prominent. | +| Chat / AI assistant | `60-70%` | Real assistant exists, but it is still mostly profile-generation/config assistance, not a full trading copilot. | +| Overall product maturity | `75-80%` | The product is functional and substantial, but not yet “quiet”, fully unified, or deeply assistant-driven. | + +## What Exists Today + +### 1. Trading runtime is real + +Evidence: + +- [backend/src/index.ts](/opt/bytelyst/trading/learning_ai_invt_trdg/backend/src/index.ts) +- [backend/src/services/TradeExecutor.ts](/opt/bytelyst/trading/learning_ai_invt_trdg/backend/src/services/TradeExecutor.ts) +- [backend/src/services/apiServer.ts](/opt/bytelyst/trading/learning_ai_invt_trdg/backend/src/services/apiServer.ts) + +Implemented behavior includes: + +- authenticated trading APIs +- live trading loop +- manual trader/profile routing +- order and position lifecycle tracking +- user-scoped Alpaca credential usage +- health tracking and loop telemetry + +Conclusion: + +- this is not a toy bot shell +- execution/runtime infrastructure is one of the stronger parts of the app + +### 2. Realtime web state is real + +Evidence: + +- [web/src/hooks/useWebSocket.ts](/opt/bytelyst/trading/learning_ai_invt_trdg/web/src/hooks/useWebSocket.ts) +- [web/src/hooks/useWebSocket.dom.test.tsx](/opt/bytelyst/trading/learning_ai_invt_trdg/web/src/hooks/useWebSocket.dom.test.tsx) + +Implemented behavior includes: + +- authenticated socket connection +- runtime symbol/order/history/alert updates +- health and reconciliation metrics in app state +- tested event application behavior + +Conclusion: + +- realtime state propagation is substantially implemented +- this is a strong, working foundation for trading/ops UX + +### 3. Trade Plans are a real workflow + +Evidence: + +- [web/src/views/SimpleView.tsx](/opt/bytelyst/trading/learning_ai_invt_trdg/web/src/views/SimpleView.tsx) +- [web/src/views/SimpleView.test.ts](/opt/bytelyst/trading/learning_ai_invt_trdg/web/src/views/SimpleView.test.ts) +- [backend/src/index.ts](/opt/bytelyst/trading/learning_ai_invt_trdg/backend/src/index.ts) + +Implemented behavior includes: + +- short-term buy plans +- profit-exit attachment to existing holdings +- concurrent same-symbol setup support +- long-term conversion / resume exit management +- portfolio linkage +- setup-level runtime events +- setup activity history and timeline + +Conclusion: + +- Trade Plans are feature-complete enough for real use +- remaining work is mostly refinement, not existence + +### 4. Reconciliation and ops tooling are deep + +Evidence: + +- [backend/src/services/reconciliationService.ts](/opt/bytelyst/trading/learning_ai_invt_trdg/backend/src/services/reconciliationService.ts) +- [backend/src/services/healthTracker.ts](/opt/bytelyst/trading/learning_ai_invt_trdg/backend/src/services/healthTracker.ts) +- [web/src/tabs/ReconciliationAuditPanel.tsx](/opt/bytelyst/trading/learning_ai_invt_trdg/web/src/tabs/ReconciliationAuditPanel.tsx) +- [web/src/tabs/AdminTab.tsx](/opt/bytelyst/trading/learning_ai_invt_trdg/web/src/tabs/AdminTab.tsx) + +Implemented behavior includes: + +- reconciliation cycles +- parity / backfill / coverage / repair services +- health metrics and watchdog signals +- admin reconciliation audit UI +- operational event surfaces + +Conclusion: + +- the platform has real operational depth +- the remaining issue is not missing machinery; it is reducing how often users/admins need it + +### 5. Chat assistant is real, but narrower + +Evidence: + +- [web/src/components/ChatControl.tsx](/opt/bytelyst/trading/learning_ai_invt_trdg/web/src/components/ChatControl.tsx) +- [web/src/components/ChatControl.dom.test.tsx](/opt/bytelyst/trading/learning_ai_invt_trdg/web/src/components/ChatControl.dom.test.tsx) +- [backend/src/services/apiServer.ts](/opt/bytelyst/trading/learning_ai_invt_trdg/backend/src/services/apiServer.ts:2626) +- [backend/src/services/aiClient.ts](/opt/bytelyst/trading/learning_ai_invt_trdg/backend/src/services/aiClient.ts) + +Implemented behavior includes: + +- authenticated `/api/chat` +- LLM provider fallback chain +- deterministic local fallback if AI output fails +- profile create/update/explain intent +- draft editing and apply flow in the UI +- quick actions for common profile prompts + +Conclusion: + +- this is a real assistant workflow +- but it is still mostly “profile configuration assistant”, not a broad product copilot + +## Major Gaps + +### 1. Chat is not yet a full portfolio / execution / reconciliation copilot + +Current state: + +- chat can generate or modify profiles +- chat is not yet deeply integrated into: + - portfolio explanation + - trade-fire reasoning + - exit-block diagnosis + - reconciliation guidance + - post-trade review + +Impact: + +- the product has strong underlying runtime data, but the assistant does not yet exploit it fully + +Priority: + +- highest + +### 2. Explanation surfaces are still shallow + +Evidence: + +- [web/src/lib/StrategyExplanationService.ts](/opt/bytelyst/trading/learning_ai_invt_trdg/web/src/lib/StrategyExplanationService.ts) + +Current state: + +- explanation logic is heuristic and limited +- comments in code explicitly note simplified behavior + +Impact: + +- users can see state, but not always the most useful “why” behind it + +Priority: + +- high + +### 3. Reconciliation maturity is strong, but still noisy + +Current state: + +- there are many reconciliation and quarantine paths +- this is good operationally, but it usually means the user/admin experience is still noisy + +Impact: + +- reliability is protected +- clarity and quietness can still improve + +Priority: + +- high + +### 4. Error surfaces are still uneven + +Examples: + +- user-facing flows still rely on thrown messages in several places +- some UI/runtime areas still log directly with `console.error` + +Impact: + +- technically functional +- not fully product-polished + +Priority: + +- medium + +### 5. Chat lacks deeper action verbs + +Current state: + +- assistant handles `create_profile`, `update_profile`, `explain` +- it does not clearly own richer actions like: + - explain current position + - explain why no trade fired + - inspect blocked exit + - summarize reconciliation anomalies + - propose fix without leaving chat + +Impact: + +- user expectation of “AI trading assistant” is only partially met + +Priority: + +- highest + +## Recommended Next Implementation Order + +### Phase A: Make chat a real copilot + +1. Add assistant intents for: + - explain current holding + - explain why a symbol/profile did not enter + - explain why an exit is blocked + - summarize reconciliation issues + - suggest safe next actions +2. Feed live runtime context into `/api/chat`: + - holdings + - active profiles + - recent orders + - recent operational events + - reconciliation health summary +3. Return structured action types, not only free text: + - `explain_position` + - `explain_blocker` + - `summarize_reconciliation` + - `recommend_profile_change` + +### Phase B: Improve explanation quality outside chat + +1. Replace heuristic-only explanation paths with lifecycle-aware reasoning +2. Reuse the same explanation contracts in: + - Portfolio + - Trade Plans + - Overview + - Chat + +### Phase C: Quiet the ops layer + +1. Reduce quarantine/manual-review frequency +2. Improve user-facing messages for blocked trades/exits +3. Add clearer reconciliation state summaries for admins and end users + +### Phase D: Polish error UX + +1. unify user-safe error formatting +2. reduce raw thrown-message exposure +3. reduce console-noise paths in stable workflows + +## Recommended Success Criteria + +The chat/bot experience should be considered “substantially complete” when: + +- chat can explain active positions and blocked actions using live runtime context +- chat can explain “why nothing happened” for a profile or symbol +- Trade Plans, Portfolio, and Chat share consistent lifecycle reasoning +- reconciliation remains available but becomes rarer and quieter +- user-facing errors are product-grade instead of raw internal messages + +## Bottom Line + +The app already has real bot functionality and a real assistant. + +The biggest remaining gap is not “build chat from scratch.” It is: + +- turning the current assistant from a profile generator into a true trading copilot + +That is the highest-value next product move because the runtime, socket state, Trade Plans, and reconciliation foundations are already in place.