docs(roadmaps): add AI diagnostic assistant, A/B testing, and churn prediction roadmaps
- AI Diagnostic Assistant: LLM-powered root cause analysis, error clustering, natural language queries - Intelligent A/B Testing: Thompson sampling, Bayesian early stopping, AI hypothesis generation - Predictive Churn & Health: XGBoost models, health scoring, automated retention campaigns All roadmaps include: - Implementation tracking tables with status/commit columns - Quick reference sections with file structures - Phase-by-phase task breakdowns with [X.Y.Z] codes
This commit is contained in:
saravanakumardb1
parent
d510867b87
commit
e98380003b
597
docs/roadmaps/AI_DIAGNOSTIC_ASSISTANT_ROADMAP.md
Normal file
597
docs/roadmaps/AI_DIAGNOSTIC_ASSISTANT_ROADMAP.md
Normal file
@ -0,0 +1,597 @@
|
|||||||
|
# AI Diagnostic Assistant — Implementation Roadmap
|
||||||
|
|
||||||
|
> **Module:** `platform-service/src/modules/ai-diagnostics/`
|
||||||
|
> **Admin UI:** `/ops/ai-diagnostics/`
|
||||||
|
> **Target:** LLM-powered root cause analysis from telemetry + debug sessions
|
||||||
|
> **Estimated Effort:** 2–3 weeks
|
||||||
|
> **Status:** 🟡 Planning
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Executive Summary
|
||||||
|
|
||||||
|
This roadmap delivers an **AI-powered diagnostic assistant** that analyzes error patterns, debug session data, and telemetry to automatically suggest root causes—like having a senior engineer on-call 24/7. Engineers can ask natural language questions like _"Why did the iOS keyboard crash yesterday?"_ and receive AI-generated hypotheses with supporting evidence.
|
||||||
|
|
||||||
|
### Key Differentiators vs. Manual Debugging
|
||||||
|
|
||||||
|
| Feature | Manual Debugging | AI Diagnostic Assistant |
|
||||||
|
| ----------------- | --------------------------- | ----------------------------------- |
|
||||||
|
| Query | SQL + log grep | **Natural language** |
|
||||||
|
| Pattern Detection | Hours of manual correlation | **AI finds hidden patterns** |
|
||||||
|
| Context Assembly | Check 5+ systems manually | **Auto-assembles timeline** |
|
||||||
|
| Hypothesis | Engineer intuition | **LLM-generated + evidence** |
|
||||||
|
| Learning | Per-engineer experience | **Accumulates across all sessions** |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 1: Data Pipeline & Embeddings (Week 1)
|
||||||
|
|
||||||
|
**Goal:** Extract, normalize, and embed error data for semantic search and clustering.
|
||||||
|
|
||||||
|
### 1.1 Error Fingerprinting & Clustering
|
||||||
|
|
||||||
|
- [ ] **1.1.1** Create `modules/ai-diagnostics/types.ts`
|
||||||
|
- [ ] `ErrorClusterDoc` — grouped similar errors with signature
|
||||||
|
- [ ] `ErrorFingerprint` — normalized stack trace hash
|
||||||
|
- [ ] `ClusterAnalysis` — AI-generated pattern description
|
||||||
|
- [ ] Zod schemas for all inputs
|
||||||
|
|
||||||
|
_Commit format:_ `git commit -m "feat(ai-diagnostics): add error clustering types [1.1.1]"` → `https://github.com/saravanakumardb1/learning_ai_common_plat/commit/<hash>`
|
||||||
|
|
||||||
|
- [ ] **1.1.2** Add Cosmos containers to `cosmos-init.ts`
|
||||||
|
- [ ] `error_clusters` (pk: `/productId`, TTL: 90 days)
|
||||||
|
- [ ] `error_fingerprints` (pk: `/fingerprintHash`, unique index)
|
||||||
|
- [ ] `diagnostic_insights` (pk: `/clusterId`, AI-generated analyses)
|
||||||
|
|
||||||
|
_Commit format:_ `git commit -m "feat(ai-diagnostics): add cosmos containers for error clustering [1.1.2]"`
|
||||||
|
|
||||||
|
- [ ] **1.1.3** Implement error normalization
|
||||||
|
- [ ] Stack trace parsing (remove line numbers, file paths)
|
||||||
|
- [ ] Message templating (replace UUIDs, timestamps, user IDs with placeholders)
|
||||||
|
- [ ] Fingerprint generation (SHA-256 of normalized error)
|
||||||
|
- [ ] Similarity scoring (Levenshtein for near-matches)
|
||||||
|
|
||||||
|
_Commit format:_ `git commit -m "feat(ai-diagnostics): implement error normalization and fingerprinting [1.1.3]"`
|
||||||
|
|
||||||
|
### 1.2 Vector Embeddings for Semantic Search
|
||||||
|
|
||||||
|
- [ ] **1.2.1** Create embedding pipeline
|
||||||
|
- [ ] Azure OpenAI `text-embedding-3-small` integration
|
||||||
|
- [ ] Error message + stack trace → 1536-dim vector
|
||||||
|
- [ ] Batch embedding job (100 errors at a time)
|
||||||
|
- [ ] **1.2.2** Cosmos DB vector search setup
|
||||||
|
- [ ] Store embeddings in `error_clusters` documents
|
||||||
|
- [ ] Cosine similarity query function
|
||||||
|
- [ ] Similar error lookup by vector distance
|
||||||
|
- [ ] **1.2.3** Clustering algorithm
|
||||||
|
- [ ] HDBSCAN for density-based clustering
|
||||||
|
- [ ] DBSCAN fallback for smaller datasets
|
||||||
|
- [ ] Auto-determine cluster count (no manual k)
|
||||||
|
- [ ] Re-cluster nightly as new errors arrive
|
||||||
|
|
||||||
|
### 1.3 Telemetry Ingestion for Context
|
||||||
|
|
||||||
|
- [ ] **1.3.1** Link telemetry to errors
|
||||||
|
- [ ] `correlationId` propagation across services
|
||||||
|
- [ ] 5-minute window: error → preceding telemetry events
|
||||||
|
- [ ] Session state reconstruction (what user was doing)
|
||||||
|
- [ ] **1.3.2** Enrich error context
|
||||||
|
- [ ] Device info (OS version, model, memory)
|
||||||
|
- [ ] App state (screen, feature flags, config)
|
||||||
|
- [ ] Recent API calls (network trace from diagnostics)
|
||||||
|
- [ ] Recent user actions (breadcrumb trail)
|
||||||
|
|
||||||
|
**Phase 1 Exit Criteria:**
|
||||||
|
|
||||||
|
- [ ] Errors auto-clustered with 90%+ accuracy
|
||||||
|
- [ ] Vector search returns semantically similar errors
|
||||||
|
- [ ] 10,000+ historical errors embedded and clustered
|
||||||
|
- [ ] Correlation pipeline links errors to telemetry context
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 2: LLM Analysis Engine (Week 1–2)
|
||||||
|
|
||||||
|
### 2.1 Prompt Engineering & Analysis Pipeline
|
||||||
|
|
||||||
|
- [ ] **2.1.1** Create analysis prompts
|
||||||
|
- [ ] `ROOT_CAUSE_ANALYSIS` prompt template
|
||||||
|
|
||||||
|
```
|
||||||
|
Given this error cluster:
|
||||||
|
- Error signature: {fingerprint}
|
||||||
|
- Sample stack traces: {samples}
|
||||||
|
- Common context: {deviceStats}, {appState}
|
||||||
|
- Preceding events: {breadcrumbSummary}
|
||||||
|
- Similar resolved issues: {relatedClusters}
|
||||||
|
|
||||||
|
Analyze and provide:
|
||||||
|
1. Likely root cause category (config, dependency, logic, resource, external)
|
||||||
|
2. Specific hypothesis with reasoning
|
||||||
|
3. Evidence confidence (high/medium/low)
|
||||||
|
4. Suggested investigation steps
|
||||||
|
5. Potential fix direction
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] `PATTERN_SUMMARY` prompt for cluster descriptions
|
||||||
|
- [ ] `COMPARATIVE_ANALYSIS` for error vs. baseline
|
||||||
|
|
||||||
|
- [ ] **2.1.2** LLM integration
|
||||||
|
- [ ] Azure OpenAI GPT-4o-mini for analysis (cost-effective)
|
||||||
|
- [ ] GPT-4o for complex multi-factor analysis
|
||||||
|
- [ ] Response JSON schema enforcement
|
||||||
|
- [ ] Retry logic with exponential backoff
|
||||||
|
|
||||||
|
### 2.2 Insight Generation Service
|
||||||
|
|
||||||
|
- [ ] **2.2.1** Create `modules/ai-diagnostics/analyzer.ts`
|
||||||
|
- [ ] `analyzeCluster(clusterId)` — full analysis workflow
|
||||||
|
- [ ] `generateInsight(errorContext)` — single error analysis
|
||||||
|
- [ ] `compareClusters(clusterA, clusterB)` — diff analysis
|
||||||
|
- [ ] **2.2.2** Analysis workflow
|
||||||
|
- [ ] Fetch cluster data + related telemetry
|
||||||
|
- [ ] Build LLM context (respect token limits)
|
||||||
|
- [ ] Call LLM with structured prompt
|
||||||
|
- [ ] Parse and validate response
|
||||||
|
- [ ] Store insight in `diagnostic_insights`
|
||||||
|
- [ ] **2.2.3** Confidence scoring
|
||||||
|
- [ ] Evidence count weighting
|
||||||
|
- [ ] Similar resolved issue bonus
|
||||||
|
- [ ] Recency decay (older patterns = lower confidence)
|
||||||
|
- [ ] Multi-model consensus (if available)
|
||||||
|
|
||||||
|
### 2.3 Continuous Learning
|
||||||
|
|
||||||
|
- [ ] **2.3.1** Feedback loop
|
||||||
|
- [ ] Engineer feedback: "Was this insight helpful? 👍/👎"
|
||||||
|
- [ ] Resolution tracking (link commits to clusters)
|
||||||
|
- [ ] Confidence recalibration based on outcomes
|
||||||
|
- [ ] **2.3.2** Pattern accumulation
|
||||||
|
- [ ] "Known issues" database (manually curated)
|
||||||
|
- [ ] Historical fix patterns (what solved similar issues)
|
||||||
|
- [ ] Regression detection (old issue reappearing)
|
||||||
|
|
||||||
|
**Phase 2 Exit Criteria:**
|
||||||
|
|
||||||
|
- [ ] LLM generates root cause hypotheses with evidence
|
||||||
|
- [ ] Confidence scores align with actual resolution rates
|
||||||
|
- [ ] Analysis completes in < 5 seconds for typical clusters
|
||||||
|
- [ ] Feedback loop capturing engineer ratings
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 3: Natural Language Query Interface (Week 2)
|
||||||
|
|
||||||
|
### 3.1 Query Understanding
|
||||||
|
|
||||||
|
- [ ] **3.1.1** Create `modules/ai-diagnostics/query-parser.ts`
|
||||||
|
- [ ] Intent classification (root cause, pattern search, comparison, trend)
|
||||||
|
- [ ] Entity extraction (product, time range, error type, user segment)
|
||||||
|
- [ ] Temporal parsing ("yesterday", "last week", "since v2.1")
|
||||||
|
- [ ] Constraint identification ("only iOS", "excluding beta users")
|
||||||
|
- [ ] **3.1.2** Query patterns
|
||||||
|
- [ ] Root cause: _"Why did X happen?"_ → analyze cluster
|
||||||
|
- [ ] Pattern search: _"Show me similar crashes"_ → vector search
|
||||||
|
- [ ] Comparison: _"Did error rate increase after release?"_ → trend analysis
|
||||||
|
- [ ] User impact: _"How many users affected by Y?"_ → aggregation query
|
||||||
|
|
||||||
|
### 3.2 Query Execution Engine
|
||||||
|
|
||||||
|
- [ ] **3.2.1** Query → data pipeline
|
||||||
|
- [ ] Map entities to Cosmos queries
|
||||||
|
- [ ] Fetch relevant clusters, telemetry, sessions
|
||||||
|
- [ ] Assemble context for response generation
|
||||||
|
- [ ] **3.2.2** Response generation
|
||||||
|
- [ ] Direct answers for simple queries
|
||||||
|
- [ ] AI-generated summaries for complex analysis
|
||||||
|
- [ ] Data + visualization suggestions
|
||||||
|
- [ ] Drill-down links for exploration
|
||||||
|
|
||||||
|
### 3.3 REST API Routes
|
||||||
|
|
||||||
|
- [ ] **3.3.1** Create `modules/ai-diagnostics/routes.ts`
|
||||||
|
- [ ] `POST /ai-diagnostics/query` — natural language question
|
||||||
|
- [ ] `GET /ai-diagnostics/clusters/:id/analysis` — pre-computed insight
|
||||||
|
- [ ] `POST /ai-diagnostics/clusters/:id/analyze` — trigger fresh analysis
|
||||||
|
- [ ] `GET /ai-diagnostics/suggestions` — auto-suggested investigations
|
||||||
|
- [ ] `POST /ai-diagnostics/feedback` — submit insight rating
|
||||||
|
|
||||||
|
**Phase 3 Exit Criteria:**
|
||||||
|
|
||||||
|
- [ ] Natural language queries parse correctly (90%+ intent accuracy)
|
||||||
|
- [ ] Query → response pipeline < 3 seconds
|
||||||
|
- [ ] Complex queries return structured answers with evidence
|
||||||
|
- [ ] API routes tested and documented
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 4: Admin Dashboard UI (Week 2–3)
|
||||||
|
|
||||||
|
### 4.1 AI Insights Page
|
||||||
|
|
||||||
|
- [ ] **4.1.1** Create `/ops/ai-diagnostics/page.tsx`
|
||||||
|
- [ ] Smart search bar (natural language input)
|
||||||
|
- [ ] Suggested queries based on recent errors
|
||||||
|
- [ ] Recent AI-generated insights list
|
||||||
|
- [ ] Trending clusters (auto-detected anomalies)
|
||||||
|
- [ ] **4.1.2** Query results view
|
||||||
|
- [ ] AI-generated answer with confidence badge
|
||||||
|
- [ ] Supporting evidence cards (cluster stats, sample errors)
|
||||||
|
- [ ] Related debug sessions (linked traces)
|
||||||
|
- [ ] Timeline visualization of error pattern
|
||||||
|
- [ ] "Investigate further" actions
|
||||||
|
|
||||||
|
### 4.2 Cluster Detail with AI Analysis
|
||||||
|
|
||||||
|
- [ ] **4.2.1** Enhance error cluster detail
|
||||||
|
- [ ] AI-generated summary card ("This appears to be...")
|
||||||
|
- [ ] Root cause hypothesis with confidence
|
||||||
|
- [ ] Evidence breakdown (stack samples, device patterns, API failures)
|
||||||
|
- [ ] Suggested fixes from similar resolved issues
|
||||||
|
- [ ] "Request deeper analysis" button (GPT-4o)
|
||||||
|
- [ ] **4.2.2** Interactive investigation
|
||||||
|
- [ ] Compare with other clusters ("Show me similar issues")
|
||||||
|
- [ ] Filter by context (OS version, app version, feature flags)
|
||||||
|
- [ ] View affected user journeys (breadcrumb trails)
|
||||||
|
|
||||||
|
### 4.3 Proactive Alerts
|
||||||
|
|
||||||
|
- [ ] **4.3.1** Anomaly detection
|
||||||
|
- [ ] Auto-detect emerging error clusters
|
||||||
|
- [ ] Spike in existing cluster frequency
|
||||||
|
- [ ] New error types after releases
|
||||||
|
- [ ] **4.3.2** AI-generated alerts
|
||||||
|
- [ ] Slack/Teams notification with summary
|
||||||
|
- [ ] "Investigate in AI Diagnostics" deep link
|
||||||
|
- [ ] Auto-started debug session recommendations
|
||||||
|
|
||||||
|
**Phase 4 Exit Criteria:**
|
||||||
|
|
||||||
|
- [ ] Admin can ask questions and get AI-generated answers
|
||||||
|
- [ ] Cluster detail shows AI analysis with evidence
|
||||||
|
- [ ] Proactive alerts for emerging issues
|
||||||
|
- [ ] Full test coverage (UI + API)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 5: Advanced Capabilities (Future)
|
||||||
|
|
||||||
|
### 5.1 Multi-Modal Analysis
|
||||||
|
|
||||||
|
- [ ] Analyze screenshots from debug sessions for UI issues
|
||||||
|
- [ ] Voice transcription analysis (for voice app errors)
|
||||||
|
- [ ] Performance trace visualization with AI annotations
|
||||||
|
|
||||||
|
### 5.2 Predictive Diagnostics
|
||||||
|
|
||||||
|
- [ ] Pre-crash pattern detection (warn before crash happens)
|
||||||
|
- [ ] Resource exhaustion prediction (memory, disk, API quotas)
|
||||||
|
- [ ] Config drift detection ("this setting combination often fails")
|
||||||
|
|
||||||
|
### 5.3 Self-Healing Suggestions
|
||||||
|
|
||||||
|
- [ ] Auto-generated config recommendations
|
||||||
|
- [ ] Feature flag rollback suggestions
|
||||||
|
- [ ] Circuit breaker threshold recommendations
|
||||||
|
|
||||||
|
## Implementation Tracking
|
||||||
|
|
||||||
|
| Phase | Task | Status | Commit |
|
||||||
|
| ----- | -------------------------- | ------ | ------ |
|
||||||
|
| 1.1 | Error clustering types | ⬜ | — |
|
||||||
|
| 1.1 | Cosmos containers | ⬜ | — |
|
||||||
|
| 1.1 | Error normalization | ⬜ | — |
|
||||||
|
| 1.2 | Embedding pipeline | ⬜ | — |
|
||||||
|
| 1.2 | Vector search setup | ⬜ | — |
|
||||||
|
| 1.2 | Clustering algorithm | ⬜ | — |
|
||||||
|
| 1.3 | Telemetry linking | ⬜ | — |
|
||||||
|
| 1.3 | Error context enrichment | ⬜ | — |
|
||||||
|
| 2.1 | Analysis prompts | ⬜ | — |
|
||||||
|
| 2.1 | LLM integration | ⬜ | — |
|
||||||
|
| 2.2 | Insight generation service | ⬜ | — |
|
||||||
|
| 2.2 | Analysis workflow | ⬜ | — |
|
||||||
|
| 2.2 | Confidence scoring | ⬜ | — |
|
||||||
|
| 2.3 | Feedback loop | ⬜ | — |
|
||||||
|
| 2.3 | Pattern accumulation | ⬜ | — |
|
||||||
|
| 3.1 | Query parser | ⬜ | — |
|
||||||
|
| 3.1 | Query patterns | ⬜ | — |
|
||||||
|
| 3.2 | Query execution | ⬜ | — |
|
||||||
|
| 3.2 | Response generation | ⬜ | — |
|
||||||
|
| 3.3 | REST API routes | ⬜ | — |
|
||||||
|
| 4.1 | AI insights page | ⬜ | — |
|
||||||
|
| 4.1 | Query results view | ⬜ | — |
|
||||||
|
| 4.2 | Cluster detail | ⬜ | — |
|
||||||
|
| 4.2 | Interactive investigation | ⬜ | — |
|
||||||
|
| 4.3 | Proactive alerts | ⬜ | — |
|
||||||
|
|
||||||
|
**Legend:** ⬜ Not started | 🟡 In progress | ✅ Complete | ⏸️ Deferred
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Quick Reference for Implementing Agent
|
||||||
|
|
||||||
|
**📋 Full Roadmap:** `/Users/sd9235/code/mygh/learning_ai_common_plat/docs/roadmaps/AI_DIAGNOSTIC_ASSISTANT_ROADMAP.md`
|
||||||
|
|
||||||
|
**Key Files to Modify/Create:**
|
||||||
|
|
||||||
|
```
|
||||||
|
services/platform-service/
|
||||||
|
├── src/
|
||||||
|
│ ├── modules/ai-diagnostics/
|
||||||
|
│ │ ├── types.ts # [1.1.1] Error clustering types
|
||||||
|
│ │ ├── repository.ts # [1.2] Data access layer
|
||||||
|
│ │ ├── analyzer.ts # [2.2] LLM analysis engine
|
||||||
|
│ │ ├── query-parser.ts # [3.1] NL query understanding
|
||||||
|
│ │ ├── query-executor.ts # [3.2] Query execution
|
||||||
|
│ │ ├── routes.ts # [3.3] REST API
|
||||||
|
│ │ └── ai-diagnostics.test.ts # Tests
|
||||||
|
│ ├── lib/
|
||||||
|
│ │ ├── cosmos-init.ts # [1.1.2] Add containers
|
||||||
|
│ │ ├── embedding-client.ts # [1.2.1] Azure OpenAI embeddings
|
||||||
|
│ │ └── pii-redaction.ts # Reuse existing
|
||||||
|
│ └── server.ts # [3.3] Register routes
|
||||||
|
dashboards/admin-web/
|
||||||
|
├── src/
|
||||||
|
│ ├── app/(dashboard)/
|
||||||
|
│ │ ├── ai-diagnostics/
|
||||||
|
│ │ │ ├── page.tsx # [4.1] Main insights page
|
||||||
|
│ │ │ └── [id]/
|
||||||
|
│ │ │ └── page.tsx # [4.2] Cluster detail
|
||||||
|
│ ├── lib/
|
||||||
|
│ │ └── ai-diagnostics-client.ts # API client
|
||||||
|
│ └── components/
|
||||||
|
│ └── ai-diagnostics/ # Reusable components
|
||||||
|
```
|
||||||
|
|
||||||
|
**Commit Message Format:**
|
||||||
|
|
||||||
|
```
|
||||||
|
feat(ai-diagnostics): <description> [<task.code>]
|
||||||
|
```
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add services/platform-service/src/modules/ai-diagnostics/
|
||||||
|
git commit -m "feat(ai-diagnostics): add error clustering types and cosmos containers [1.1.1-1.1.2]"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Testing Requirements:**
|
||||||
|
|
||||||
|
- Unit tests: 20+ Vitest tests for clustering, embeddings, LLM responses
|
||||||
|
- Integration tests: End-to-end query → analysis pipeline
|
||||||
|
|
||||||
|
**Dependencies:**
|
||||||
|
|
||||||
|
- Telemetry module (error events)
|
||||||
|
- Azure OpenAI (embeddings + GPT-4o)
|
||||||
|
- Existing diagnostics module (optional linking)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### ErrorClusterDoc
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
interface ErrorClusterDoc {
|
||||||
|
id: string; // ec_<uuid>
|
||||||
|
productId: string; // partition key
|
||||||
|
fingerprintHash: string; // SHA-256 of normalized error
|
||||||
|
|
||||||
|
// Cluster metadata
|
||||||
|
firstSeenAt: string; // ISO 8601
|
||||||
|
lastSeenAt: string;
|
||||||
|
occurrenceCount: number; // Total occurrences
|
||||||
|
uniqueUsers: number; // Affected user count
|
||||||
|
|
||||||
|
// Error signature
|
||||||
|
errorType: string; // Exception class/name
|
||||||
|
messageTemplate: string; // Normalized message with placeholders
|
||||||
|
stackSignature: string; // Normalized stack frames
|
||||||
|
|
||||||
|
// Vector embedding for semantic search
|
||||||
|
embedding: number[]; // 1536-dim from text-embedding-3-small
|
||||||
|
embeddingVersion: string; // Model version for re-embedding
|
||||||
|
|
||||||
|
// Context patterns (auto-extracted)
|
||||||
|
commonContext: {
|
||||||
|
osVersions: Array<{ version: string; count: number }>;
|
||||||
|
appVersions: Array<{ version: string; count: number }>;
|
||||||
|
deviceModels: Array<{ model: string; count: number }>;
|
||||||
|
screenContexts: Array<{ screen: string; count: number }>;
|
||||||
|
};
|
||||||
|
|
||||||
|
// Related data
|
||||||
|
relatedClusterIds: string[]; // Similar clusters (vector similarity)
|
||||||
|
mergedIntoClusterId?: string; // If deduplicated
|
||||||
|
|
||||||
|
// Resolution tracking
|
||||||
|
status: 'active' | 'investigating' | 'resolved' | 'ignored';
|
||||||
|
resolvedAt?: string;
|
||||||
|
resolutionCommit?: string; // Link to fix
|
||||||
|
|
||||||
|
// Timestamps
|
||||||
|
createdAt: string;
|
||||||
|
updatedAt: string;
|
||||||
|
ttl: number; // 90 days
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### DiagnosticInsightDoc
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
interface DiagnosticInsightDoc {
|
||||||
|
id: string; // di_<uuid>
|
||||||
|
clusterId: string; // partition key (with productId)
|
||||||
|
productId: string;
|
||||||
|
|
||||||
|
// AI-generated analysis
|
||||||
|
analysisType: 'root_cause' | 'pattern' | 'comparison' | 'trend';
|
||||||
|
generatedAt: string;
|
||||||
|
|
||||||
|
// LLM output
|
||||||
|
rootCauseCategory: 'config' | 'dependency' | 'logic' | 'resource' | 'external' | 'unknown';
|
||||||
|
hypothesis: string; // Natural language explanation
|
||||||
|
reasoning: string; // Why LLM thinks this
|
||||||
|
confidence: 'high' | 'medium' | 'low';
|
||||||
|
confidenceScore: number; // 0.0–1.0
|
||||||
|
|
||||||
|
// Evidence
|
||||||
|
evidence: Array<{
|
||||||
|
type:
|
||||||
|
| 'stack_trace'
|
||||||
|
| 'telemetry_pattern'
|
||||||
|
| 'device_correlation'
|
||||||
|
| 'api_failure'
|
||||||
|
| 'similar_issue';
|
||||||
|
description: string;
|
||||||
|
strength: 'strong' | 'moderate' | 'weak';
|
||||||
|
data: Record<string, unknown>;
|
||||||
|
}>;
|
||||||
|
|
||||||
|
// Suggested actions
|
||||||
|
suggestedInvestigation: string[];
|
||||||
|
potentialFixDirection?: string;
|
||||||
|
similarResolvedIssues?: Array<{
|
||||||
|
clusterId: string;
|
||||||
|
resolution: string;
|
||||||
|
confidence: number;
|
||||||
|
}>;
|
||||||
|
|
||||||
|
// Feedback
|
||||||
|
feedbackStats: {
|
||||||
|
helpful: number;
|
||||||
|
notHelpful: number;
|
||||||
|
engineerNotes: string[];
|
||||||
|
};
|
||||||
|
|
||||||
|
// LLM metadata
|
||||||
|
modelUsed: string; // gpt-4o, gpt-4o-mini
|
||||||
|
promptTokens: number;
|
||||||
|
completionTokens: number;
|
||||||
|
|
||||||
|
createdAt: string;
|
||||||
|
ttl: number; // 90 days
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### NaturalLanguageQueryDoc
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
interface NaturalLanguageQueryDoc {
|
||||||
|
id: string; // nq_<uuid>
|
||||||
|
userId: string; // Admin who asked
|
||||||
|
productId?: string; // Optional filter
|
||||||
|
|
||||||
|
// Query
|
||||||
|
rawQuery: string; // "Why did iOS keyboard crash yesterday?"
|
||||||
|
parsedIntent: 'root_cause' | 'pattern_search' | 'comparison' | 'trend' | 'impact';
|
||||||
|
extractedEntities: {
|
||||||
|
products?: string[];
|
||||||
|
timeRange?: { start: string; end: string };
|
||||||
|
errorTypes?: string[];
|
||||||
|
platforms?: string[];
|
||||||
|
userSegments?: string[];
|
||||||
|
};
|
||||||
|
|
||||||
|
// Execution
|
||||||
|
executedQuery: string; // Translated Cosmos query
|
||||||
|
dataSources: string[]; // Clusters, telemetry, sessions accessed
|
||||||
|
executionTimeMs: number;
|
||||||
|
|
||||||
|
// Response
|
||||||
|
aiResponse: string; // Generated answer
|
||||||
|
confidence: number; // Overall confidence
|
||||||
|
supportingData: Array<{
|
||||||
|
type: 'cluster' | 'telemetry' | 'session';
|
||||||
|
id: string;
|
||||||
|
relevanceScore: number;
|
||||||
|
}>;
|
||||||
|
|
||||||
|
// Feedback
|
||||||
|
userRating?: 'helpful' | 'not_helpful';
|
||||||
|
userComment?: string;
|
||||||
|
|
||||||
|
createdAt: string;
|
||||||
|
ttl: number; // 30 days
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Appendix B: API Reference
|
||||||
|
|
||||||
|
| Method | Endpoint | Auth | Description |
|
||||||
|
| ------ | --------------------------------------- | ----- | --------------------------------------- |
|
||||||
|
| POST | `/ai-diagnostics/query` | Admin | Natural language diagnostic query |
|
||||||
|
| GET | `/ai-diagnostics/clusters` | Admin | List error clusters (with AI summaries) |
|
||||||
|
| GET | `/ai-diagnostics/clusters/:id` | Admin | Cluster detail with AI analysis |
|
||||||
|
| POST | `/ai-diagnostics/clusters/:id/analyze` | Admin | Trigger fresh LLM analysis |
|
||||||
|
| GET | `/ai-diagnostics/clusters/:id/analysis` | Admin | Get pre-computed insight |
|
||||||
|
| GET | `/ai-diagnostics/suggestions` | Admin | AI-suggested investigations |
|
||||||
|
| POST | `/ai-diagnostics/feedback` | Admin | Rate insight helpfulness |
|
||||||
|
| POST | `/ai-diagnostics/search` | Admin | Semantic search across errors |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Appendix C: Integration Points
|
||||||
|
|
||||||
|
### With Telemetry Module
|
||||||
|
|
||||||
|
- Error events auto-create/update clusters
|
||||||
|
- Telemetry context enriches error analysis
|
||||||
|
- Correlation IDs link errors to user journeys
|
||||||
|
|
||||||
|
### With Diagnostics Module
|
||||||
|
|
||||||
|
- Debug sessions linked to error clusters
|
||||||
|
- Screenshots from sessions aid visual analysis
|
||||||
|
- Network traces provide API failure context
|
||||||
|
|
||||||
|
### With Event Bus
|
||||||
|
|
||||||
|
| Event | Action |
|
||||||
|
| ------------------------------- | --------------------------------------------------------- |
|
||||||
|
| `telemetry.error.ingested` | Update/create cluster, trigger re-analysis if new pattern |
|
||||||
|
| `diagnostics.session.completed` | Link session to related clusters, analyze captured logs |
|
||||||
|
| `diagnostics.ingest.fatal` | High-priority cluster analysis, alert if novel pattern |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Appendix D: Cost Estimation
|
||||||
|
|
||||||
|
| Component | Monthly Cost (est.) |
|
||||||
|
| ------------------------ | ------------------------------- |
|
||||||
|
| Azure OpenAI embeddings | $50–100 (10K errors/day) |
|
||||||
|
| GPT-4o-mini analysis | $100–200 (1K analyses/day) |
|
||||||
|
| GPT-4o deep analysis | $50–100 (100 deep analyses/day) |
|
||||||
|
| Cosmos DB vector storage | $20–50 |
|
||||||
|
| **Total** | **$220–450/month** |
|
||||||
|
|
||||||
|
Optimization:
|
||||||
|
|
||||||
|
- Cache frequent cluster analyses (24hr TTL)
|
||||||
|
- Use GPT-4o-mini for 90% of queries
|
||||||
|
- Batch embedding jobs during off-peak
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Current Status
|
||||||
|
|
||||||
|
- [ ] **Design complete** — Target: 2026-03-10
|
||||||
|
- [ ] **Phase 1: Data Pipeline** — Not started
|
||||||
|
- [ ] **Phase 2: LLM Engine** — Not started
|
||||||
|
- [ ] **Phase 3: Query Interface** — Not started
|
||||||
|
- [ ] **Phase 4: Admin UI** — Not started
|
||||||
|
- [ ] **Phase 5: Advanced Capabilities** — Future
|
||||||
|
|
||||||
|
**Estimated Timeline:** 2–3 weeks (Phases 1–4)
|
||||||
|
|
||||||
|
**Dependencies:**
|
||||||
|
|
||||||
|
- Telemetry module (must be collecting errors)
|
||||||
|
- Diagnostics module (optional, for rich context)
|
||||||
|
- Azure OpenAI deployment (embedding + GPT-4o access)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
_Last Updated: 2026-03-03_
|
||||||
719
docs/roadmaps/INTELLIGENT_AB_TESTING_ROADMAP.md
Normal file
719
docs/roadmaps/INTELLIGENT_AB_TESTING_ROADMAP.md
Normal file
@ -0,0 +1,719 @@
|
|||||||
|
# Intelligent A/B Testing — Implementation Roadmap
|
||||||
|
|
||||||
|
> **Module:** `platform-service/src/modules/ab-testing/`
|
||||||
|
> **Admin UI:** `/ops/experiments/`
|
||||||
|
> **Target:** AI-powered experiment management with auto-allocation, early stopping, and hypothesis generation
|
||||||
|
> **Estimated Effort:** 2.5–3 weeks
|
||||||
|
> **Status:** 🟡 Planning
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Executive Summary
|
||||||
|
|
||||||
|
This roadmap delivers an **intelligent A/B testing platform** that goes beyond traditional feature flags. Unlike manual percentage rollouts, this system uses statistical algorithms for ** Thompson sampling**-based auto-allocation, **Bayesian early stopping** when variants clearly win/lose, and **LLM-powered hypothesis generation** from feature flag usage patterns.
|
||||||
|
|
||||||
|
### Key Differentiators vs. Static Feature Flags
|
||||||
|
|
||||||
|
| Capability | Static Flags (Current) | Intelligent A/B Testing |
|
||||||
|
| ------------------ | ---------------------- | ----------------------------------------- |
|
||||||
|
| Traffic Allocation | Manual percentage | **Multi-armed bandit optimization** |
|
||||||
|
| Stopping Decision | Manual monitoring | **Auto-stop at statistical significance** |
|
||||||
|
| Winner Selection | Human judgment | **Bayesian probability of superiority** |
|
||||||
|
| Test Duration | Fixed (often wrong) | **Dynamic based on effect size** |
|
||||||
|
| Hypothesis | Human-written | **AI-generated from usage patterns** |
|
||||||
|
| Sample Size | Guesswork | **Power analysis + sequential testing** |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 1: Core Experiment Engine (Week 1)
|
||||||
|
|
||||||
|
### 1.1 Data Model & Schemas
|
||||||
|
|
||||||
|
- [ ] **1.1.1** Create `modules/ab-testing/types.ts`
|
||||||
|
- [ ] `ExperimentDoc` — experiment definition and config
|
||||||
|
- [ ] `VariantDoc` — variant metadata + metrics
|
||||||
|
- [ ] `AssignmentDoc` — user → variant assignments
|
||||||
|
- [ ] `MetricDoc` — event types being tracked
|
||||||
|
- [ ] `ExperimentResult` — statistical analysis results
|
||||||
|
- [ ] Zod schemas for all inputs
|
||||||
|
- [ ] **1.1.2** Add Cosmos containers to `cosmos-init.ts`
|
||||||
|
- [ ] `experiments` (pk: `/productId`, TTL: 2 years for completed)
|
||||||
|
- [ ] `experiment_variants` (pk: `/experimentId`)
|
||||||
|
- [ ] `experiment_assignments` (pk: `/userId`, query by experiment)
|
||||||
|
- [ ] `experiment_events` (pk: `/experimentId` + `/timestamp` for time-series)
|
||||||
|
- [ ] `experiment_metrics` (pk: `/experimentId`, computed aggregates)
|
||||||
|
|
||||||
|
### 1.2 Assignment & Bucketing
|
||||||
|
|
||||||
|
- [ ] **1.2.1** Create deterministic bucketing
|
||||||
|
- [ ] Consistent hashing (userId + experimentId → variant)
|
||||||
|
- [ ] FNV-1a hash algorithm (same as feature flags)
|
||||||
|
- [ ] Sticky assignments (user always sees same variant)
|
||||||
|
- [ ] Override capability (force specific variant for QA)
|
||||||
|
- [ ] **1.2.2** Assignment strategies
|
||||||
|
- [ ] `random` — Simple randomization (control vs static)
|
||||||
|
- [ ] `thompson` — Thompson sampling (multi-armed bandit)
|
||||||
|
- [ ] `epsilon_greedy` — Epsilon-greedy exploration
|
||||||
|
- [ ] `ucb` — Upper Confidence Bound algorithm
|
||||||
|
- [ ] **1.2.3** Audience targeting
|
||||||
|
- [ ] User property filters (platform, version, region, subscription tier)
|
||||||
|
- [ ] Percentage rollout within target segment
|
||||||
|
- [ ] Exclusion lists (beta users, internal accounts)
|
||||||
|
|
||||||
|
### 1.3 Event Tracking Pipeline
|
||||||
|
|
||||||
|
- [ ] **1.3.1** Metric definitions
|
||||||
|
- [ ] `conversion` — Binary (did/didn't convert)
|
||||||
|
- [ ] `count` — Integer events (sessions, messages)
|
||||||
|
- [ ] `duration` — Time-based (session length, task time)
|
||||||
|
- [ ] `revenue` — Monetary (purchase amount, LTV)
|
||||||
|
- [ ] `custom` — Arbitrary numeric values
|
||||||
|
- [ ] **1.3.2** Event ingestion
|
||||||
|
- [ ] `POST /ab-testing/events` batch endpoint
|
||||||
|
- [ ] Client SDK: `track(event, value, metadata)`
|
||||||
|
- [ ] Automatic attribution (which variant caused this event)
|
||||||
|
- [ ] Deduplication (eventId + userId uniqueness)
|
||||||
|
|
||||||
|
**Phase 1 Exit Criteria:**
|
||||||
|
|
||||||
|
- [ ] Experiments created with multiple variants
|
||||||
|
- [ ] Users consistently assigned to variants
|
||||||
|
- [ ] Events tracked and attributed correctly
|
||||||
|
- [ ] 20+ tests for assignment and ingestion
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 2: Statistical Analysis Engine (Week 1–2)
|
||||||
|
|
||||||
|
### 2.1 Bayesian Inference
|
||||||
|
|
||||||
|
- [ ] **2.1.1** Create `modules/ab-testing/statistics.ts`
|
||||||
|
- [ ] `BetaDistribution` for conversion rates
|
||||||
|
- [ ] `GammaDistribution` for count/duration metrics
|
||||||
|
- [ ] `NormalDistribution` for continuous metrics
|
||||||
|
- [ ] Monte Carlo simulation (10,000 samples)
|
||||||
|
- [ ] **2.1.2** Probability calculations
|
||||||
|
- [ ] `probabilityVariantBeatsControl(variant, control)`
|
||||||
|
- [ ] `expectedLossIfChosen(variant)`
|
||||||
|
- [ ] `probabilityBeatAllVariants(variant)`
|
||||||
|
- [ ] **2.1.3** Credible intervals
|
||||||
|
- [ ] 95% credible interval for each variant's true metric
|
||||||
|
- [ ] Visualization-ready (lower, mean, upper bounds)
|
||||||
|
|
||||||
|
### 2.2 Early Stopping Rules
|
||||||
|
|
||||||
|
- [ ] **2.2.1** Stopping criteria
|
||||||
|
- [ ] **Winner found:** Variant has > 95% probability of beating control
|
||||||
|
- [ ] **Loser clear:** Control has > 95% probability of beating variant
|
||||||
|
- [ ] **Practical significance:** Minimum detectable effect not reached
|
||||||
|
- [ ] **Time bound:** Max duration reached (safety limit)
|
||||||
|
- [ ] **2.2.2** Auto-promotion
|
||||||
|
- [ ] Auto-rollout winner to 100% when threshold hit
|
||||||
|
- [ ] Notify admins via Slack/email
|
||||||
|
- [ ] Create audit log entry
|
||||||
|
- [ ] **2.2.3** Guardrails
|
||||||
|
- [ ] Minimum sample size before early stopping (100 users/variant)
|
||||||
|
- [ ] Business hours only for auto-actions
|
||||||
|
- [ ] Require approval for revenue-impacting experiments
|
||||||
|
|
||||||
|
### 2.3 Thompson Sampling
|
||||||
|
|
||||||
|
- [ ] **2.3.1** Multi-armed bandit implementation
|
||||||
|
- [ ] Sample from posterior distributions
|
||||||
|
- [ ] Assign user to variant with highest sample
|
||||||
|
- [ ] Re-balance traffic every hour based on performance
|
||||||
|
- [ ] **2.3.2** Exploration vs exploitation
|
||||||
|
- [ ] Exploration rate decays over time
|
||||||
|
- [ ] High uncertainty = more exploration
|
||||||
|
- [ ] Clear winner = more traffic to winner
|
||||||
|
- [ ] **2.3.3** Regret minimization
|
||||||
|
- [ ] Track cumulative regret vs optimal variant
|
||||||
|
- [ ] Regret bounds reporting
|
||||||
|
|
||||||
|
**Phase 2 Exit Criteria:**
|
||||||
|
|
||||||
|
- [ ] Bayesian probabilities calculated correctly
|
||||||
|
- [ ] Early stopping triggers at appropriate thresholds
|
||||||
|
- [ ] Thompson sampling re-allocates traffic dynamically
|
||||||
|
- [ ] Statistical tests validate correctness
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 3: AI-Powered Hypothesis Generation (Week 2)
|
||||||
|
|
||||||
|
### 3.1 Pattern Detection
|
||||||
|
|
||||||
|
- [ ] **3.1.1** Usage pattern analysis
|
||||||
|
- [ ] Analyze feature flag usage telemetry
|
||||||
|
- [ ] Segment analysis (iOS vs Android, free vs pro)
|
||||||
|
- [ ] Temporal patterns (day of week, time of day)
|
||||||
|
- [ ] User behavior sequences (funnel analysis)
|
||||||
|
- [ ] **3.1.2** Anomaly detection
|
||||||
|
- [ ] Unexpected drop in feature adoption
|
||||||
|
- [ ] Performance regression signals
|
||||||
|
- [ ] User segment showing different behavior
|
||||||
|
- [ ] **3.1.3** Opportunity identification
|
||||||
|
- [ ] Underperforming features (low adoption)
|
||||||
|
- [ ] High-dropoff flows
|
||||||
|
- [ ] Competitor feature gaps
|
||||||
|
|
||||||
|
### 3.2 Hypothesis Generation
|
||||||
|
|
||||||
|
- [ ] **3.2.1** LLM hypothesis prompts
|
||||||
|
|
||||||
|
```
|
||||||
|
Given this feature usage data:
|
||||||
|
- Feature: {featureName}
|
||||||
|
- Current adoption: {adoptionRate}% (baseline: {baseline}%)
|
||||||
|
- Segment performance: {segmentData}
|
||||||
|
- User feedback: {feedbackSamples}
|
||||||
|
- Competitor analysis: {competitorFeatures}
|
||||||
|
|
||||||
|
Generate experiment hypotheses:
|
||||||
|
1. Primary hypothesis: "Changing X will improve Y because..."
|
||||||
|
2. Secondary hypotheses (2-3 alternatives)
|
||||||
|
3. Expected effect size (conservative estimate)
|
||||||
|
4. Success metric recommendation
|
||||||
|
5. Risk assessment
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **3.2.2** Hypothesis ranking
|
||||||
|
- [ ] Expected impact scoring
|
||||||
|
- [ ] Implementation difficulty estimate
|
||||||
|
- [ ] Statistical power prediction
|
||||||
|
- [ ] Risk-adjusted expected value
|
||||||
|
- [ ] **3.2.3** Suggested experiment design
|
||||||
|
- [ ] Variant count recommendation
|
||||||
|
- [ ] Traffic allocation suggestion
|
||||||
|
- [ ] Duration estimate
|
||||||
|
- [ ] Required sample size calculation
|
||||||
|
|
||||||
|
### 3.3 Auto-Experiment Suggestions
|
||||||
|
|
||||||
|
- [ ] **3.3.1** Weekly AI reports
|
||||||
|
- [ ] Top 5 experiment opportunities
|
||||||
|
- [ ] Hypotheses with supporting evidence
|
||||||
|
- [ ] Prioritized by expected impact
|
||||||
|
- [ ] **3.3.2** One-click experiment creation
|
||||||
|
- [ ] Pre-fill experiment from hypothesis
|
||||||
|
- [ ] Suggested variants with descriptions
|
||||||
|
- [ ] Pre-configured metrics
|
||||||
|
|
||||||
|
**Phase 3 Exit Criteria:**
|
||||||
|
|
||||||
|
- [ ] AI generates meaningful hypotheses from usage data
|
||||||
|
- [ ] Hypothesis quality rated by product team (80%+ useful)
|
||||||
|
- [ ] Auto-suggested experiments created in 1 click
|
||||||
|
- [ ] Weekly reports generated automatically
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 4: Admin Dashboard UI (Week 2–3)
|
||||||
|
|
||||||
|
### 4.1 Experiments List Page
|
||||||
|
|
||||||
|
- [ ] **4.1.1** Create `/ops/experiments/page.tsx`
|
||||||
|
- [ ] Experiment cards (status, duration, sample size)
|
||||||
|
- [ ] Quick filters (running, completed, draft)
|
||||||
|
- [ ] AI-generated hypothesis badge
|
||||||
|
- [ ] Health indicators (traffic balance, event flow)
|
||||||
|
- [ ] **4.1.2** Experiment creation wizard
|
||||||
|
- [ ] Step 1: Define hypothesis (AI suggestions available)
|
||||||
|
- [ ] Step 2: Create variants (name, description, config)
|
||||||
|
- [ ] Step 3: Select metrics (primary + secondary)
|
||||||
|
- [ ] Step 4: Audience targeting
|
||||||
|
- [ ] Step 5: Traffic allocation (manual or Thompson)
|
||||||
|
- [ ] Step 6: Review and launch
|
||||||
|
|
||||||
|
### 4.2 Live Experiment Dashboard
|
||||||
|
|
||||||
|
- [ ] **4.2.1** Create `/ops/experiments/[id]/page.tsx`
|
||||||
|
- [ ] Real-time metrics comparison
|
||||||
|
- [ ] Variant performance table (conversions, counts, durations)
|
||||||
|
- [ ] Bayesian probability visualization
|
||||||
|
- [ ] Credible interval charts
|
||||||
|
- [ ] **4.2.2** Statistical summary card
|
||||||
|
- [ ] Probability of beating control (per variant)
|
||||||
|
- [ ] Expected lift if implemented
|
||||||
|
- [ ] Sample size progress bar
|
||||||
|
- [ ] Days to significance estimate
|
||||||
|
- [ ] **4.2.3** Action buttons
|
||||||
|
- [ ] Adjust traffic allocation
|
||||||
|
- [ ] Pause/resume experiment
|
||||||
|
- [ ] Stop and declare winner
|
||||||
|
- [ ] Rollout winner to 100%
|
||||||
|
- [ ] Archive experiment
|
||||||
|
|
||||||
|
### 4.3 Results & Reporting
|
||||||
|
|
||||||
|
- [ ] **4.3.1** Results page
|
||||||
|
- [ ] Final statistical summary
|
||||||
|
- [ ] Variant comparison visualization
|
||||||
|
- [ ] Segment breakdown (iOS vs Android, etc.)
|
||||||
|
- [ ] Confidence intervals over time
|
||||||
|
- [ ] **4.3.2** AI insights panel
|
||||||
|
- [ ] Why this result occurred (LLM summary)
|
||||||
|
- [ ] Unexpected findings
|
||||||
|
- [ ] Follow-up experiment suggestions
|
||||||
|
- [ ] **4.3.3** Export capabilities
|
||||||
|
- [ ] CSV export of raw data
|
||||||
|
- [ ] PDF report generation
|
||||||
|
- [ ] API endpoint for data warehouse sync
|
||||||
|
|
||||||
|
**Phase 4 Exit Criteria:**
|
||||||
|
|
||||||
|
- [ ] Full experiment lifecycle manageable in UI
|
||||||
|
- [ ] Real-time stats visible and accurate
|
||||||
|
- [ ] Bayesian visualizations clear to non-statisticians
|
||||||
|
- [ ] Export and reporting functional
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 5: Advanced Capabilities (Future)
|
||||||
|
|
||||||
|
### 5.1 Multi-Variate Testing
|
||||||
|
|
||||||
|
- [ ] Test multiple variables simultaneously
|
||||||
|
- [ ] Full factorial and fractional factorial designs
|
||||||
|
- [ ] Interaction effect detection
|
||||||
|
|
||||||
|
### 5.2 Sequential Experimentation
|
||||||
|
|
||||||
|
- [ ] Multi-phase experiments (qualification → main → validation)
|
||||||
|
- [ ] Holdout groups for long-term validation
|
||||||
|
- [ ] Global holdout (never-exposed users)
|
||||||
|
|
||||||
|
### 5.3 Personalization Layer
|
||||||
|
|
||||||
|
- [ ] Contextual bandits (different variants for different users)
|
||||||
|
- [ ] ML model for variant selection
|
||||||
|
- [ ] Automatic personalization optimization
|
||||||
|
|
||||||
|
### 5.4 Experiment Coordination
|
||||||
|
|
||||||
|
- [ ] Mutually exclusive experiments
|
||||||
|
- [ ] Experiment priority rules
|
||||||
|
- [ ] Layered experimentation (orthogonal tests)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Appendix A: Data Models
|
||||||
|
|
||||||
|
### ExperimentDoc
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
interface ExperimentDoc {
|
||||||
|
id: string; // exp_<uuid>
|
||||||
|
productId: string; // partition key
|
||||||
|
|
||||||
|
// Experiment definition
|
||||||
|
name: string;
|
||||||
|
description: string;
|
||||||
|
hypothesis: string;
|
||||||
|
aiGeneratedHypothesis?: boolean; // Flag for AI-suggested
|
||||||
|
|
||||||
|
// Status lifecycle: draft → running → paused | stopped | completed
|
||||||
|
status: 'draft' | 'running' | 'paused' | 'stopped' | 'completed';
|
||||||
|
|
||||||
|
// Variants
|
||||||
|
controlVariantId: string; // Baseline variant
|
||||||
|
variantIds: string[]; // All variant IDs
|
||||||
|
|
||||||
|
// Configuration
|
||||||
|
allocationStrategy: 'random' | 'thompson' | 'epsilon_greedy' | 'ucb';
|
||||||
|
targetPercent: number; // % of eligible traffic
|
||||||
|
|
||||||
|
// Audience targeting
|
||||||
|
targeting: {
|
||||||
|
platforms?: string[]; // ios, android, web
|
||||||
|
appVersions?: { min: string; max?: string };
|
||||||
|
regions?: string[];
|
||||||
|
userSegments?: string[]; // pro, free, enterprise
|
||||||
|
userProperties?: Record<string, string | number | boolean>;
|
||||||
|
};
|
||||||
|
|
||||||
|
// Metrics
|
||||||
|
primaryMetric: {
|
||||||
|
name: string;
|
||||||
|
type: 'conversion' | 'count' | 'duration' | 'revenue' | 'custom';
|
||||||
|
eventName: string; // Telemetry event to track
|
||||||
|
aggregation: 'sum' | 'mean' | 'count' | 'unique';
|
||||||
|
direction: 'increase' | 'decrease'; // Is higher better?
|
||||||
|
minimumDetectableEffect: number; // % change we want to detect
|
||||||
|
};
|
||||||
|
secondaryMetrics: Array<{
|
||||||
|
name: string;
|
||||||
|
type: 'conversion' | 'count' | 'duration' | 'revenue' | 'custom';
|
||||||
|
eventName: string;
|
||||||
|
}>;
|
||||||
|
|
||||||
|
// Guardrails
|
||||||
|
guardrails: {
|
||||||
|
minSampleSizePerVariant: number; // Default: 100
|
||||||
|
maxDurationDays: number; // Safety limit, default: 30
|
||||||
|
autoStopEnabled: boolean;
|
||||||
|
winnerThreshold: number; // % probability to auto-stop, default: 95
|
||||||
|
requireApprovalFor: 'none' | 'revenue' | 'all';
|
||||||
|
};
|
||||||
|
|
||||||
|
// Scheduling
|
||||||
|
startAt?: string; // Scheduled start (ISO 8601)
|
||||||
|
endAt?: string; // Scheduled end or actual stop
|
||||||
|
|
||||||
|
// Stats (denormalized for fast reads)
|
||||||
|
totalParticipants: number;
|
||||||
|
totalEvents: number;
|
||||||
|
|
||||||
|
// Timestamps
|
||||||
|
createdAt: string;
|
||||||
|
updatedAt: string;
|
||||||
|
startedAt?: string;
|
||||||
|
completedAt?: string;
|
||||||
|
ttl: number; // 2 years for completed
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### VariantDoc
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
interface VariantDoc {
|
||||||
|
id: string; // var_<uuid>
|
||||||
|
experimentId: string; // partition key
|
||||||
|
|
||||||
|
// Variant definition
|
||||||
|
name: string; // "Control", "New Button Color", etc.
|
||||||
|
description?: string;
|
||||||
|
isControl: boolean;
|
||||||
|
|
||||||
|
// Feature flag configuration
|
||||||
|
flagConfig: Record<string, unknown>; // Arbitrary config payload
|
||||||
|
|
||||||
|
// Traffic allocation (dynamic for bandit strategies)
|
||||||
|
currentAllocationPercent: number; // 0–100%
|
||||||
|
|
||||||
|
// Statistics (real-time computed)
|
||||||
|
stats: {
|
||||||
|
participants: number;
|
||||||
|
events: number;
|
||||||
|
|
||||||
|
// Primary metric
|
||||||
|
primaryMetricValue: number; // Mean or conversion rate
|
||||||
|
primaryMetricStdDev?: number;
|
||||||
|
|
||||||
|
// For conversion metrics
|
||||||
|
conversions?: number;
|
||||||
|
conversionRate?: number; // 0–1
|
||||||
|
|
||||||
|
// Bayesian posterior parameters
|
||||||
|
betaAlpha?: number; // For Beta distribution
|
||||||
|
betaBeta?: number;
|
||||||
|
|
||||||
|
gammaShape?: number; // For Gamma distribution
|
||||||
|
gammaScale?: number;
|
||||||
|
};
|
||||||
|
|
||||||
|
// Bayesian results
|
||||||
|
bayesianResults?: {
|
||||||
|
probabilityBeatsControl: number; // 0–1
|
||||||
|
probabilityBeatsAll: number; // 0–1
|
||||||
|
expectedLiftPercent: number; // Relative to control
|
||||||
|
expectedLoss: number; // Risk of choosing this variant
|
||||||
|
credibleInterval: {
|
||||||
|
lower: number;
|
||||||
|
mean: number;
|
||||||
|
upper: number;
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
createdAt: string;
|
||||||
|
updatedAt: string;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### ExperimentAssignmentDoc
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
interface ExperimentAssignmentDoc {
|
||||||
|
id: string; // ea_<uuid>
|
||||||
|
userId: string; // partition key (for user lookups)
|
||||||
|
|
||||||
|
experimentId: string;
|
||||||
|
variantId: string;
|
||||||
|
|
||||||
|
// Assignment metadata
|
||||||
|
assignedAt: string; // First assignment
|
||||||
|
firstExposedAt?: string; // First actual exposure (feature use)
|
||||||
|
|
||||||
|
// Context at assignment
|
||||||
|
assignmentContext: {
|
||||||
|
platform: string;
|
||||||
|
appVersion: string;
|
||||||
|
osVersion: string;
|
||||||
|
deviceModel?: string;
|
||||||
|
region?: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
// Events attributed to this assignment
|
||||||
|
eventCount: number;
|
||||||
|
lastEventAt?: string;
|
||||||
|
|
||||||
|
// TTL: Remove after experiment completes + analysis period
|
||||||
|
ttl: number; // experimentEnd + 90 days
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### ExperimentEventDoc
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
interface ExperimentEventDoc {
|
||||||
|
id: string; // ee_<uuid>
|
||||||
|
experimentId: string; // partition key
|
||||||
|
timestamp: string; // Sort key for time-series queries
|
||||||
|
|
||||||
|
// Attribution
|
||||||
|
userId: string;
|
||||||
|
variantId: string;
|
||||||
|
assignmentId: string;
|
||||||
|
|
||||||
|
// Event details
|
||||||
|
metricName: string;
|
||||||
|
metricType: 'conversion' | 'count' | 'duration' | 'revenue' | 'custom';
|
||||||
|
value: number; // Numeric value
|
||||||
|
|
||||||
|
// Conversion tracking (for binary metrics)
|
||||||
|
converted: boolean; // For conversion metrics
|
||||||
|
|
||||||
|
// Context
|
||||||
|
eventMetadata?: Record<string, unknown>;
|
||||||
|
|
||||||
|
// Denormalized for filtering
|
||||||
|
platform: string;
|
||||||
|
appVersion: string;
|
||||||
|
|
||||||
|
// TTL: Shorter for raw events
|
||||||
|
ttl: number; // 90 days
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Implementation Tracking
|
||||||
|
|
||||||
|
| Phase | Task | Status | Commit |
|
||||||
|
| ----- | ----------------------------- | ------ | ------ |
|
||||||
|
| 1.1 | Experiment types & schemas | ⬜ | — |
|
||||||
|
| 1.1 | Cosmos containers | ⬜ | — |
|
||||||
|
| 1.2 | Deterministic bucketing | ⬜ | — |
|
||||||
|
| 1.2 | Assignment strategies | ⬜ | — |
|
||||||
|
| 1.2 | Audience targeting | ⬜ | — |
|
||||||
|
| 1.3 | Metric definitions | ⬜ | — |
|
||||||
|
| 1.3 | Event ingestion | ⬜ | — |
|
||||||
|
| 2.1 | Bayesian inference engine | ⬜ | — |
|
||||||
|
| 2.1 | Probability calculations | ⬜ | — |
|
||||||
|
| 2.1 | Credible intervals | ⬜ | — |
|
||||||
|
| 2.2 | Early stopping rules | ⬜ | — |
|
||||||
|
| 2.2 | Auto-promotion | ⬜ | — |
|
||||||
|
| 2.2 | Guardrails | ⬜ | — |
|
||||||
|
| 2.3 | Thompson sampling | ⬜ | — |
|
||||||
|
| 2.3 | Exploration vs exploitation | ⬜ | — |
|
||||||
|
| 2.3 | Regret minimization | ⬜ | — |
|
||||||
|
| 3.1 | Pattern detection | ⬜ | — |
|
||||||
|
| 3.1 | Anomaly detection | ⬜ | — |
|
||||||
|
| 3.2 | Hypothesis generation prompts | ⬜ | — |
|
||||||
|
| 3.2 | Hypothesis ranking | ⬜ | — |
|
||||||
|
| 3.3 | Auto-experiment suggestions | ⬜ | — |
|
||||||
|
| 4.1 | Experiments list page | ⬜ | — |
|
||||||
|
| 4.1 | Creation wizard | ⬜ | — |
|
||||||
|
| 4.2 | Live dashboard | ⬜ | — |
|
||||||
|
| 4.2 | Statistical summary | ⬜ | — |
|
||||||
|
| 4.3 | Results & reporting | ⬜ | — |
|
||||||
|
| 4.3 | AI insights panel | ⬜ | — |
|
||||||
|
|
||||||
|
**Legend:** ⬜ Not started | 🟡 In progress | ✅ Complete | ⏸️ Deferred
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Quick Reference for Implementing Agent
|
||||||
|
|
||||||
|
**📋 Full Roadmap:** `/Users/sd9235/code/mygh/learning_ai_common_plat/docs/roadmaps/INTELLIGENT_AB_TESTING_ROADMAP.md`
|
||||||
|
|
||||||
|
**Key Files to Modify/Create:**
|
||||||
|
|
||||||
|
```
|
||||||
|
services/platform-service/
|
||||||
|
├── src/
|
||||||
|
│ ├── modules/ab-testing/
|
||||||
|
│ │ ├── types.ts # [1.1] Experiment, Variant, Assignment types
|
||||||
|
│ │ ├── repository.ts # [1.2] Data access layer
|
||||||
|
│ │ ├── bucketing.ts # [1.2] FNV-1a hash, sticky assignments
|
||||||
|
│ │ ├── statistics.ts # [2.1] Bayesian inference, Beta/Normal distributions
|
||||||
|
│ │ ├── allocation.ts # [2.3] Thompson sampling, bandit strategies
|
||||||
|
│ │ ├── hypothesis-generator.ts # [3.2] LLM pattern analysis
|
||||||
|
│ │ ├── routes.ts # [4] REST API
|
||||||
|
│ │ └── ab-testing.test.ts # Tests
|
||||||
|
│ ├── lib/
|
||||||
|
│ │ └── cosmos-init.ts # [1.1] Add containers
|
||||||
|
│ └── server.ts # Register routes
|
||||||
|
dashboards/admin-web/
|
||||||
|
├── src/
|
||||||
|
│ ├── app/(dashboard)/
|
||||||
|
│ │ ├── experiments/
|
||||||
|
│ │ │ ├── page.tsx # [4.1] Experiments list
|
||||||
|
│ │ │ ├── new/page.tsx # [4.1] Creation wizard
|
||||||
|
│ │ │ └── [id]/
|
||||||
|
│ │ │ └── page.tsx # [4.2] Live dashboard
|
||||||
|
│ ├── lib/
|
||||||
|
│ │ └── experiments-client.ts # API client
|
||||||
|
│ └── components/
|
||||||
|
│ └── experiments/ # Bayesian charts, variant cards
|
||||||
|
```
|
||||||
|
|
||||||
|
**Commit Message Format:**
|
||||||
|
|
||||||
|
```
|
||||||
|
feat(ab-testing): <description> [<task.code>]
|
||||||
|
```
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add services/platform-service/src/modules/ab-testing/
|
||||||
|
git commit -m "feat(ab-testing): add experiment types and cosmos containers [1.1]"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Testing Requirements:**
|
||||||
|
|
||||||
|
- Unit tests: 25+ Vitest tests for bucketing, statistics, bandit algorithms
|
||||||
|
- Statistical validation: A/A tests, known distribution tests
|
||||||
|
- Integration: End-to-end experiment lifecycle
|
||||||
|
|
||||||
|
**Dependencies:**
|
||||||
|
|
||||||
|
- Feature flags module (reuse bucketing logic)
|
||||||
|
- Telemetry module (event tracking)
|
||||||
|
- Azure OpenAI (hypothesis generation)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Appendix B: Statistical Methods
|
||||||
|
|
||||||
|
### Bayesian A/B Testing
|
||||||
|
|
||||||
|
**Conversion Metrics (Beta-Binomial):**
|
||||||
|
|
||||||
|
```
|
||||||
|
Posterior: Beta(α + conversions, β + non-conversions)
|
||||||
|
Where α = β = 1 (uniform prior)
|
||||||
|
|
||||||
|
Probability variant beats control:
|
||||||
|
P(variant > control) = Σ(i=0 to n) [BetaCDF_control(i)] * [BetaPDF_variant(i)]
|
||||||
|
```
|
||||||
|
|
||||||
|
**Continuous Metrics (Normal):**
|
||||||
|
|
||||||
|
```
|
||||||
|
Posterior: Normal(μ_n, σ_n²)
|
||||||
|
Where μ_n, σ_n updated via conjugate prior
|
||||||
|
|
||||||
|
Probability variant beats control via Monte Carlo sampling
|
||||||
|
```
|
||||||
|
|
||||||
|
### Thompson Sampling
|
||||||
|
|
||||||
|
```
|
||||||
|
For each incoming user:
|
||||||
|
For each variant:
|
||||||
|
Sample θ_i from variant's posterior distribution
|
||||||
|
Assign user to variant with max(θ_i)
|
||||||
|
|
||||||
|
Update variant's posterior after observing outcome
|
||||||
|
```
|
||||||
|
|
||||||
|
### Early Stopping
|
||||||
|
|
||||||
|
```
|
||||||
|
Stop experiment when:
|
||||||
|
max_variant P(beats control) > 0.95 → Winner found
|
||||||
|
OR max_variant P(beats control) < 0.05 → No winner
|
||||||
|
OR days_running > max_duration
|
||||||
|
AND samples_per_variant > min_sample_size
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Appendix C: API Reference
|
||||||
|
|
||||||
|
| Method | Endpoint | Auth | Description |
|
||||||
|
| ------ | -------------------------------------- | -------- | -------------------------------- |
|
||||||
|
| POST | `/ab-testing/experiments` | Admin | Create experiment |
|
||||||
|
| GET | `/ab-testing/experiments` | Admin | List experiments |
|
||||||
|
| GET | `/ab-testing/experiments/:id` | Admin | Get experiment details |
|
||||||
|
| PATCH | `/ab-testing/experiments/:id` | Admin | Update experiment |
|
||||||
|
| DELETE | `/ab-testing/experiments/:id` | Admin | Stop/archive experiment |
|
||||||
|
| POST | `/ab-testing/experiments/:id/start` | Admin | Start experiment |
|
||||||
|
| POST | `/ab-testing/experiments/:id/pause` | Admin | Pause experiment |
|
||||||
|
| POST | `/ab-testing/experiments/:id/complete` | Admin | Complete with winner |
|
||||||
|
| POST | `/ab-testing/assign` | Any auth | Get variant assignment for user |
|
||||||
|
| POST | `/ab-testing/events` | Any auth | Track experiment event |
|
||||||
|
| GET | `/ab-testing/experiments/:id/results` | Admin | Get statistical results |
|
||||||
|
| GET | `/ab-testing/suggestions` | Admin | AI-generated experiment ideas |
|
||||||
|
| POST | `/ab-testing/hypotheses` | Admin | Generate hypothesis from pattern |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Appendix D: Integration Points
|
||||||
|
|
||||||
|
### With Feature Flags Module
|
||||||
|
|
||||||
|
- Experiments build on feature flag infrastructure
|
||||||
|
- Flag state = variant assignment
|
||||||
|
- Consistent bucketing with existing flags
|
||||||
|
|
||||||
|
### With Telemetry Module
|
||||||
|
|
||||||
|
- Experiment events enriched with telemetry context
|
||||||
|
- Automatic metric tracking from existing events
|
||||||
|
- Funnel analysis using telemetry breadcrumbs
|
||||||
|
|
||||||
|
### With Event Bus
|
||||||
|
|
||||||
|
| Event | Action |
|
||||||
|
| ----------------------------- | ----------------------------------- |
|
||||||
|
| `ab.experiment.started` | Notify stakeholders, log audit |
|
||||||
|
| `ab.experiment.completed` | Generate report, suggest follow-ups |
|
||||||
|
| `ab.variant.declared_winner` | Trigger auto-rollout if enabled |
|
||||||
|
| `ab.early_stopping.triggered` | Alert experiment owner |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Appendix E: Cost Estimation
|
||||||
|
|
||||||
|
| Component | Monthly Cost (est.) |
|
||||||
|
| ---------------------------- | ------------------------ |
|
||||||
|
| Cosmos DB (experiment data) | $100–200 |
|
||||||
|
| LLM hypothesis generation | $50–100 (weekly reports) |
|
||||||
|
| Compute (statistical engine) | $50 (negligible) |
|
||||||
|
| **Total** | **$200–350/month** |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Current Status
|
||||||
|
|
||||||
|
- [ ] **Design complete** — Target: 2026-03-10
|
||||||
|
- [ ] **Phase 1: Core Engine** — Not started
|
||||||
|
- [ ] **Phase 2: Statistics** — Not started
|
||||||
|
- [ ] **Phase 3: AI Hypotheses** — Not started
|
||||||
|
- [ ] **Phase 4: Admin UI** — Not started
|
||||||
|
- [ ] **Phase 5: Advanced** — Future
|
||||||
|
|
||||||
|
**Estimated Timeline:** 2.5–3 weeks (Phases 1–4)
|
||||||
|
|
||||||
|
**Dependencies:**
|
||||||
|
|
||||||
|
- Feature flags module (for assignment infrastructure)
|
||||||
|
- Telemetry module (for event tracking)
|
||||||
|
- Azure OpenAI (for hypothesis generation)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
_Last Updated: 2026-03-03_
|
||||||
848
docs/roadmaps/PREDICTIVE_CHURN_HEALTH_SCORING_ROADMAP.md
Normal file
848
docs/roadmaps/PREDICTIVE_CHURN_HEALTH_SCORING_ROADMAP.md
Normal file
@ -0,0 +1,848 @@
|
|||||||
|
# Predictive Churn & Health Scoring — Implementation Roadmap
|
||||||
|
|
||||||
|
> **Module:** `platform-service/src/modules/predictive-analytics/`
|
||||||
|
> **Admin UI:** `/ops/health-dashboard/`
|
||||||
|
> **Target:** ML-powered churn prediction, health scoring, and proactive retention
|
||||||
|
> **Estimated Effort:** 3 weeks
|
||||||
|
> **Status:** 🟡 Planning
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Executive Summary
|
||||||
|
|
||||||
|
This roadmap delivers a **predictive analytics platform** that forecasts user churn 7–30 days in advance and computes product health scores from telemetry. Unlike reactive dashboards that show what happened, this system **predicts what will happen**—enabling proactive retention campaigns, resource allocation, and product improvements before users leave.
|
||||||
|
|
||||||
|
### Key Differentiators vs. Reactive Analytics
|
||||||
|
|
||||||
|
| Capability | Traditional Analytics | Predictive Churn & Health |
|
||||||
|
| ------------------- | -------------------------- | --------------------------------- |
|
||||||
|
| Insight Type | Historical (what happened) | **Predictive (what will happen)** |
|
||||||
|
| Churn Detection | After user leaves | **7–30 days before churn** |
|
||||||
|
| Health View | Current snapshot only | **Trending + forecasted** |
|
||||||
|
| Interventions | Reactive recovery | **Proactive prevention** |
|
||||||
|
| Product Insights | Manual pattern search | **Auto-detected risk signals** |
|
||||||
|
| Resource Allocation | Guesswork | **Risk-weighted prioritization** |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 1: Feature Engineering Pipeline (Week 1)
|
||||||
|
|
||||||
|
### 1.1 Telemetry Feature Extraction
|
||||||
|
|
||||||
|
- [ ] **1.1.1** Create `modules/predictive-analytics/feature-extractor.ts`
|
||||||
|
- [ ] User behavior features (session frequency, depth, recency)
|
||||||
|
- [ ] Engagement features (feature usage diversity, core action completion)
|
||||||
|
- [ ] Performance features (error rate, latency exposure, crash frequency)
|
||||||
|
- [ ] Social features (sharing, collaboration, network effects)
|
||||||
|
- [ ] Revenue features (payment history, plan changes, support tickets)
|
||||||
|
- [ ] **1.1.2** Time-window aggregations
|
||||||
|
- [ ] Last 24 hours (recent behavior)
|
||||||
|
- [ ] Last 7 days (weekly patterns)
|
||||||
|
- [ ] Last 30 days (monthly trends)
|
||||||
|
- [ ] Life-to-date (all-time totals)
|
||||||
|
- [ ] **1.1.3** Rolling window features
|
||||||
|
- [ ] 7-day rolling average (trend smoothing)
|
||||||
|
- [ ] Week-over-week change (acceleration)
|
||||||
|
- [ ] Cohort-normalized scores (vs. similar users)
|
||||||
|
|
||||||
|
### 1.2 Feature Store
|
||||||
|
|
||||||
|
- [ ] **1.2.1** Create `modules/predictive-analytics/feature-store.ts`
|
||||||
|
- [ ] `UserFeatureVector` — normalized feature values per user
|
||||||
|
- [ ] `ProductHealthMetrics` — aggregated product-level scores
|
||||||
|
- [ ] Feature versioning (track feature schema changes)
|
||||||
|
- [ ] **1.2.2** Add Cosmos containers to `cosmos-init.ts`
|
||||||
|
- [ ] `user_features` (pk: `/userId`, TTL: 90 days)
|
||||||
|
- [ ] `product_health` (pk: `/productId` + `/date`, time-series)
|
||||||
|
- [ ] `feature_definitions` (pk: `/productId`, feature metadata)
|
||||||
|
- [ ] **1.2.3** Feature computation jobs
|
||||||
|
- [ ] Daily feature computation (nightly batch)
|
||||||
|
- [ ] Real-time feature updates (on key events)
|
||||||
|
- [ ] Feature backfill (compute historical features)
|
||||||
|
|
||||||
|
### 1.3 Product-Specific Feature Catalog
|
||||||
|
|
||||||
|
- [ ] **1.3.1** Define features per product
|
||||||
|
- [ ] **NomGap:** Fast completion rate, protocol adherence, streak length, autophagy engagement
|
||||||
|
- [ ] **JarvisJr:** Session frequency, agent diversity, voice/text ratio, skill progression
|
||||||
|
- [ ] **ChronoMind:** Timer completion rate, cascade effectiveness, routine adherence, urgency response
|
||||||
|
- [ ] **MindLyst:** Brain usage diversity, triage accuracy, memory capture frequency, reflection completion
|
||||||
|
- [ ] **PeakPulse:** Session frequency, goal completion, streak maintenance, social sharing
|
||||||
|
- [ ] **LysnrAI:** Dictation frequency, accuracy rate, hotkey usage, vocabulary growth
|
||||||
|
- [ ] **1.3.2** Feature importance tracking
|
||||||
|
- [ ] Which features correlate with churn/retention
|
||||||
|
- [ ] Feature drift detection (behavior changes over time)
|
||||||
|
- [ ] Auto-suggest new features based on patterns
|
||||||
|
|
||||||
|
**Phase 1 Exit Criteria:**
|
||||||
|
|
||||||
|
- [ ] 50+ features extracted per product
|
||||||
|
- [ ] Feature store populated for all active users
|
||||||
|
- [ ] Daily feature computation job running
|
||||||
|
- [ ] Feature importance analysis completed
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 2: Churn Prediction Model (Week 1–2)
|
||||||
|
|
||||||
|
### 2.1 Model Architecture
|
||||||
|
|
||||||
|
- [ ] **2.1.1** Create `modules/predictive-analytics/churn-model.ts`
|
||||||
|
- [ ] Binary classification (will churn in 7 days? 30 days?)
|
||||||
|
- [ ] Gradient Boosted Trees (XGBoost/LightGBM) baseline
|
||||||
|
- [ ] Neural network ensemble (for comparison)
|
||||||
|
- [ ] **2.1.2** Training pipeline
|
||||||
|
- [ ] Label definition: No activity for N days = churned
|
||||||
|
- [ ] Train/validation/test split (time-based, not random)
|
||||||
|
- [ ] Cross-validation with temporal folds
|
||||||
|
- [ ] Hyperparameter tuning (optuna/ray tune)
|
||||||
|
- [ ] **2.1.3** Model evaluation
|
||||||
|
- [ ] ROC-AUC (discrimination ability)
|
||||||
|
- [ ] Precision/Recall at different thresholds
|
||||||
|
- [ ] Calibration (predicted prob vs. actual rate)
|
||||||
|
- [ ] Per-product performance breakdown
|
||||||
|
|
||||||
|
### 2.2 Prediction Service
|
||||||
|
|
||||||
|
- [ ] **2.2.1** Real-time scoring API
|
||||||
|
- [ ] `POST /predictive/churn-score` — single user prediction
|
||||||
|
- [ ] `POST /predictive/churn-batch` — batch scoring
|
||||||
|
- [ ] Latency < 100ms for single prediction
|
||||||
|
- [ ] **2.2.2** Risk segmentation
|
||||||
|
- [ ] Risk buckets: Critical (>80%), High (60–80%), Medium (30–60%), Low (<30%)
|
||||||
|
- [ ] Risk score components (which features drive the score)
|
||||||
|
- [ ] Confidence intervals on predictions
|
||||||
|
- [ ] **2.2.3** Model versioning
|
||||||
|
- [ ] A/B test model versions
|
||||||
|
- [ ] Shadow mode (predict without acting)
|
||||||
|
- [ ] Rollback capability
|
||||||
|
|
||||||
|
### 2.3 Explanation Engine
|
||||||
|
|
||||||
|
- [ ] **2.3.1** SHAP value computation
|
||||||
|
- [ ] Feature contributions to each prediction
|
||||||
|
- [ ] Global feature importance (what drives churn overall)
|
||||||
|
- [ ] Local explanations (why this specific user is at risk)
|
||||||
|
- [ ] **2.3.2** Natural language explanations
|
||||||
|
```
|
||||||
|
"This user shows 78% churn risk because:
|
||||||
|
- Session frequency dropped 60% in the last week
|
||||||
|
- No core feature usage in 5 days
|
||||||
|
- Error rate increased 3x vs. their baseline
|
||||||
|
- Similar users who showed these patterns had 85% churn rate"
|
||||||
|
```
|
||||||
|
- [ ] **2.3.3** Actionable insight extraction
|
||||||
|
- [ ] Top 3 risk factors per user
|
||||||
|
- [ ] Suggested intervention based on risk profile
|
||||||
|
- [ ] Priority ranking (who to contact first)
|
||||||
|
|
||||||
|
**Phase 2 Exit Criteria:**
|
||||||
|
|
||||||
|
- [ ] Model achieves > 75% AUC on test set
|
||||||
|
- [ ] Real-time scoring API < 100ms latency
|
||||||
|
- [ ] Explanations generated for all predictions
|
||||||
|
- [ ] Risk segmentation validated against historical churn
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 3: Product Health Scoring (Week 2)
|
||||||
|
|
||||||
|
### 3.1 Health Metric Framework
|
||||||
|
|
||||||
|
- [ ] **3.1.1** Create `modules/predictive-analytics/health-scoring.ts`
|
||||||
|
- [ ] Health dimensions: Acquisition, Activation, Retention, Revenue, Engagement
|
||||||
|
- [ ] Composite health score (weighted average)
|
||||||
|
- [ ] Per-dimension scores with drill-down
|
||||||
|
- [ ] **3.1.2** Product health indicators
|
||||||
|
- [ ] Daily Active Users (DAU) trend
|
||||||
|
- [ ] New user activation rate (Day-1, Day-7)
|
||||||
|
- [ ] Cohort retention curves (Day-1, Day-7, Day-30)
|
||||||
|
- [ ] Feature adoption rates (new feature uptake)
|
||||||
|
- [ ] Error rates and stability scores
|
||||||
|
- [ ] Support ticket volume and sentiment
|
||||||
|
- [ ] Revenue metrics (MRR, ARPU, LTV)
|
||||||
|
|
||||||
|
### 3.2 Health Score Computation
|
||||||
|
|
||||||
|
- [ ] **3.2.1** Baseline establishment
|
||||||
|
- [ ] Historical 90-day baseline for each metric
|
||||||
|
- [ ] Peer product comparison (ChronoMind vs. JarvisJr benchmarks)
|
||||||
|
- [ ] Industry benchmarks (if available)
|
||||||
|
- [ ] **3.2.2** Scoring algorithm
|
||||||
|
- [ ] Z-score normalization (how many std devs from baseline)
|
||||||
|
- [ ] Trend direction (improving vs. declining)
|
||||||
|
- [ ] Volatility adjustment (consistent vs. erratic)
|
||||||
|
- [ ] 0–100 health score scale
|
||||||
|
- [ ] **3.2.3** Alert thresholds
|
||||||
|
- [ ] Critical: Score < 60 or 20% drop from baseline
|
||||||
|
- [ ] Warning: Score 60–75 or 10% drop
|
||||||
|
- [ ] Healthy: Score > 75 and stable
|
||||||
|
|
||||||
|
### 3.3 Anomaly Detection
|
||||||
|
|
||||||
|
- [ ] **3.3.1** Statistical anomaly detection
|
||||||
|
- [ ] Prophet/ARIMA for time-series forecasting
|
||||||
|
- [ ] Forecast vs. actual deviation detection
|
||||||
|
- [ ] Seasonal pattern recognition (day-of-week, monthly)
|
||||||
|
- [ ] **3.3.2** Multi-dimensional anomaly detection
|
||||||
|
- [ ] Correlation breakdown detection (metrics usually correlated diverging)
|
||||||
|
- [ ] Cohort-specific anomalies (specific region, platform, segment)
|
||||||
|
- [ ] **3.3.3** Root cause suggestion
|
||||||
|
- [ ] Correlation with deployments/releases
|
||||||
|
- [ ] Error spike correlation
|
||||||
|
- [ ] External factor detection (holidays, events)
|
||||||
|
|
||||||
|
**Phase 3 Exit Criteria:**
|
||||||
|
|
||||||
|
- [ ] Health scores computed daily for all products
|
||||||
|
- [ ] Anomaly detection with < 5% false positive rate
|
||||||
|
- [ ] Historical baseline established for all metrics
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 4: Proactive Intervention System (Week 2–3)
|
||||||
|
|
||||||
|
### 4.1 Retention Campaign Automation
|
||||||
|
|
||||||
|
- [ ] **4.1.1** Campaign trigger rules
|
||||||
|
- [ ] High-risk user enters segment → trigger email
|
||||||
|
- [ ] Medium-risk + specific behavior → trigger in-app message
|
||||||
|
- [ ] Critical risk → trigger personal outreach task
|
||||||
|
- [ ] **4.1.2** Personalized messaging
|
||||||
|
- [ ] Message variant based on risk factors
|
||||||
|
- [ ] Feature recommendations based on unused capabilities
|
||||||
|
- [ ] Success stories from similar users
|
||||||
|
- [ ] **4.1.3** Campaign effectiveness tracking
|
||||||
|
- [ ] Control group vs. treatment
|
||||||
|
- [ ] Churn rate comparison
|
||||||
|
- [ ] Revenue impact measurement
|
||||||
|
|
||||||
|
### 4.2 Auto-Trigger Flows
|
||||||
|
|
||||||
|
- [ ] **4.2.1** Platform integrations
|
||||||
|
- [ ] Email delivery via existing `modules/delivery/`
|
||||||
|
- [ ] Push notifications via `modules/notifications/`
|
||||||
|
- [ ] Slack notifications for CS team
|
||||||
|
- [ ] CRM integration (create outreach tasks)
|
||||||
|
- [ ] **4.2.2** Smart scheduling
|
||||||
|
- [ ] Optimal contact time prediction
|
||||||
|
- [ ] Frequency capping (don't spam)
|
||||||
|
- [ ] Multi-channel orchestration
|
||||||
|
- [ ] **4.2.3** Feedback loop
|
||||||
|
- [ ] Track intervention outcomes
|
||||||
|
- [ ] Retrain model with intervention effectiveness
|
||||||
|
- [ ] A/B test intervention strategies
|
||||||
|
|
||||||
|
### 4.3 Risk Dashboard for CS Team
|
||||||
|
|
||||||
|
- [ ] **4.3.1** At-risk user list
|
||||||
|
- [ ] Sortable by churn probability
|
||||||
|
- [ ] Filter by product, segment, risk factors
|
||||||
|
- [ ] Last activity preview
|
||||||
|
- [ ] **4.3.2** User risk profile
|
||||||
|
- [ ] Churn probability trend over time
|
||||||
|
- [ ] Key risk factors highlighted
|
||||||
|
- [ ] Recommended actions
|
||||||
|
- [ ] User activity timeline
|
||||||
|
- [ ] **4.3.3** Intervention tracking
|
||||||
|
- [ ] Contact history
|
||||||
|
- [ ] Response tracking
|
||||||
|
- [ ] Outcome recording (retained/churned)
|
||||||
|
|
||||||
|
**Phase 4 Exit Criteria:**
|
||||||
|
|
||||||
|
- [ ] Automated campaigns triggered for high-risk users
|
||||||
|
- [ ] CS team dashboard with at-risk user queue
|
||||||
|
- [ ] Intervention effectiveness measurement in place
|
||||||
|
- [ ] Feedback loop improving model accuracy
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 5: Admin Dashboard UI (Week 3)
|
||||||
|
|
||||||
|
### 5.1 Product Health Overview
|
||||||
|
|
||||||
|
- [ ] **5.1.1** Create `/ops/health-dashboard/page.tsx`
|
||||||
|
- [ ] Health score cards for each product
|
||||||
|
- [ ] Trend sparklines (7-day, 30-day)
|
||||||
|
- [ ] Alert summary (critical issues count)
|
||||||
|
- [ ] Product comparison table
|
||||||
|
- [ ] **5.1.2** Health detail view
|
||||||
|
- [ ] Dimension breakdown (acquisition, activation, retention, etc.)
|
||||||
|
- [ ] Metric time-series charts
|
||||||
|
- [ ] Anomaly markers on charts
|
||||||
|
- [ ] Cohort retention curves
|
||||||
|
|
||||||
|
### 5.2 Churn Prediction Dashboard
|
||||||
|
|
||||||
|
- [ ] **5.2.1** Churn risk overview
|
||||||
|
- [ ] Risk distribution pie chart
|
||||||
|
- [ ] At-risk user count by product
|
||||||
|
- [ ] Predicted churn impact (revenue at risk)
|
||||||
|
- [ ] Model performance metrics (AUC, calibration)
|
||||||
|
- [ ] **5.2.2** User risk explorer
|
||||||
|
- [ ] Search/filter at-risk users
|
||||||
|
- [ ] Risk score with explanation
|
||||||
|
- [ ] Top risk factors
|
||||||
|
- [ ] Recommended interventions
|
||||||
|
- [ ] **5.2.3** Model insights
|
||||||
|
- [ ] Global feature importance chart
|
||||||
|
- [ ] Model performance over time
|
||||||
|
- [ ] Feature drift alerts
|
||||||
|
|
||||||
|
### 5.3 Campaign Management
|
||||||
|
|
||||||
|
- [ ] **5.3.1** Campaign list
|
||||||
|
- [ ] Active/paused/completed campaigns
|
||||||
|
- [ ] Trigger rules summary
|
||||||
|
- [ ] Performance stats (sent, opened, converted)
|
||||||
|
- [ ] **5.3.2** Campaign editor
|
||||||
|
- [ ] Trigger condition builder
|
||||||
|
- [ ] Message template editor
|
||||||
|
- [ ] Audience targeting
|
||||||
|
- [ ] A/B test configuration
|
||||||
|
- [ ] **5.3.3** Campaign analytics
|
||||||
|
- [ ] Funnel: triggered → sent → opened → retained
|
||||||
|
- [ ] Revenue impact
|
||||||
|
- [ ] Comparison to control group
|
||||||
|
|
||||||
|
**Phase 5 Exit Criteria:**
|
||||||
|
|
||||||
|
- [ ] Health dashboard shows all products with trends
|
||||||
|
- [ ] Churn predictions visible with explanations
|
||||||
|
- [ ] Campaign creation and management functional
|
||||||
|
- [ ] Full test coverage
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 6: Advanced Capabilities (Future)
|
||||||
|
|
||||||
|
### 6.1 Cohort-Specific Models
|
||||||
|
|
||||||
|
- [ ] Segment-specific churn models (iOS vs. Android, free vs. pro)
|
||||||
|
- [ ] Regional models (different behaviors by geography)
|
||||||
|
- [ ] Temporal models (seasonal churn patterns)
|
||||||
|
|
||||||
|
### 6.2 LTV Prediction
|
||||||
|
|
||||||
|
- [ ] Predict lifetime value at signup
|
||||||
|
- [ ] Predict upgrade probability (free → pro)
|
||||||
|
- [ ] Optimize acquisition channels by predicted LTV
|
||||||
|
|
||||||
|
### 6.3 Product Recommendations
|
||||||
|
|
||||||
|
- [ ] Suggest features to at-risk users based on successful cohorts
|
||||||
|
- [ ] Personalized onboarding based on predicted needs
|
||||||
|
- [ ] Next-best-action recommendations
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Appendix A: Data Models
|
||||||
|
|
||||||
|
### UserChurnPredictionDoc
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
interface UserChurnPredictionDoc {
|
||||||
|
id: string; // cp_<uuid>
|
||||||
|
userId: string; // partition key
|
||||||
|
productId: string;
|
||||||
|
|
||||||
|
// Prediction
|
||||||
|
predictionHorizon: 7 | 14 | 30; // Days
|
||||||
|
churnProbability: number; // 0–1
|
||||||
|
riskSegment: 'critical' | 'high' | 'medium' | 'low';
|
||||||
|
|
||||||
|
// Feature vector snapshot
|
||||||
|
features: Record<string, number>; // Normalized feature values
|
||||||
|
featureVersion: string; // Schema version
|
||||||
|
|
||||||
|
// Model info
|
||||||
|
modelVersion: string;
|
||||||
|
modelType: 'xgboost' | 'neural';
|
||||||
|
predictionTimestamp: string;
|
||||||
|
|
||||||
|
// Explanation (SHAP values)
|
||||||
|
explanation: {
|
||||||
|
topRiskFactors: Array<{
|
||||||
|
feature: string;
|
||||||
|
contribution: number; // SHAP value
|
||||||
|
direction: 'positive' | 'negative'; // Increases or decreases churn risk
|
||||||
|
}>;
|
||||||
|
globalFeatureImportance: Array<{
|
||||||
|
feature: string;
|
||||||
|
importance: number;
|
||||||
|
}>;
|
||||||
|
};
|
||||||
|
|
||||||
|
// Natural language summary
|
||||||
|
nlExplanation: string; // Auto-generated explanation
|
||||||
|
|
||||||
|
// Intervention
|
||||||
|
suggestedActions: string[];
|
||||||
|
interventionHistory: Array<{
|
||||||
|
action: string;
|
||||||
|
timestamp: string;
|
||||||
|
outcome?: 'responded' | 'ignored' | 'churned' | 'retained';
|
||||||
|
}>;
|
||||||
|
|
||||||
|
// Validation (ground truth)
|
||||||
|
actualChurned?: boolean;
|
||||||
|
validationDate?: string;
|
||||||
|
|
||||||
|
createdAt: string;
|
||||||
|
ttl: number; // predictionHorizon + 90 days
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### ProductHealthScoreDoc
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
interface ProductHealthScoreDoc {
|
||||||
|
id: string; // ph_<uuid>
|
||||||
|
productId: string; // partition key
|
||||||
|
date: string; // Sort key (YYYY-MM-DD)
|
||||||
|
|
||||||
|
// Composite score
|
||||||
|
overallHealthScore: number; // 0–100
|
||||||
|
healthStatus: 'critical' | 'warning' | 'healthy';
|
||||||
|
|
||||||
|
// Dimension scores
|
||||||
|
dimensions: {
|
||||||
|
acquisition: {
|
||||||
|
score: number; // 0–100
|
||||||
|
metrics: {
|
||||||
|
newUsers: number;
|
||||||
|
activationRateDay1: number;
|
||||||
|
activationRateDay7: number;
|
||||||
|
cac: number;
|
||||||
|
};
|
||||||
|
trend: 'improving' | 'stable' | 'declining';
|
||||||
|
};
|
||||||
|
activation: {
|
||||||
|
score: number;
|
||||||
|
metrics: {
|
||||||
|
firstValueMomentRate: number;
|
||||||
|
timeToFirstAction: number;
|
||||||
|
onboardingCompletionRate: number;
|
||||||
|
};
|
||||||
|
trend: 'improving' | 'stable' | 'declining';
|
||||||
|
};
|
||||||
|
retention: {
|
||||||
|
score: number;
|
||||||
|
metrics: {
|
||||||
|
dau: number;
|
||||||
|
mau: number;
|
||||||
|
dauMauRatio: number;
|
||||||
|
day7Retention: number;
|
||||||
|
day30Retention: number;
|
||||||
|
};
|
||||||
|
trend: 'improving' | 'stable' | 'declining';
|
||||||
|
};
|
||||||
|
engagement: {
|
||||||
|
score: number;
|
||||||
|
metrics: {
|
||||||
|
avgSessionLength: number;
|
||||||
|
sessionsPerUser: number;
|
||||||
|
featureAdoption: Record<string, number>;
|
||||||
|
};
|
||||||
|
trend: 'improving' | 'stable' | 'declining';
|
||||||
|
};
|
||||||
|
revenue: {
|
||||||
|
score: number;
|
||||||
|
metrics: {
|
||||||
|
mrr: number;
|
||||||
|
arpu: number;
|
||||||
|
churnRate: number;
|
||||||
|
upgradeRate: number;
|
||||||
|
};
|
||||||
|
trend: 'improving' | 'stable' | 'declining';
|
||||||
|
};
|
||||||
|
stability: {
|
||||||
|
score: number;
|
||||||
|
metrics: {
|
||||||
|
crashFreeRate: number;
|
||||||
|
errorRate: number;
|
||||||
|
avgLatency: number;
|
||||||
|
uptimePercent: number;
|
||||||
|
};
|
||||||
|
trend: 'improving' | 'stable' | 'declining';
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
// Anomalies detected
|
||||||
|
anomalies: Array<{
|
||||||
|
metric: string;
|
||||||
|
expectedValue: number;
|
||||||
|
actualValue: number;
|
||||||
|
deviationPercent: number;
|
||||||
|
severity: 'critical' | 'warning';
|
||||||
|
suggestedCause?: string;
|
||||||
|
}>;
|
||||||
|
|
||||||
|
// Forecasts
|
||||||
|
forecasts: {
|
||||||
|
next7Days: {
|
||||||
|
expectedHealthScore: number;
|
||||||
|
confidenceInterval: [number, number];
|
||||||
|
};
|
||||||
|
next30Days: {
|
||||||
|
expectedHealthScore: number;
|
||||||
|
confidenceInterval: [number, number];
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
// Benchmarks
|
||||||
|
vsBaseline7Day: number; // % change vs. 7-day baseline
|
||||||
|
vsBaseline30Day: number; // % change vs. 30-day baseline
|
||||||
|
|
||||||
|
createdAt: string;
|
||||||
|
ttl: number; // 2 years
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### RetentionCampaignDoc
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
interface RetentionCampaignDoc {
|
||||||
|
id: string; // rc_<uuid>
|
||||||
|
productId: string; // partition key
|
||||||
|
|
||||||
|
// Campaign definition
|
||||||
|
name: string;
|
||||||
|
description: string;
|
||||||
|
status: 'draft' | 'active' | 'paused' | 'completed';
|
||||||
|
|
||||||
|
// Trigger conditions
|
||||||
|
trigger: {
|
||||||
|
type: 'churn_risk' | 'health_score_drop' | 'behavioral' | 'scheduled';
|
||||||
|
conditions: Array<{
|
||||||
|
field: string;
|
||||||
|
operator: 'gt' | 'lt' | 'eq' | 'in';
|
||||||
|
value: unknown;
|
||||||
|
}>;
|
||||||
|
};
|
||||||
|
|
||||||
|
// Audience
|
||||||
|
audience: {
|
||||||
|
riskSegments?: string[]; // 'critical', 'high', etc.
|
||||||
|
products?: string[];
|
||||||
|
userSegments?: string[];
|
||||||
|
excludeRecentContact?: number; // Hours (frequency capping)
|
||||||
|
};
|
||||||
|
|
||||||
|
// Message content
|
||||||
|
messages: Array<{
|
||||||
|
channel: 'email' | 'push' | 'in_app' | 'slack_cs';
|
||||||
|
templateId: string;
|
||||||
|
variant?: string; // A/B test variant
|
||||||
|
delayHours?: number; // Delay after trigger
|
||||||
|
conditions?: Array<{
|
||||||
|
field: string;
|
||||||
|
operator: string;
|
||||||
|
value: unknown;
|
||||||
|
}>;
|
||||||
|
}>;
|
||||||
|
|
||||||
|
// Performance tracking
|
||||||
|
stats: {
|
||||||
|
triggered: number;
|
||||||
|
sent: number;
|
||||||
|
opened: number;
|
||||||
|
clicked: number;
|
||||||
|
converted: number;
|
||||||
|
controlGroupSize: number;
|
||||||
|
controlChurnRate: number;
|
||||||
|
treatmentChurnRate: number;
|
||||||
|
};
|
||||||
|
|
||||||
|
createdAt: string;
|
||||||
|
updatedAt: string;
|
||||||
|
ttl: number; // 1 year after completion
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### UserFeatureVectorDoc
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
interface UserFeatureVectorDoc {
|
||||||
|
id: string; // fv_<uuid>
|
||||||
|
userId: string; // partition key
|
||||||
|
productId: string;
|
||||||
|
|
||||||
|
// Computed features
|
||||||
|
features: {
|
||||||
|
// Recency features
|
||||||
|
daysSinceLastSession: number;
|
||||||
|
daysSinceLastCoreAction: number;
|
||||||
|
|
||||||
|
// Frequency features
|
||||||
|
sessionsLast7Days: number;
|
||||||
|
sessionsLast30Days: number;
|
||||||
|
avgSessionsPerWeek: number;
|
||||||
|
|
||||||
|
// Engagement depth
|
||||||
|
avgSessionDuration: number;
|
||||||
|
actionsPerSession: number;
|
||||||
|
uniqueFeaturesUsed: number;
|
||||||
|
|
||||||
|
// Product-specific (examples)
|
||||||
|
// NomGap
|
||||||
|
fastCompletionRate?: number;
|
||||||
|
streakLength?: number;
|
||||||
|
|
||||||
|
// JarvisJr
|
||||||
|
agentDiversityScore?: number;
|
||||||
|
voiceSessionRatio?: number;
|
||||||
|
|
||||||
|
// ChronoMind
|
||||||
|
timerCompletionRate?: number;
|
||||||
|
routineAdherenceScore?: number;
|
||||||
|
|
||||||
|
// Error/stability
|
||||||
|
errorRateLast7Days: number;
|
||||||
|
crashCountLast30Days: number;
|
||||||
|
|
||||||
|
// Revenue
|
||||||
|
planTier: number; // 0=free, 1=pro, 2=enterprise
|
||||||
|
lifetimeValue: number;
|
||||||
|
daysSinceLastPayment?: number;
|
||||||
|
};
|
||||||
|
|
||||||
|
// Normalized (0–1) for model input
|
||||||
|
normalizedFeatures: Record<string, number>;
|
||||||
|
|
||||||
|
// Metadata
|
||||||
|
featureSchemaVersion: string;
|
||||||
|
computedAt: string;
|
||||||
|
|
||||||
|
// Time windows
|
||||||
|
observationWindow: {
|
||||||
|
start: string;
|
||||||
|
end: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
ttl: number; // 90 days
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Implementation Tracking
|
||||||
|
|
||||||
|
| Phase | Task | Status | Commit |
|
||||||
|
| ----- | ----------------------------- | ------ | ------ |
|
||||||
|
| 1.1 | Telemetry feature extraction | ⬜ | — |
|
||||||
|
| 1.1 | Time-window aggregations | ⬜ | — |
|
||||||
|
| 1.1 | Rolling window features | ⬜ | — |
|
||||||
|
| 1.2 | Feature store | ⬜ | — |
|
||||||
|
| 1.2 | Cosmos containers | ⬜ | — |
|
||||||
|
| 1.2 | Feature computation jobs | ⬜ | — |
|
||||||
|
| 1.3 | Product-specific features | ⬜ | — |
|
||||||
|
| 1.3 | Feature importance tracking | ⬜ | — |
|
||||||
|
| 2.1 | XGBoost model architecture | ⬜ | — |
|
||||||
|
| 2.1 | Training pipeline | ⬜ | — |
|
||||||
|
| 2.1 | Model evaluation | ⬜ | — |
|
||||||
|
| 2.2 | Real-time scoring API | ⬜ | — |
|
||||||
|
| 2.2 | Risk segmentation | ⬜ | — |
|
||||||
|
| 2.2 | Model versioning | ⬜ | — |
|
||||||
|
| 2.3 | SHAP explanations | ⬜ | — |
|
||||||
|
| 2.3 | Natural language explanations | ⬜ | — |
|
||||||
|
| 2.3 | Actionable insights | ⬜ | — |
|
||||||
|
| 3.1 | Health metric framework | ⬜ | — |
|
||||||
|
| 3.1 | Health indicators | ⬜ | — |
|
||||||
|
| 3.2 | Baseline establishment | ⬜ | — |
|
||||||
|
| 3.2 | Scoring algorithm | ⬜ | — |
|
||||||
|
| 3.2 | Alert thresholds | ⬜ | — |
|
||||||
|
| 3.3 | Anomaly detection | ⬜ | — |
|
||||||
|
| 4.1 | Campaign trigger rules | ⬜ | — |
|
||||||
|
| 4.1 | Personalized messaging | ⬜ | — |
|
||||||
|
| 4.2 | Platform integrations | ⬜ | — |
|
||||||
|
| 4.3 | CS team dashboard | ⬜ | — |
|
||||||
|
| 5.1 | Health overview UI | ⬜ | — |
|
||||||
|
| 5.2 | Churn prediction dashboard | ⬜ | — |
|
||||||
|
| 5.3 | Campaign management | ⬜ | — |
|
||||||
|
|
||||||
|
**Legend:** ⬜ Not started | 🟡 In progress | ✅ Complete | ⏸️ Deferred
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Quick Reference for Implementing Agent
|
||||||
|
|
||||||
|
**📋 Full Roadmap:** `/Users/sd9235/code/mygh/learning_ai_common_plat/docs/roadmaps/PREDICTIVE_CHURN_HEALTH_SCORING_ROADMAP.md`
|
||||||
|
|
||||||
|
**Key Files to Modify/Create:**
|
||||||
|
|
||||||
|
```
|
||||||
|
services/platform-service/
|
||||||
|
├── src/
|
||||||
|
│ ├── modules/predictive-analytics/
|
||||||
|
│ │ ├── types.ts # [1.2] Feature, HealthScore, Prediction types
|
||||||
|
│ │ ├── repository.ts # Data access layer
|
||||||
|
│ │ ├── feature-extractor.ts # [1.1] Telemetry → features
|
||||||
|
│ │ ├── feature-store.ts # [1.2] Feature vector storage
|
||||||
|
│ │ ├── churn-model.ts # [2.1] XGBoost training & inference
|
||||||
|
│ │ ├── scoring-api.ts # [2.2] Real-time prediction endpoint
|
||||||
|
│ │ ├── explanation-engine.ts # [2.3] SHAP + NL explanations
|
||||||
|
│ │ ├── health-scoring.ts # [3] Health dimension calculation
|
||||||
|
│ │ ├── anomaly-detection.ts # [3.3] Prophet/ARIMA forecasting
|
||||||
|
│ │ ├── campaign-engine.ts # [4] Retention automation
|
||||||
|
│ │ ├── routes.ts # [5] REST API
|
||||||
|
│ │ └── predictive-analytics.test.ts # Tests
|
||||||
|
│ ├── lib/
|
||||||
|
│ │ └── cosmos-init.ts # [1.2] Add containers
|
||||||
|
│ └── server.ts # Register routes
|
||||||
|
dashboards/admin-web/
|
||||||
|
├── src/
|
||||||
|
│ ├── app/(dashboard)/
|
||||||
|
│ │ ├── health-dashboard/
|
||||||
|
│ │ │ └── page.tsx # [5.1] Product health overview
|
||||||
|
│ │ └── predictive/
|
||||||
|
│ │ ├── at-risk/
|
||||||
|
│ │ │ └── page.tsx # [4.3] At-risk user list
|
||||||
|
│ │ └── campaigns/
|
||||||
|
│ │ └── page.tsx # [5.3] Campaign management
|
||||||
|
│ ├── lib/
|
||||||
|
│ │ └── predictive-client.ts # API client
|
||||||
|
│ └── components/
|
||||||
|
│ └── predictive/ # Risk cards, health charts
|
||||||
|
```
|
||||||
|
|
||||||
|
**Commit Message Format:**
|
||||||
|
|
||||||
|
```
|
||||||
|
feat(predictive-analytics): <description> [<task.code>]
|
||||||
|
```
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add services/platform-service/src/modules/predictive-analytics/
|
||||||
|
git commit -m "feat(predictive-analytics): add feature extraction and store [1.1-1.2]"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Testing Requirements:**
|
||||||
|
|
||||||
|
- Unit tests: 20+ Vitest tests for feature extraction, model inference
|
||||||
|
- Model validation: AUC, calibration, precision@k metrics
|
||||||
|
- Integration: End-to-end prediction pipeline
|
||||||
|
|
||||||
|
**Dependencies:**
|
||||||
|
|
||||||
|
- Telemetry module (feature extraction)
|
||||||
|
- Delivery module (retention campaigns)
|
||||||
|
- Azure ML or scikit-learn (model training)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Appendix B: API Reference
|
||||||
|
|
||||||
|
| Method | Endpoint | Auth | Description |
|
||||||
|
| ------ | -------------------------------------- | ------------- | ----------------------------- |
|
||||||
|
| GET | `/predictive/health` | Admin | Get all product health scores |
|
||||||
|
| GET | `/predictive/health/:productId` | Admin | Get product health detail |
|
||||||
|
| GET | `/predictive/health/:productId/trends` | Admin | Historical health trends |
|
||||||
|
| POST | `/predictive/churn-score` | Admin/Service | Get churn prediction for user |
|
||||||
|
| POST | `/predictive/churn-batch` | Admin | Batch churn scoring |
|
||||||
|
| GET | `/predictive/at-risk-users` | Admin/CS | List users by risk segment |
|
||||||
|
| GET | `/predictive/users/:id/risk-profile` | Admin/CS | User churn risk details |
|
||||||
|
| GET | `/predictive/model/performance` | Admin | Model accuracy metrics |
|
||||||
|
| GET | `/predictive/model/features` | Admin | Feature importance ranking |
|
||||||
|
| GET | `/predictive/campaigns` | Admin | List retention campaigns |
|
||||||
|
| POST | `/predictive/campaigns` | Admin | Create campaign |
|
||||||
|
| PATCH | `/predictive/campaigns/:id` | Admin | Update campaign |
|
||||||
|
| GET | `/predictive/campaigns/:id/stats` | Admin | Campaign performance |
|
||||||
|
| POST | `/predictive/campaigns/:id/trigger` | Admin | Manual trigger for testing |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Appendix C: Integration Points
|
||||||
|
|
||||||
|
### With Telemetry Module
|
||||||
|
|
||||||
|
- Raw events feed feature extraction
|
||||||
|
- Error rates flow into health scores
|
||||||
|
- Correlation IDs link behaviors to predictions
|
||||||
|
|
||||||
|
### With Diagnostics Module
|
||||||
|
|
||||||
|
- Debug sessions enrich feature vectors
|
||||||
|
- Error clusters correlate with churn risk
|
||||||
|
- Screenshot patterns analyzed for UX issues
|
||||||
|
|
||||||
|
### With Event Bus
|
||||||
|
|
||||||
|
| Event | Action |
|
||||||
|
| ----------------------------------- | ---------------------------------------- |
|
||||||
|
| `predictive.churn.risk_detected` | Trigger retention campaign |
|
||||||
|
| `predictive.health.critical` | Alert leadership, suggest debug sessions |
|
||||||
|
| `predictive.anomaly.detected` | Create incident, notify on-call |
|
||||||
|
| `user.retention.campaign_responded` | Update model with outcome |
|
||||||
|
|
||||||
|
### With Delivery Module
|
||||||
|
|
||||||
|
- Retention campaigns use email templates
|
||||||
|
- Push notifications for urgent interventions
|
||||||
|
- A/B test message variants
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Appendix D: Cost Estimation
|
||||||
|
|
||||||
|
| Component | Monthly Cost (est.) |
|
||||||
|
| ------------------------------------ | -------------------------- |
|
||||||
|
| Cosmos DB (features + predictions) | $150–300 |
|
||||||
|
| Model training (Azure ML) | $100–200 |
|
||||||
|
| Inference compute | $50–100 |
|
||||||
|
| Email delivery (retention campaigns) | $50–200 (volume-dependent) |
|
||||||
|
| **Total** | **$350–800/month** |
|
||||||
|
|
||||||
|
ROI: If system prevents 5% of predicted churn at $50 LTV with 10K at-risk users/month:
|
||||||
|
|
||||||
|
- 500 users retained × $50 = $25K/month value
|
||||||
|
- 10:1+ ROI
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Appendix E: Success Metrics
|
||||||
|
|
||||||
|
### Model Performance
|
||||||
|
|
||||||
|
- [ ] AUC > 75% (discrimination)
|
||||||
|
- [ ] Calibration slope 0.9–1.1 (well-calibrated probabilities)
|
||||||
|
- [ ] Precision@10% > 60% (high-risk predictions are accurate)
|
||||||
|
|
||||||
|
### Business Impact
|
||||||
|
|
||||||
|
- [ ] 10%+ reduction in churn rate for targeted cohorts
|
||||||
|
- [ ] 5%+ increase in re-engagement campaign response
|
||||||
|
- [ ] CS team satisfaction with at-risk user visibility
|
||||||
|
|
||||||
|
### Operational
|
||||||
|
|
||||||
|
- [ ] Prediction latency < 100ms
|
||||||
|
- [ ] Feature freshness < 24 hours
|
||||||
|
- [ ] Model retraining automation
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Current Status
|
||||||
|
|
||||||
|
- [ ] **Design complete** — Target: 2026-03-10
|
||||||
|
- [ ] **Phase 1: Feature Pipeline** — Not started
|
||||||
|
- [ ] **Phase 2: Churn Model** — Not started
|
||||||
|
- [ ] **Phase 3: Health Scoring** — Not started
|
||||||
|
- [ ] **Phase 4: Interventions** — Not started
|
||||||
|
- [ ] **Phase 5: Admin UI** — Not started
|
||||||
|
- [ ] **Phase 6: Advanced** — Future
|
||||||
|
|
||||||
|
**Estimated Timeline:** 3 weeks (Phases 1–5)
|
||||||
|
|
||||||
|
**Dependencies:**
|
||||||
|
|
||||||
|
- Telemetry module (for feature extraction)
|
||||||
|
- Azure ML or similar (for model training)
|
||||||
|
- Delivery module (for retention campaigns)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
_Last Updated: 2026-03-03_
|
||||||
Loading…
Reference in New Issue
Block a user