saravanakumardb1 0edafab501 docs: Phase 2 complete — Android telemetry checked off, roadmap updated

2026-02-17 10:16:45 -08:00

22 KiB

Raw Blame History

Client Telemetry — Implementation Roadmap

Status: Phase 2 complete ✅, Phase 3 in progress
Last updated: 2026-02-17
Design doc: CLIENT_TELEMETRY_DESIGN.md
Repos: learning_ai_common_plat (platform-service) · learning_voice_ai_agent (all clients + dashboards)

Phase 0 — Design & Review

Write comprehensive telemetry design doc — schema, APIs, admin UX, privacy guardrails (c59049e)
Systematic review: identify and fix 18 bugs/gaps in the design doc (083cf02)
- TTL format (ISO → seconds), regionCode prefix format, missing pk field
- Auth model for keyboard extension (X-Install-Token)
- Config endpoint query params (userId/anonymousInstallId)
- Error clustering made version-agnostic (affectedVersions array)
- GDPR erasure endpoint added
- iOS offline queue strategy (App Group UserDefaults, FIFO eviction)
- Global defaults for batchSize/flushInterval/maxQueueSize

Phase 1 — MVP (iOS Keyboard + Backend + Admin UI)

Platform-Service Telemetry Module

types.ts — Zod schemas for events, policies, clusters, queries (ce4c4ff)
repository.ts — Cosmos DB CRUD for events, policies, clusters (ce4c4ff)
routes.ts — Fastify routes: ingestion, config, admin query, clusters, policy CRUD, GDPR erasure (ce4c4ff)
telemetry.test.ts — 34 Vitest tests for schemas + policy evaluation (ce4c4ff)
Register telemetry routes in server.ts (ce4c4ff)
Add Cosmos containers (telemetry_events, telemetry_error_clusters, telemetry_collection_policies) to cosmos-init.ts (ce4c4ff)

iOS Keyboard Telemetry Client

LysnrTelemetry.swift — Singleton client with App Group offline queue, X-Install-Token auth, 200-event cap (e546475)
Instrument KeyboardViewController.swift — 10+ telemetry points (e546475)
- session_started / session_ended (with full DictationContext)
- backend_selected (azure / local + reason)
- recognition_started / recognition_failed
- mic_permission_denied
- insert_noop detection
- error_recovery_attempted (local→azure, azure→local)
- Session summary metrics (duration, segments, words, transcript length)

Admin Dashboard — Client Logs Page

/ops/client-logs/page.tsx — Events table + Error Clusters tab (d202f94)
- Stat cards (total events, errors, warnings, keyboard events)
- Filters (platform, channel, level, module, free-text search)
- Expandable event detail rows (device, tags, metrics, dictation context)
- Error Clusters tab with severity, affected versions, user count
/api/telemetry/route.ts — API route proxying to platform-service (d202f94)
platform-client.ts — queryTelemetryEvents + queryTelemetryClusters (d202f94)
sidebar-nav.tsx — "Client Logs" nav item with FileText icon (d202f94)

Phase 2 — Full Platform Coverage

iOS Main App

TelemetryService.swift — Main app telemetry service with App Group queue drain on foreground (a173baa)
LysnrAIApp.swift — scenePhase integration for activate/deactivate lifecycle (a173baa)
- app_foregrounded / app_backgrounded events
- Keyboard queue flush on every foreground transition
- 60-second periodic flush timer

Desktop App (Python)

platform_telemetry.py — PlatformTelemetry singleton with urllib.request POST, threaded flush timer, persistent install_id in ~/.LysnrAI/install_id (a173baa)
main.py instrumentation (a173baa)
- app_started / app_stopped lifecycle events
- dictation_started (with backend tag)
- dictation_completed (with duration_ms, word_count, transcript_length metrics)
- mic_permission_denied / recording_start_failed error events

Web User Dashboard

telemetry.ts — Browser client with sendBeacon, localStorage install ID, auto-flush on visibility change (130e1d6)
/api/telemetry/ingest/route.ts — Server-side proxy to platform-service (130e1d6)
providers.tsx — initTelemetry() called on app mount (130e1d6)

Tracker Dashboard

telemetry.ts — Browser client (same pattern as user dashboard) (a102609)
/api/telemetry/ingest/route.ts — Server-side proxy to platform-service (a102609)
providers.tsx — initTelemetry() called on app mount (a102609)

Admin Dashboard Self-Telemetry

telemetry.ts — Browser client tracking admin page views, filter usage, policy changes (a102609)
/api/telemetry/admin-ingest/route.ts — Separate proxy from admin query route (a102609)
providers.tsx — initTelemetry() called on app mount (a102609)

Android

TelemetryClient.kt — Kotlin singleton with OkHttp POST, SharedPreferences offline queue, persistent install ID (9196f48)
Instrument LysnrInputMethodService.kt — 10 telemetry points (9196f48)
- session_started / session_ended (with words_inserted metric)
- dictation_started (with backend + reason tags)
- dictation_completed (with duration_ms, word_count, segment_count, transcript_length)
- mic_permission_denied
- recognition_failed (with errorCode + errorDomain)
- error_recovery_attempted (azure→local fallback)
Offline queue using SharedPreferences with FIFO eviction (9196f48)
Flush on app foreground via ProcessLifecycleOwner + 60s periodic flush timer (9196f48)

Phase 3 — Intelligence & Admin Tooling

Error Clustering & Alerting

Automated error fingerprinting (hash of platform + channel + module + eventName + errorDomain + errorCode)
Cluster severity escalation (warn → error → fatal based on count + affected users)
Slack/email alerting when cluster severity escalates
Dashboard: cluster timeline chart showing occurrence rate over time
Dashboard: "Resolve" / "Ignore" actions on clusters

Geo Enrichment

Server-side IP → country/region lookup on ingestion (GeoLite2 or Azure Maps)
Populate regionCode field (e.g., US:WA) for events without client-provided region
Admin UI: geographic heatmap of error distribution
Policy targeting by regionCode ranges

Collection Policy Builder UI

Admin page: /ops/telemetry-policies
CRUD UI for collection policies (name, enabled, targeting rules, sampling rates)
Targeting builder: platform checkboxes, version range inputs, percentage slider
Live preview: "N clients would match this policy"
Policy activation/deactivation toggle
Scheduling: startsAt / expiresAt date pickers

Privacy & Compliance

PII regex scanner on ingestion (email, phone, SSN patterns → redact before storage)
Admin UI: GDPR erasure tool (search by userId → delete all events)
Retention policy enforcement (TTL-based auto-expiry per container)
Audit log entries for policy changes and data deletions

Performance & Scale

Client-side config caching (poll /api/telemetry/config with If-None-Match ETag)
Server-side rate limiting per installId (100 events/min default)
Cosmos DB indexing policy tuning for telemetry_events (composite indexes on query patterns)
Batch ingestion deduplication by event.id
Prometheus metrics for ingestion throughput and error rates

Architecture Summary

┌─────────────────────┐    ┌──────────────────────┐    ┌───────────────────┐
│  iOS Keyboard Ext   │    │   iOS Main App       │    │  Desktop (Python) │
│  LysnrTelemetry     │───▶│  TelemetryService    │    │  PlatformTelemetry│
│  (App Group queue)  │    │  (drains queue)      │    │  (urllib POST)    │
└─────────────────────┘    └──────────┬───────────┘    └────────┬──────────┘
  Full Access ON ──┐                  │                          │
  direct POST      │                  │                          │
                   ▼                  ▼                          ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                     Platform Service (Fastify, port 4003)              │
│  POST   /api/telemetry/events       — batch ingestion                 │
│  GET    /api/telemetry/config       — client collection config        │
│  GET    /api/telemetry/query        — admin event search              │
│  GET    /api/telemetry/clusters     — admin error clusters            │
│  CRUD   /api/telemetry/policies     — collection policy management    │
│  DELETE /api/telemetry/user/:userId — GDPR erasure                    │
└────────────────────────────┬────────────────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                        Azure Cosmos DB                                 │
│  telemetry_events              partitionKeyPath: /pk                   │
│    pk value = productId:yyyyMM:platform   (e.g. lysnrai:202602:ios)   │
│  telemetry_error_clusters      partitionKeyPath: /pk                   │
│    pk value = productId:platform:module   (e.g. lysnrai:ios:dictation)│
│  telemetry_collection_policies partitionKeyPath: /productId            │
└─────────────────────────────────────────────────────────────────────────┘

┌─────────────────────────┐          ┌──────────────────────┐
│  Admin Dashboard        │  GET     │  User Dashboard      │  POST
│  /ops/client-logs       │─────────▶│  /api/telemetry/     │─────────▶ platform
│  (queries via           │  query/  │  ingest              │  /events  -service
│   platform-service API) │  clusters│  (browser → proxy)   │
└─────────────────────────┘          └──────────────────────┘

┌───────────────────────┐
│  Android              │
│  TelemetryClient.kt   │──▶ POST /api/telemetry/events ──▶ platform-service
│  (SharedPreferences)  │
└───────────────────────┘

Test Coverage

Component	Test File	Tests	Coverage
Platform-service schemas	`telemetry.test.ts`	34	Zod schemas: event, ingest, policy CRUD, query
Platform-service route logic	`telemetry.test.ts`	43	`containsPII` (6), `computePk` (4), `normalizeMessage` (7), `generateFingerprint` (8), `policyMatchesContext` (13), `mergePolicies` (5)
Desktop Python client	`tests/cloud/test_platform_telemetry.py`	19	Event format (6), queue behavior (2), session mgmt (2), flush/HTTP (5), install ID (2), singleton (2)
Web dashboard client	`user-dashboard-web/src/__tests__/telemetry.test.ts`	12	`trackEvent` (3), `trackPageView` (1), `flush` (4), install ID (2), `initTelemetry` (2)
Tracker dashboard client	`tracker-dashboard-web/src/__tests__/telemetry.test.ts`	10	`trackEvent` (3), `trackPageView` (1), `flush` (4), `initTelemetry` (2)
Admin dashboard client	`admin-dashboard-web/src/__tests__/telemetry.test.ts`	10	`trackEvent` (3), `trackPageView` (1), `flush` (4), `initTelemetry` (2)
Total		128

Verification commands

# Platform-service (77 telemetry tests within 611 total)
cd ../learning_ai_common_plat && pnpm --filter @lysnrai/platform-service test

# Desktop Python (19 tests)
cd learning_voice_ai_agent && python -m pytest tests/cloud/test_platform_telemetry.py -v

# Web user-dashboard (12 tests)
cd learning_voice_ai_agent/user-dashboard-web && npx vitest run src/__tests__/telemetry.test.ts

Not yet tested

iOS LysnrTelemetry.swift — needs XCTest target (Swift unit tests)
iOS TelemetryService.swift — needs XCTest target
Android TelemetryClient.kt — needs Android instrumented tests or Robolectric
Admin dashboard /api/telemetry/route.ts — API route integration test
Platform-service HTTP integration tests (Fastify inject)
End-to-end: client → platform-service → Cosmos read-back

Bugs Found During Review

The following bugs were discovered during systematic review of the roadmap against actual code and fixed:

#	Severity	Issue	Fix
1	High	Desktop Python `id` used `uuid.uuid4().hex` (32 hex, no dashes) — fails Zod `.uuid()` server validation	Changed to `str(uuid.uuid4())`
2	High	Web telemetry `osFamily='web'` not in Zod `OsFamilyEnum` — fails server validation	Changed to `'other'`
3	Medium	Status said "Phase 2 complete" but Android is all unchecked	Fixed status line
4	Medium	Architecture diagram showed wrong pk for `telemetry_error_clusters` (`/productId` → actual `/pk` = `productId:platform:module`)	Fixed diagram
5	Medium	Tracker dashboard telemetry missing from roadmap entirely	Added as Phase 2 pending
6	Medium	Admin dashboard self-telemetry (page views) not mentioned	Added as Phase 2 pending
7	Low	Architecture diagram missing Android client box	Added with "not yet implemented" note
8	Low	Architecture diagram implied Admin reads Cosmos directly (it queries Platform Service)	Fixed data flow arrows
9	Low	Web `telemetry.ts` JSDoc said "via the admin dashboard proxy" (wrong dashboard)	Fixed to "user dashboard's /api/telemetry/ingest proxy"
10	Low	Commit log missing roadmap doc commit	Added

Commit Log

Date	Repo	Commit	Description
2026-02-16	common-plat	`c59049e`	Design doc: client telemetry & log insights
2026-02-16	common-plat	`083cf02`	Fix 18 gaps in telemetry design doc (rev 2)
2026-02-16	common-plat	`ce4c4ff`	Telemetry module — ingest, config, query, clusters, policies (34 tests)
2026-02-17	voice-agent	`e546475`	iOS keyboard telemetry client + KeyboardViewController instrumentation
2026-02-17	voice-agent	`d202f94`	Admin dashboard Client Logs page + sidebar nav
2026-02-17	voice-agent	`a173baa`	iOS main app TelemetryService + Desktop Python platform_telemetry
2026-02-17	voice-agent	`130e1d6`	Web user-dashboard telemetry client + ingest proxy
2026-02-17	common-plat	`c3d6977`	Telemetry roadmap doc (this file)
2026-02-17	voice-agent	`ae77438`	Fix: desktop uuid format + web osFamily — pass Zod validation
2026-02-17	common-plat	`20f77d5`	Tests: route-logic tests — PII, pk, fingerprint, policy matching (34→77)
2026-02-17	voice-agent	`08efdb6`	Tests: Python client (19) + web dashboard (12) telemetry tests
2026-02-17	voice-agent	`a102609`	Tracker + admin self-telemetry clients + tests (20 tests)
2026-02-17	voice-agent	`9196f48`	Android TelemetryClient + keyboard instrumentation + ProcessLifecycleOwner

22 KiB Raw Blame History

Client Telemetry — Implementation Roadmap

Phase 0 — Design & Review

Phase 1 — MVP (iOS Keyboard + Backend + Admin UI)

Platform-Service Telemetry Module

iOS Keyboard Telemetry Client

Admin Dashboard — Client Logs Page

Phase 2 — Full Platform Coverage

iOS Main App

Desktop App (Python)

Web User Dashboard

Tracker Dashboard

Admin Dashboard Self-Telemetry

Android

Phase 3 — Intelligence & Admin Tooling

Error Clustering & Alerting

Geo Enrichment

Collection Policy Builder UI

Privacy & Compliance

Performance & Scale

Architecture Summary

Test Coverage

Verification commands

Not yet tested

Bugs Found During Review

Commit Log

22 KiB

Raw Blame History