docs(devops): add environment variables and Key Vault audit

Comprehensive audit of environment variables and Azure Key Vault secrets: - Scanned all 35+ environment variables used across services - Audited all 17 secrets in Azure Key Vault - Identified critical gap: ZERO LysnrAI secrets in Key Vault - MindLyst: 12 secrets (fully populated) - MyWisprAI: 7 secrets (partially populated) - LysnrAI: 0 secrets (100% missing) - Listed 13 missing critical/high priority secrets - Provided remediation plan with seed script instructions - Added secret rotation strategy - Included quick fix commands for Azure CLI Critical findings: - Missing Cosmos DB, JWT, Stripe, Gemini API keys for LysnrAI - Code expects lysnr-* prefixed secrets but none exist - Immediate action required to seed Key Vault Co-Authored-By: Warp <agent@warp.dev>
2026-02-14 23:05:56 -08:00 · 2026-02-14 23:05:56 -08:00 · ea44a73b62
commit ea44a73b62
parent 598a84513a
1 changed files with 485 additions and 0 deletions
--- a/docs/devops/ENVIRONMENT_VARIABLES_AND_KEYVAULT_AUDIT.md
+++ b/docs/devops/ENVIRONMENT_VARIABLES_AND_KEYVAULT_AUDIT.md
@ -0,0 +1,485 @@
+# Environment Variables & Azure Key Vault Audit
+
+> **Last Updated:** 2026-02-15  
+> **Purpose:** Complete audit of environment variables, Azure Key Vault secrets, and gap analysis
+
+---
+
+## 🎯 Executive Summary
+
+### Critical Findings:
+1. ❌ **ZERO LysnrAI secrets** exist in Azure Key Vault despite code expecting them
+2. ✅ **MindLyst secrets** are fully populated (12 secrets)
+3. ✅ **MyWisprAI secrets** are partially populated (7 secrets)
+4. ⚠️ **Mismatch** between code expectations and actual Key Vault state
+5. ⚠️ **Missing Stripe secrets** for billing functionality
+6. ⚠️ **Missing Gemini API key** for extraction service
+
+---
+
+## 📊 Key Vault Current State
+
+### Azure Key Vault: `kv-mywisprai`
+**Location:** East US  
+**Total Secrets:** 17
+
+| Secret Name | Product | Purpose | Created | Status |
+|-------------|---------|---------|---------|--------|
+| `mindlyst-appinsights-connection-string` | MindLyst | Application Insights telemetry | 2026-02-14 | ✅ Active |
+| `mindlyst-blob-connection-string` | MindLyst | Blob Storage access | 2026-02-14 | ✅ Active |
+| `mindlyst-cosmos-database` | MindLyst | Cosmos DB database name | 2026-02-14 | ✅ Active |
+| `mindlyst-cosmos-endpoint` | MindLyst | Cosmos DB endpoint | 2026-02-14 | ✅ Active |
+| `mindlyst-cosmos-key` | MindLyst | Cosmos DB primary key | 2026-02-14 | ✅ Active |
+| `mindlyst-notification-hub-connection-string` | MindLyst | Push notifications | 2026-02-14 | ✅ Active |
+| `mindlyst-openai-api-version` | MindLyst | OpenAI API version | 2026-02-14 | ✅ Active |
+| `mindlyst-openai-deployment` | MindLyst | OpenAI deployment name | 2026-02-14 | ✅ Active |
+| `mindlyst-openai-endpoint` | MindLyst | Azure OpenAI endpoint | 2026-02-14 | ✅ Active |
+| `mindlyst-openai-key` | MindLyst | Azure OpenAI key | 2026-02-14 | ✅ Active |
+| `mindlyst-speech-key` | MindLyst | Azure Speech Services key | 2026-02-14 | ✅ Active |
+| `mindlyst-speech-region` | MindLyst | Azure Speech region | 2026-02-14 | ✅ Active |
+| `wispr-azure-openai-deployment` | MyWisprAI | OpenAI deployment name | 2026-02-07 | ✅ Active |
+| `wispr-azure-openai-endpoint` | MyWisprAI | Azure OpenAI endpoint | 2026-02-07 | ✅ Active |
+| `wispr-azure-openai-key` | MyWisprAI | Azure OpenAI key | 2026-02-07 | ✅ Active |
+| `wispr-azure-speech-key` | MyWisprAI | Azure Speech Services key | 2026-02-07 | ✅ Active |
+| `wispr-azure-speech-region` | MyWisprAI | Azure Speech region | 2026-02-07 | ✅ Active |
+
+---
+
+## 🔍 Code Expectations vs Reality
+
+### Expected Secrets (from `packages/config/src/keyvault.ts`)
+
+The `LYSNR_SECRETS` constant defines these mappings:
+
+| Key Vault Secret Name | Environment Variable | Status in KV | Priority |
+|-----------------------|---------------------|--------------|----------|
+| `lysnr-cosmos-key` | `COSMOS_KEY` | ❌ **MISSING** | 🔴 Critical |
+| `lysnr-cosmos-endpoint` | `COSMOS_ENDPOINT` | ❌ **MISSING** | 🔴 Critical |
+| `lysnr-jwt-secret` | `JWT_SECRET` | ❌ **MISSING** | 🔴 Critical |
+| `lysnr-stripe-secret-key` | `STRIPE_SECRET_KEY` | ❌ **MISSING** | 🔴 Critical |
+| `lysnr-stripe-webhook-secret` | `STRIPE_WEBHOOK_SECRET` | ❌ **MISSING** | 🔴 Critical |
+| `lysnr-billing-internal-key` | `BILLING_INTERNAL_KEY` | ❌ **MISSING** | 🟠 High |
+| `lysnr-blob-connection-string` | `AZURE_BLOB_CONNECTION_STRING` | ❌ **MISSING** | 🔴 Critical |
+| `lysnr-blob-account-key` | `AZURE_BLOB_ACCOUNT_KEY` | ❌ **MISSING** | 🔴 Critical |
+| `lysnr-gemini-api-key` | `GEMINI_API_KEY` | ❌ **MISSING** | 🔴 Critical |
+| `lysnr-seed-secret` | `SEED_SECRET` | ❌ **MISSING** | 🟡 Medium |
+| `lysnr-azure-speech-key` | `AZURE_SPEECH_KEY` | ❌ **MISSING** | 🟠 High |
+| `lysnr-azure-openai-key` | `AZURE_OPENAI_KEY` | ❌ **MISSING** | 🟠 High |
+| `lysnr-azure-openai-endpoint` | `AZURE_OPENAI_ENDPOINT` | ❌ **MISSING** | 🟠 High |
+
+**Total Expected:** 13 secrets  
+**Total Missing:** 13 secrets (100%)
+
+---
+
+## 📋 Complete Environment Variable Inventory
+
+### Core Platform Variables (from `.env.example`)
+
+#### 🔐 Azure Key Vault
+| Variable | Required | Default | Used By | In KV? |
+|----------|----------|---------|---------|--------|
+| `AZURE_KEYVAULT_URL` | Optional | - | All services | N/A |
+
+**Purpose:** Enable Key Vault secret resolution. Falls back to env vars if not set.
+
+---
+
+#### 🗄️ Azure Cosmos DB
+| Variable | Required | Default | Used By | In KV? |
+|----------|----------|---------|---------|--------|
+| `COSMOS_ENDPOINT` | ✅ Yes | - | All services | ❌ No (lysnr) |
+| `COSMOS_KEY` | ✅ Yes | - | All services | ❌ No (lysnr) |
+| `COSMOS_DATABASE` | No | `lysnrai` | All services | N/A |
+
+**Purpose:** Database connection for all products (lysnrai, mindlyst, mywisprai databases).
+
+---
+
+#### 🔑 Authentication & Security
+| Variable | Required | Default | Used By | In KV? |
+|----------|----------|---------|---------|--------|
+| `JWT_SECRET` | ✅ Yes | - | platform-service, extraction-service | ❌ No |
+
+**Purpose:** JWT token signing and verification across all services.
+
+---
+
+#### 💾 Azure Blob Storage
+| Variable | Required | Default | Used By | In KV? |
+|----------|----------|---------|---------|--------|
+| `AZURE_BLOB_CONNECTION_STRING` | Optional | - | platform-service | ❌ No |
+| `AZURE_BLOB_ACCOUNT_NAME` | Optional | `bytelystblobs` | platform-service | N/A |
+| `AZURE_BLOB_ACCOUNT_KEY` | Optional | - | platform-service | ❌ No |
+
+**Purpose:** File uploads, avatar storage, document storage.  
+**Note:** Only needed for platform-service blob module.
+
+---
+
+#### 💳 Stripe (Billing & Subscriptions)
+| Variable | Required | Default | Used By | In KV? |
+|----------|----------|---------|---------|--------|
+| `STRIPE_SECRET_KEY` | Optional | - | platform-service | ❌ No |
+| `STRIPE_WEBHOOK_SECRET` | Optional | - | platform-service | ❌ No |
+| `STRIPE_PRICE_PRO` | Optional | - | platform-service | N/A |
+| `STRIPE_PRICE_ENTERPRISE` | Optional | - | platform-service | N/A |
+| `BILLING_INTERNAL_KEY` | Optional | - | platform-service | ❌ No |
+
+**Purpose:** Subscription management, payment processing, webhook validation.  
+**Status:** ❌ **Required for production but completely missing from Key Vault**
+
+---
+
+#### 🤖 AI Services - Extraction
+| Variable | Required | Default | Used By | In KV? |
+|----------|----------|---------|---------|--------|
+| `GEMINI_API_KEY` | ✅ Yes | - | extraction-service (Python sidecar) | ❌ No |
+| `PYTHON_SIDECAR_URL` | No | `http://localhost:4006` | extraction-service | N/A |
+| `DEFAULT_MODEL_ID` | No | `gemini-2.5-flash` | extraction-service | N/A |
+
+**Purpose:** Text extraction, content analysis, AI-powered features.  
+**Status:** ❌ **Critical for extraction service functionality**
+
+---
+
+#### 🧠 Azure OpenAI (Optional)
+| Variable | Required | Default | Used By | In KV? |
+|----------|----------|---------|---------|--------|
+| `AZURE_OPENAI_KEY` | Optional | - | extraction-service | ❌ No (lysnr) |
+| `AZURE_OPENAI_ENDPOINT` | Optional | - | extraction-service | ❌ No (lysnr) |
+| `AZURE_OPENAI_DEPLOYMENT_NAME` | Optional | - | extraction-service | N/A |
+
+**Purpose:** Alternative to Gemini for text extraction.  
+**Existing:** ✅ Available for `wispr` and `mindlyst` products in Key Vault.
+
+---
+
+#### 🎤 Azure Speech Services
+| Variable | Required | Default | Used By | In KV? |
+|----------|----------|---------|---------|--------|
+| `AZURE_SPEECH_KEY` | Optional | - | LysnrAI product | ❌ No (lysnr) |
+| `AZURE_SPEECH_REGION` | Optional | `eastus` | LysnrAI product | N/A |
+
+**Purpose:** Speech-to-text, text-to-speech for voice AI features.  
+**Existing:** ✅ Available for `wispr` and `mindlyst` products in Key Vault.
+
+---
+
+#### 🚀 Service Configuration
+| Variable | Required | Default | Used By | In KV? |
+|----------|----------|---------|---------|--------|
+| `PORT` | No | varies | All services | N/A |
+| `HOST` | No | `0.0.0.0` | All services | N/A |
+| `NODE_ENV` | No | `development` | All services | N/A |
+| `SERVICE_NAME` | No | auto | All services | N/A |
+| `CORS_ORIGIN` | No | `*` (dev) | All services | N/A |
+| `DEFAULT_PRODUCT_ID` | No | `lysnrai` | All services | N/A |
+
+**Purpose:** Runtime configuration, not secrets.
+
+---
+
+#### 📊 Application Insights (Monitoring)
+| Variable | Required | Default | Used By | In KV? |
+|----------|----------|---------|---------|--------|
+| `APPLICATIONINSIGHTS_CONNECTION_STRING` | Optional | - | All services (future) | ✅ Yes (mindlyst) |
+| `APPINSIGHTS_INSTRUMENTATIONKEY` | Optional | - | All services (future) | N/A |
+
+**Purpose:** Telemetry, logging, monitoring.  
+**Note:** Only MindLyst has this configured currently.
+
+---
+
+#### 🔔 Notification Hubs (Mobile Push)
+| Variable | Required | Default | Used By | In KV? |
+|----------|----------|---------|---------|--------|
+| `AZURE_NH_CONNECTION_STRING` | Optional | - | platform-service (future) | ✅ Yes (mindlyst) |
+| `AZURE_NH_HUB_NAME` | Optional | - | platform-service (future) | N/A |
+
+**Purpose:** Push notifications for mobile apps.  
+**Note:** Only MindLyst has this configured currently.
+
+---
+
+## 🚨 Critical Gaps & Missing Secrets
+
+### Priority 1: Critical (Service Breaking) - LysnrAI Product
+
+These secrets are **required** for LysnrAI services to function and are **completely missing** from Key Vault:
+
+1. ❌ **`lysnr-cosmos-key`** → `COSMOS_KEY`
+   - **Impact:** Cannot connect to Cosmos DB
+   - **Affects:** All services (platform, extraction)
+   - **Action:** Add to Key Vault immediately
+
+2. ❌ **`lysnr-cosmos-endpoint`** → `COSMOS_ENDPOINT`
+   - **Impact:** Cannot locate Cosmos DB
+   - **Affects:** All services
+   - **Action:** Add to Key Vault immediately
+
+3. ❌ **`lysnr-jwt-secret`** → `JWT_SECRET`
+   - **Impact:** Cannot issue or verify JWTs
+   - **Affects:** All services (auth fails)
+   - **Action:** Add to Key Vault immediately
+
+4. ❌ **`lysnr-gemini-api-key`** → `GEMINI_API_KEY`
+   - **Impact:** Extraction service cannot function
+   - **Affects:** extraction-service
+   - **Action:** Add to Key Vault immediately
+
+5. ❌ **`lysnr-blob-connection-string`** → `AZURE_BLOB_CONNECTION_STRING`
+   - **Impact:** Cannot upload/download files
+   - **Affects:** platform-service (blob module)
+   - **Action:** Add to Key Vault immediately
+
+6. ❌ **`lysnr-blob-account-key`** → `AZURE_BLOB_ACCOUNT_KEY`
+   - **Impact:** Cannot generate SAS tokens
+   - **Affects:** platform-service (blob module)
+   - **Action:** Add to Key Vault immediately
+
+### Priority 2: High (Business Functionality)
+
+7. ❌ **`lysnr-stripe-secret-key`** → `STRIPE_SECRET_KEY`
+   - **Impact:** Cannot process payments
+   - **Affects:** platform-service (billing, subscriptions)
+   - **Action:** Add to Key Vault for production
+
+8. ❌ **`lysnr-stripe-webhook-secret`** → `STRIPE_WEBHOOK_SECRET`
+   - **Impact:** Cannot validate Stripe webhooks
+   - **Affects:** platform-service (payment events)
+   - **Action:** Add to Key Vault for production
+
+9. ❌ **`lysnr-billing-internal-key`** → `BILLING_INTERNAL_KEY`
+   - **Impact:** Cannot authenticate internal billing calls
+   - **Affects:** platform-service
+   - **Action:** Add to Key Vault for production
+
+10. ❌ **`lysnr-azure-speech-key`** → `AZURE_SPEECH_KEY`
+    - **Impact:** No speech features (if needed)
+    - **Affects:** LysnrAI product (voice features)
+    - **Action:** Add if voice features are required
+
+11. ❌ **`lysnr-azure-openai-key`** → `AZURE_OPENAI_KEY`
+    - **Impact:** Cannot use Azure OpenAI (Gemini is alternative)
+    - **Affects:** extraction-service
+    - **Action:** Add if Azure OpenAI is preferred over Gemini
+
+12. ❌ **`lysnr-azure-openai-endpoint`** → `AZURE_OPENAI_ENDPOINT`
+    - **Impact:** Cannot use Azure OpenAI
+    - **Affects:** extraction-service
+    - **Action:** Add if Azure OpenAI is preferred over Gemini
+
+### Priority 3: Medium (Optional)
+
+13. ❌ **`lysnr-seed-secret`** → `SEED_SECRET`
+    - **Impact:** Cannot run database seeding securely
+    - **Affects:** Development/testing
+    - **Action:** Add for secure seeding
+
+---
+
+## 📦 Product-Specific Secret Patterns
+
+### MindLyst Product (12 secrets) ✅
+**Prefix:** `mindlyst-*`  
+**Status:** Fully populated  
+**Secrets:**
+- Cosmos DB (endpoint, key, database)
+- Blob Storage (connection string)
+- Azure OpenAI (endpoint, key, deployment, api-version)
+- Azure Speech (key, region)
+- Notification Hub (connection string)
+- Application Insights (connection string)
+
+### MyWisprAI Product (7 secrets) ⚠️
+**Prefix:** `wispr-*`  
+**Status:** Partially populated  
+**Secrets:**
+- Azure OpenAI (endpoint, key, deployment)
+- Azure Speech (key, region)
+
+**Missing:**
+- Cosmos DB secrets
+- Blob Storage secrets
+- Stripe secrets
+- JWT secret
+
+### LysnrAI Product (0 secrets) ❌
+**Prefix:** `lysnr-*`  
+**Status:** Completely missing  
+**Expected:** 13 secrets  
+**Actual:** 0 secrets
+
+**This is the primary gap!** The platform services are designed for LysnrAI but have zero secrets in Key Vault.
+
+---
+
+## 🔧 Remediation Plan
+
+### Step 1: Seed LysnrAI Secrets (Immediate)
+
+Use the provided script with populated .env file:
+
+```bash
+# 1. Create .env with all secrets
+cp .env.example .env
+# Edit .env and fill in real values
+
+# 2. Run the seeding script
+./scripts/seed-keyvault.sh
+
+# 3. Verify secrets were created
+az keyvault secret list --vault-name kv-mywisprai --output table
+```
+
+The script will create all 13 `lysnr-*` secrets automatically.
+
+### Step 2: Update Services to Use Key Vault
+
+Add to each service's startup (already implemented in code):
+
+```typescript
+import { resolveKeyVaultSecrets, LYSNR_SECRETS } from '@bytelyst/config';
+
+// Before loading config
+await resolveKeyVaultSecrets(Object.values(LYSNR_SECRETS));
+
+// Then load config (will use KV values)
+const config = loadConfig();
+```
+
+### Step 3: Test Key Vault Integration
+
+```bash
+# Set Key Vault URL
+export AZURE_KEYVAULT_URL=https://kv-mywisprai.vault.azure.net
+
+# Remove local .env secrets to test KV
+mv .env .env.backup
+
+# Start service (should fetch from KV)
+pnpm --filter @lysnrai/platform-service dev
+
+# Check logs for KV resolution
+# Should see: "[keyvault] Resolving 13 secrets from Key Vault"
+```
+
+### Step 4: Verify Each Service
+
+| Service | Port | Test Endpoint | Expected Status |
+|---------|------|---------------|-----------------|
+| platform-service | 4003 | GET /health | 200 OK |
+| extraction-service | 4005 | GET /health | 200 OK |
+
+### Step 5: Production Checklist
+
+- [ ] Seed all `lysnr-*` secrets to Key Vault
+- [ ] Add Stripe secrets (test keys first, then prod)
+- [ ] Add Gemini API key
+- [ ] Configure managed identity for services
+- [ ] Remove connection strings from code/env files
+- [ ] Enable Key Vault audit logging
+- [ ] Set up secret rotation schedule
+- [ ] Document secret access patterns
+
+---
+
+## 🔄 Secret Rotation Strategy
+
+### Recommended Rotation Schedule:
+
+| Secret Type | Rotation Frequency | Method |
+|-------------|-------------------|--------|
+| Cosmos DB keys | Quarterly | Azure Portal → rotate keys → update KV |
+| JWT secret | Quarterly | Generate new → update KV → rolling deploy |
+| Stripe keys | Annually or on breach | Stripe Dashboard → update KV |
+| API keys (Gemini, OpenAI) | Annually or on breach | Provider portal → update KV |
+| Blob storage keys | Quarterly | Azure Portal → rotate keys → update KV |
+
+### Rotation Procedure:
+1. Generate new secret in Azure Portal / provider
+2. Update Key Vault with new value
+3. Services auto-fetch new secret on next restart (or use Key Vault rotation event)
+4. Monitor for errors
+5. Revoke old secret after 24h grace period
+
+---
+
+## 🔗 Related Files & Documentation
+
+| File | Purpose |
+|------|---------|
+| `packages/config/src/keyvault.ts` | Key Vault integration code |
+| `scripts/seed-keyvault.sh` | Script to populate Key Vault |
+| `.env.example` | Template for required variables |
+| `services/platform-service/src/lib/config.ts` | Platform service config schema |
+| `services/extraction-service/src/lib/config.ts` | Extraction service config schema |
+| `docs/devops/AZURE_PORTAL_SETUP.md` | Azure portal configuration guide |
+| `docs/devops/AZURE_KEY_VAULT_AND_SECRETS_ROTATION.md` | Key Vault setup guide |
+| `docs/devops/AZURE_RESOURCE_INVENTORY.md` | Complete Azure resource inventory |
+
+---
+
+## 📊 Summary Statistics
+
+### Environment Variables:
+- **Total Unique Variables:** 35+
+- **Required for Core Functionality:** 6
+- **Optional/Feature-Specific:** 29+
+
+### Key Vault Secrets:
+- **Total Secrets in KV:** 17
+- **MindLyst Secrets:** 12 ✅
+- **MyWisprAI Secrets:** 7 ⚠️
+- **LysnrAI Secrets:** 0 ❌
+- **Expected LysnrAI Secrets:** 13
+- **Coverage Gap:** 100%
+
+### Priority Actions:
+- 🔴 **Critical (6):** Cosmos DB, JWT, Gemini, Blob Storage
+- 🟠 **High (6):** Stripe, Speech, OpenAI, Billing internal key
+- 🟡 **Medium (1):** Seed secret
+
+---
+
+## 🚀 Quick Fix Commands
+
+### Get Azure Resource Keys (for seeding):
+
+```bash
+# Cosmos DB
+az cosmosdb keys list --name cosmos-mywisprai --resource-group rg-mywisprai --type keys
+
+# Blob Storage
+az storage account keys list --account-name bytelystblobs --resource-group rg-mywisprai
+
+# OpenAI
+az cognitiveservices account keys list --name mywisprai-openai-sweden --resource-group rg-mywisprai
+
+# Speech Services
+az cognitiveservices account keys list --name mywisprai-speech --resource-group rg-mywisprai
+```
+
+### Manually Add a Secret:
+
+```bash
+az keyvault secret set \
+  --vault-name kv-mywisprai \
+  --name lysnr-cosmos-key \
+  --value "your-actual-cosmos-key-here"
+```
+
+### Verify Secret Resolution in Code:
+
+```bash
+# Check if KV is being used
+export AZURE_KEYVAULT_URL=https://kv-mywisprai.vault.azure.net
+node -e "process.env.AZURE_KEYVAULT_URL && console.log('✅ KV enabled') || console.log('❌ KV disabled')"
+```
+
+---
+
+**Generated by:** Environment audit automation  
+**Maintained by:** ByteLyst DevOps Team  
+**Next Audit:** After LysnrAI secrets are seeded