diff --git a/docs/devops/ENVIRONMENT_VARIABLES_AND_KEYVAULT_AUDIT.md b/docs/devops/ENVIRONMENT_VARIABLES_AND_KEYVAULT_AUDIT.md new file mode 100644 index 00000000..440e195c --- /dev/null +++ b/docs/devops/ENVIRONMENT_VARIABLES_AND_KEYVAULT_AUDIT.md @@ -0,0 +1,485 @@ +# Environment Variables & Azure Key Vault Audit + +> **Last Updated:** 2026-02-15 +> **Purpose:** Complete audit of environment variables, Azure Key Vault secrets, and gap analysis + +--- + +## 🎯 Executive Summary + +### Critical Findings: +1. ❌ **ZERO LysnrAI secrets** exist in Azure Key Vault despite code expecting them +2. ✅ **MindLyst secrets** are fully populated (12 secrets) +3. ✅ **MyWisprAI secrets** are partially populated (7 secrets) +4. ⚠️ **Mismatch** between code expectations and actual Key Vault state +5. ⚠️ **Missing Stripe secrets** for billing functionality +6. ⚠️ **Missing Gemini API key** for extraction service + +--- + +## 📊 Key Vault Current State + +### Azure Key Vault: `kv-mywisprai` +**Location:** East US +**Total Secrets:** 17 + +| Secret Name | Product | Purpose | Created | Status | +|-------------|---------|---------|---------|--------| +| `mindlyst-appinsights-connection-string` | MindLyst | Application Insights telemetry | 2026-02-14 | ✅ Active | +| `mindlyst-blob-connection-string` | MindLyst | Blob Storage access | 2026-02-14 | ✅ Active | +| `mindlyst-cosmos-database` | MindLyst | Cosmos DB database name | 2026-02-14 | ✅ Active | +| `mindlyst-cosmos-endpoint` | MindLyst | Cosmos DB endpoint | 2026-02-14 | ✅ Active | +| `mindlyst-cosmos-key` | MindLyst | Cosmos DB primary key | 2026-02-14 | ✅ Active | +| `mindlyst-notification-hub-connection-string` | MindLyst | Push notifications | 2026-02-14 | ✅ Active | +| `mindlyst-openai-api-version` | MindLyst | OpenAI API version | 2026-02-14 | ✅ Active | +| `mindlyst-openai-deployment` | MindLyst | OpenAI deployment name | 2026-02-14 | ✅ Active | +| `mindlyst-openai-endpoint` | MindLyst | Azure OpenAI endpoint | 2026-02-14 | ✅ Active | +| `mindlyst-openai-key` | MindLyst | Azure OpenAI key | 2026-02-14 | ✅ Active | +| `mindlyst-speech-key` | MindLyst | Azure Speech Services key | 2026-02-14 | ✅ Active | +| `mindlyst-speech-region` | MindLyst | Azure Speech region | 2026-02-14 | ✅ Active | +| `wispr-azure-openai-deployment` | MyWisprAI | OpenAI deployment name | 2026-02-07 | ✅ Active | +| `wispr-azure-openai-endpoint` | MyWisprAI | Azure OpenAI endpoint | 2026-02-07 | ✅ Active | +| `wispr-azure-openai-key` | MyWisprAI | Azure OpenAI key | 2026-02-07 | ✅ Active | +| `wispr-azure-speech-key` | MyWisprAI | Azure Speech Services key | 2026-02-07 | ✅ Active | +| `wispr-azure-speech-region` | MyWisprAI | Azure Speech region | 2026-02-07 | ✅ Active | + +--- + +## 🔍 Code Expectations vs Reality + +### Expected Secrets (from `packages/config/src/keyvault.ts`) + +The `LYSNR_SECRETS` constant defines these mappings: + +| Key Vault Secret Name | Environment Variable | Status in KV | Priority | +|-----------------------|---------------------|--------------|----------| +| `lysnr-cosmos-key` | `COSMOS_KEY` | ❌ **MISSING** | 🔴 Critical | +| `lysnr-cosmos-endpoint` | `COSMOS_ENDPOINT` | ❌ **MISSING** | 🔴 Critical | +| `lysnr-jwt-secret` | `JWT_SECRET` | ❌ **MISSING** | 🔴 Critical | +| `lysnr-stripe-secret-key` | `STRIPE_SECRET_KEY` | ❌ **MISSING** | 🔴 Critical | +| `lysnr-stripe-webhook-secret` | `STRIPE_WEBHOOK_SECRET` | ❌ **MISSING** | 🔴 Critical | +| `lysnr-billing-internal-key` | `BILLING_INTERNAL_KEY` | ❌ **MISSING** | 🟠 High | +| `lysnr-blob-connection-string` | `AZURE_BLOB_CONNECTION_STRING` | ❌ **MISSING** | 🔴 Critical | +| `lysnr-blob-account-key` | `AZURE_BLOB_ACCOUNT_KEY` | ❌ **MISSING** | 🔴 Critical | +| `lysnr-gemini-api-key` | `GEMINI_API_KEY` | ❌ **MISSING** | 🔴 Critical | +| `lysnr-seed-secret` | `SEED_SECRET` | ❌ **MISSING** | 🟡 Medium | +| `lysnr-azure-speech-key` | `AZURE_SPEECH_KEY` | ❌ **MISSING** | 🟠 High | +| `lysnr-azure-openai-key` | `AZURE_OPENAI_KEY` | ❌ **MISSING** | 🟠 High | +| `lysnr-azure-openai-endpoint` | `AZURE_OPENAI_ENDPOINT` | ❌ **MISSING** | 🟠 High | + +**Total Expected:** 13 secrets +**Total Missing:** 13 secrets (100%) + +--- + +## 📋 Complete Environment Variable Inventory + +### Core Platform Variables (from `.env.example`) + +#### 🔐 Azure Key Vault +| Variable | Required | Default | Used By | In KV? | +|----------|----------|---------|---------|--------| +| `AZURE_KEYVAULT_URL` | Optional | - | All services | N/A | + +**Purpose:** Enable Key Vault secret resolution. Falls back to env vars if not set. + +--- + +#### 🗄️ Azure Cosmos DB +| Variable | Required | Default | Used By | In KV? | +|----------|----------|---------|---------|--------| +| `COSMOS_ENDPOINT` | ✅ Yes | - | All services | ❌ No (lysnr) | +| `COSMOS_KEY` | ✅ Yes | - | All services | ❌ No (lysnr) | +| `COSMOS_DATABASE` | No | `lysnrai` | All services | N/A | + +**Purpose:** Database connection for all products (lysnrai, mindlyst, mywisprai databases). + +--- + +#### 🔑 Authentication & Security +| Variable | Required | Default | Used By | In KV? | +|----------|----------|---------|---------|--------| +| `JWT_SECRET` | ✅ Yes | - | platform-service, extraction-service | ❌ No | + +**Purpose:** JWT token signing and verification across all services. + +--- + +#### 💾 Azure Blob Storage +| Variable | Required | Default | Used By | In KV? | +|----------|----------|---------|---------|--------| +| `AZURE_BLOB_CONNECTION_STRING` | Optional | - | platform-service | ❌ No | +| `AZURE_BLOB_ACCOUNT_NAME` | Optional | `bytelystblobs` | platform-service | N/A | +| `AZURE_BLOB_ACCOUNT_KEY` | Optional | - | platform-service | ❌ No | + +**Purpose:** File uploads, avatar storage, document storage. +**Note:** Only needed for platform-service blob module. + +--- + +#### 💳 Stripe (Billing & Subscriptions) +| Variable | Required | Default | Used By | In KV? | +|----------|----------|---------|---------|--------| +| `STRIPE_SECRET_KEY` | Optional | - | platform-service | ❌ No | +| `STRIPE_WEBHOOK_SECRET` | Optional | - | platform-service | ❌ No | +| `STRIPE_PRICE_PRO` | Optional | - | platform-service | N/A | +| `STRIPE_PRICE_ENTERPRISE` | Optional | - | platform-service | N/A | +| `BILLING_INTERNAL_KEY` | Optional | - | platform-service | ❌ No | + +**Purpose:** Subscription management, payment processing, webhook validation. +**Status:** ❌ **Required for production but completely missing from Key Vault** + +--- + +#### 🤖 AI Services - Extraction +| Variable | Required | Default | Used By | In KV? | +|----------|----------|---------|---------|--------| +| `GEMINI_API_KEY` | ✅ Yes | - | extraction-service (Python sidecar) | ❌ No | +| `PYTHON_SIDECAR_URL` | No | `http://localhost:4006` | extraction-service | N/A | +| `DEFAULT_MODEL_ID` | No | `gemini-2.5-flash` | extraction-service | N/A | + +**Purpose:** Text extraction, content analysis, AI-powered features. +**Status:** ❌ **Critical for extraction service functionality** + +--- + +#### 🧠 Azure OpenAI (Optional) +| Variable | Required | Default | Used By | In KV? | +|----------|----------|---------|---------|--------| +| `AZURE_OPENAI_KEY` | Optional | - | extraction-service | ❌ No (lysnr) | +| `AZURE_OPENAI_ENDPOINT` | Optional | - | extraction-service | ❌ No (lysnr) | +| `AZURE_OPENAI_DEPLOYMENT_NAME` | Optional | - | extraction-service | N/A | + +**Purpose:** Alternative to Gemini for text extraction. +**Existing:** ✅ Available for `wispr` and `mindlyst` products in Key Vault. + +--- + +#### 🎤 Azure Speech Services +| Variable | Required | Default | Used By | In KV? | +|----------|----------|---------|---------|--------| +| `AZURE_SPEECH_KEY` | Optional | - | LysnrAI product | ❌ No (lysnr) | +| `AZURE_SPEECH_REGION` | Optional | `eastus` | LysnrAI product | N/A | + +**Purpose:** Speech-to-text, text-to-speech for voice AI features. +**Existing:** ✅ Available for `wispr` and `mindlyst` products in Key Vault. + +--- + +#### 🚀 Service Configuration +| Variable | Required | Default | Used By | In KV? | +|----------|----------|---------|---------|--------| +| `PORT` | No | varies | All services | N/A | +| `HOST` | No | `0.0.0.0` | All services | N/A | +| `NODE_ENV` | No | `development` | All services | N/A | +| `SERVICE_NAME` | No | auto | All services | N/A | +| `CORS_ORIGIN` | No | `*` (dev) | All services | N/A | +| `DEFAULT_PRODUCT_ID` | No | `lysnrai` | All services | N/A | + +**Purpose:** Runtime configuration, not secrets. + +--- + +#### 📊 Application Insights (Monitoring) +| Variable | Required | Default | Used By | In KV? | +|----------|----------|---------|---------|--------| +| `APPLICATIONINSIGHTS_CONNECTION_STRING` | Optional | - | All services (future) | ✅ Yes (mindlyst) | +| `APPINSIGHTS_INSTRUMENTATIONKEY` | Optional | - | All services (future) | N/A | + +**Purpose:** Telemetry, logging, monitoring. +**Note:** Only MindLyst has this configured currently. + +--- + +#### 🔔 Notification Hubs (Mobile Push) +| Variable | Required | Default | Used By | In KV? | +|----------|----------|---------|---------|--------| +| `AZURE_NH_CONNECTION_STRING` | Optional | - | platform-service (future) | ✅ Yes (mindlyst) | +| `AZURE_NH_HUB_NAME` | Optional | - | platform-service (future) | N/A | + +**Purpose:** Push notifications for mobile apps. +**Note:** Only MindLyst has this configured currently. + +--- + +## 🚨 Critical Gaps & Missing Secrets + +### Priority 1: Critical (Service Breaking) - LysnrAI Product + +These secrets are **required** for LysnrAI services to function and are **completely missing** from Key Vault: + +1. ❌ **`lysnr-cosmos-key`** → `COSMOS_KEY` + - **Impact:** Cannot connect to Cosmos DB + - **Affects:** All services (platform, extraction) + - **Action:** Add to Key Vault immediately + +2. ❌ **`lysnr-cosmos-endpoint`** → `COSMOS_ENDPOINT` + - **Impact:** Cannot locate Cosmos DB + - **Affects:** All services + - **Action:** Add to Key Vault immediately + +3. ❌ **`lysnr-jwt-secret`** → `JWT_SECRET` + - **Impact:** Cannot issue or verify JWTs + - **Affects:** All services (auth fails) + - **Action:** Add to Key Vault immediately + +4. ❌ **`lysnr-gemini-api-key`** → `GEMINI_API_KEY` + - **Impact:** Extraction service cannot function + - **Affects:** extraction-service + - **Action:** Add to Key Vault immediately + +5. ❌ **`lysnr-blob-connection-string`** → `AZURE_BLOB_CONNECTION_STRING` + - **Impact:** Cannot upload/download files + - **Affects:** platform-service (blob module) + - **Action:** Add to Key Vault immediately + +6. ❌ **`lysnr-blob-account-key`** → `AZURE_BLOB_ACCOUNT_KEY` + - **Impact:** Cannot generate SAS tokens + - **Affects:** platform-service (blob module) + - **Action:** Add to Key Vault immediately + +### Priority 2: High (Business Functionality) + +7. ❌ **`lysnr-stripe-secret-key`** → `STRIPE_SECRET_KEY` + - **Impact:** Cannot process payments + - **Affects:** platform-service (billing, subscriptions) + - **Action:** Add to Key Vault for production + +8. ❌ **`lysnr-stripe-webhook-secret`** → `STRIPE_WEBHOOK_SECRET` + - **Impact:** Cannot validate Stripe webhooks + - **Affects:** platform-service (payment events) + - **Action:** Add to Key Vault for production + +9. ❌ **`lysnr-billing-internal-key`** → `BILLING_INTERNAL_KEY` + - **Impact:** Cannot authenticate internal billing calls + - **Affects:** platform-service + - **Action:** Add to Key Vault for production + +10. ❌ **`lysnr-azure-speech-key`** → `AZURE_SPEECH_KEY` + - **Impact:** No speech features (if needed) + - **Affects:** LysnrAI product (voice features) + - **Action:** Add if voice features are required + +11. ❌ **`lysnr-azure-openai-key`** → `AZURE_OPENAI_KEY` + - **Impact:** Cannot use Azure OpenAI (Gemini is alternative) + - **Affects:** extraction-service + - **Action:** Add if Azure OpenAI is preferred over Gemini + +12. ❌ **`lysnr-azure-openai-endpoint`** → `AZURE_OPENAI_ENDPOINT` + - **Impact:** Cannot use Azure OpenAI + - **Affects:** extraction-service + - **Action:** Add if Azure OpenAI is preferred over Gemini + +### Priority 3: Medium (Optional) + +13. ❌ **`lysnr-seed-secret`** → `SEED_SECRET` + - **Impact:** Cannot run database seeding securely + - **Affects:** Development/testing + - **Action:** Add for secure seeding + +--- + +## 📦 Product-Specific Secret Patterns + +### MindLyst Product (12 secrets) ✅ +**Prefix:** `mindlyst-*` +**Status:** Fully populated +**Secrets:** +- Cosmos DB (endpoint, key, database) +- Blob Storage (connection string) +- Azure OpenAI (endpoint, key, deployment, api-version) +- Azure Speech (key, region) +- Notification Hub (connection string) +- Application Insights (connection string) + +### MyWisprAI Product (7 secrets) ⚠️ +**Prefix:** `wispr-*` +**Status:** Partially populated +**Secrets:** +- Azure OpenAI (endpoint, key, deployment) +- Azure Speech (key, region) + +**Missing:** +- Cosmos DB secrets +- Blob Storage secrets +- Stripe secrets +- JWT secret + +### LysnrAI Product (0 secrets) ❌ +**Prefix:** `lysnr-*` +**Status:** Completely missing +**Expected:** 13 secrets +**Actual:** 0 secrets + +**This is the primary gap!** The platform services are designed for LysnrAI but have zero secrets in Key Vault. + +--- + +## 🔧 Remediation Plan + +### Step 1: Seed LysnrAI Secrets (Immediate) + +Use the provided script with populated .env file: + +```bash +# 1. Create .env with all secrets +cp .env.example .env +# Edit .env and fill in real values + +# 2. Run the seeding script +./scripts/seed-keyvault.sh + +# 3. Verify secrets were created +az keyvault secret list --vault-name kv-mywisprai --output table +``` + +The script will create all 13 `lysnr-*` secrets automatically. + +### Step 2: Update Services to Use Key Vault + +Add to each service's startup (already implemented in code): + +```typescript +import { resolveKeyVaultSecrets, LYSNR_SECRETS } from '@bytelyst/config'; + +// Before loading config +await resolveKeyVaultSecrets(Object.values(LYSNR_SECRETS)); + +// Then load config (will use KV values) +const config = loadConfig(); +``` + +### Step 3: Test Key Vault Integration + +```bash +# Set Key Vault URL +export AZURE_KEYVAULT_URL=https://kv-mywisprai.vault.azure.net + +# Remove local .env secrets to test KV +mv .env .env.backup + +# Start service (should fetch from KV) +pnpm --filter @lysnrai/platform-service dev + +# Check logs for KV resolution +# Should see: "[keyvault] Resolving 13 secrets from Key Vault" +``` + +### Step 4: Verify Each Service + +| Service | Port | Test Endpoint | Expected Status | +|---------|------|---------------|-----------------| +| platform-service | 4003 | GET /health | 200 OK | +| extraction-service | 4005 | GET /health | 200 OK | + +### Step 5: Production Checklist + +- [ ] Seed all `lysnr-*` secrets to Key Vault +- [ ] Add Stripe secrets (test keys first, then prod) +- [ ] Add Gemini API key +- [ ] Configure managed identity for services +- [ ] Remove connection strings from code/env files +- [ ] Enable Key Vault audit logging +- [ ] Set up secret rotation schedule +- [ ] Document secret access patterns + +--- + +## 🔄 Secret Rotation Strategy + +### Recommended Rotation Schedule: + +| Secret Type | Rotation Frequency | Method | +|-------------|-------------------|--------| +| Cosmos DB keys | Quarterly | Azure Portal → rotate keys → update KV | +| JWT secret | Quarterly | Generate new → update KV → rolling deploy | +| Stripe keys | Annually or on breach | Stripe Dashboard → update KV | +| API keys (Gemini, OpenAI) | Annually or on breach | Provider portal → update KV | +| Blob storage keys | Quarterly | Azure Portal → rotate keys → update KV | + +### Rotation Procedure: +1. Generate new secret in Azure Portal / provider +2. Update Key Vault with new value +3. Services auto-fetch new secret on next restart (or use Key Vault rotation event) +4. Monitor for errors +5. Revoke old secret after 24h grace period + +--- + +## 🔗 Related Files & Documentation + +| File | Purpose | +|------|---------| +| `packages/config/src/keyvault.ts` | Key Vault integration code | +| `scripts/seed-keyvault.sh` | Script to populate Key Vault | +| `.env.example` | Template for required variables | +| `services/platform-service/src/lib/config.ts` | Platform service config schema | +| `services/extraction-service/src/lib/config.ts` | Extraction service config schema | +| `docs/devops/AZURE_PORTAL_SETUP.md` | Azure portal configuration guide | +| `docs/devops/AZURE_KEY_VAULT_AND_SECRETS_ROTATION.md` | Key Vault setup guide | +| `docs/devops/AZURE_RESOURCE_INVENTORY.md` | Complete Azure resource inventory | + +--- + +## 📊 Summary Statistics + +### Environment Variables: +- **Total Unique Variables:** 35+ +- **Required for Core Functionality:** 6 +- **Optional/Feature-Specific:** 29+ + +### Key Vault Secrets: +- **Total Secrets in KV:** 17 +- **MindLyst Secrets:** 12 ✅ +- **MyWisprAI Secrets:** 7 ⚠️ +- **LysnrAI Secrets:** 0 ❌ +- **Expected LysnrAI Secrets:** 13 +- **Coverage Gap:** 100% + +### Priority Actions: +- 🔴 **Critical (6):** Cosmos DB, JWT, Gemini, Blob Storage +- 🟠 **High (6):** Stripe, Speech, OpenAI, Billing internal key +- 🟡 **Medium (1):** Seed secret + +--- + +## 🚀 Quick Fix Commands + +### Get Azure Resource Keys (for seeding): + +```bash +# Cosmos DB +az cosmosdb keys list --name cosmos-mywisprai --resource-group rg-mywisprai --type keys + +# Blob Storage +az storage account keys list --account-name bytelystblobs --resource-group rg-mywisprai + +# OpenAI +az cognitiveservices account keys list --name mywisprai-openai-sweden --resource-group rg-mywisprai + +# Speech Services +az cognitiveservices account keys list --name mywisprai-speech --resource-group rg-mywisprai +``` + +### Manually Add a Secret: + +```bash +az keyvault secret set \ + --vault-name kv-mywisprai \ + --name lysnr-cosmos-key \ + --value "your-actual-cosmos-key-here" +``` + +### Verify Secret Resolution in Code: + +```bash +# Check if KV is being used +export AZURE_KEYVAULT_URL=https://kv-mywisprai.vault.azure.net +node -e "process.env.AZURE_KEYVAULT_URL && console.log('✅ KV enabled') || console.log('❌ KV disabled')" +``` + +--- + +**Generated by:** Environment audit automation +**Maintained by:** ByteLyst DevOps Team +**Next Audit:** After LysnrAI secrets are seeded