diff --git a/docs/devops/END_TO_END_ENCRYPTION_ROADMAP.md b/docs/devops/END_TO_END_ENCRYPTION_ROADMAP.md index 6a69902b..d8264488 100644 --- a/docs/devops/END_TO_END_ENCRYPTION_ROADMAP.md +++ b/docs/devops/END_TO_END_ENCRYPTION_ROADMAP.md @@ -1,7 +1,7 @@ # ByteLyst — End-to-End Encryption Implementation Roadmap > **Purpose:** Phased implementation plan for encryption across the ByteLyst ecosystem. -> **Status:** Phase 1 + Phase 2 (Sprint 3 + Sprint 4) COMPLETE — 9 backends + native SDKs + client-encrypt + secure-storage-web +> **Status:** Phase 1 + Phase 2 COMPLETE (10 backends + native SDKs + client-encrypt + secure-storage-web) + Sprint 6.2 migration CLI > **Author:** AI Architecture Review > **Last updated:** 2026-03-21 > **Design doc:** [`END_TO_END_ENCRYPTION_DESIGN.md`](END_TO_END_ENCRYPTION_DESIGN.md) @@ -233,14 +233,16 @@ Week 1-2 Week 3-4 Week 5-6 Week 7-8 Week 9-10 Week 11-14 #### 3.2 MindLyst Backend (port 4014) -- [ ] **3.2.1** Add dependency + create encryptor singleton -- [ ] **3.2.2** Encrypt `content` and `voiceTranscriptText` in memory_items module -- [ ] **3.2.3** Encrypt array fields in reflections module - - `repeatedThemes` (string[]), `suggestedAdjustments` (string[]), `vsLastWeek.summary` (string) +- [x] **3.2.1** Add dependency + create encryptor singleton +- [x] **3.2.2** Encrypt `rawContent` and `triageResult.summary` in memory_items module +- [x] **3.2.3** Encrypt array fields in reflections module + - `repeatedThemes` (string[]), `postponedItems` (string[]), `roleImbalanceSignals` (string[]), `suggestedAdjustments` (string[]), `vsLastWeek.summary` (string) - Array fields: JSON-serialize → encrypt → store as single EncryptedField each -- [ ] **3.2.4** Update tests +- [x] **3.2.4** Encrypt daily-briefs: `greeting`, `priorityItems`, `brainSummaries`, `streakMessage`, `motivationalQuote` +- [x] **3.2.5** Encrypt brains: `rolePrompt` +- [x] **3.2.6** Update tests -**Commit:** `feat(mindlyst): encrypt memory content, voice transcripts, and reflections` +**Commit:** `feat(mindlyst): encrypt sensitive fields across all modules` #### 3.3 NomGap Backend (port 4013) @@ -263,14 +265,15 @@ Week 1-2 Week 3-4 Week 5-6 Week 7-8 Week 9-10 Week 11-14 #### 3.5 Remaining Backends (Low Priority — Defer or Skip) -| Product | Backend Port | Encrypted Fields | Decision | -| ---------------------- | ---------------------------------------------------------------------- | --------------------------------------------------- | -------- | -| **FlowMonk** (4017) | `tasks.description` | ✅ Done (Sprint 4) — 211 tests | -| **ChronoMind** (4011) | `timers.description`, `routines.description`, `routines.steps[].notes` | ✅ Done — 182 tests | -| **PeakPulse** (4010) | `peak-sessions.notes` | ✅ Done — 65 tests | -| **LocalMemGPT** (4019) | `messages.content` | Deferred — SQLite raw SQL, needs different approach | +| Product | Backend Port | Encrypted Fields | Decision | +| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | -------- | +| **FlowMonk** (4017) | `tasks.description` | ✅ Done (Sprint 4) — 211 tests | +| **ChronoMind** (4011) | `timers.description`, `routines.description`, `routines.steps[].notes`, `shared-timers.description`, `webhooks.secret`, `webhooks.description` | ✅ Done — 182 tests | +| **PeakPulse** (4010) | `peak-sessions.notes` | ✅ Done — 65 tests | +| **MindLyst** (4014) | `memory.rawContent`, `memory.triageResult.summary`, reflections (5 fields), daily-briefs (5 fields), `brains.rolePrompt` | ✅ Done | +| **LocalMemGPT** (4019) | `messages.content`, `conversations.system_prompt` | Deferred — SQLite raw SQL (better-sqlite3), needs SQLite-specific encryption approach | -**Sprint 3 deliverable:** 9 product backends encrypted (LysnrAI, JarvisJr, NoteLett, NomGap, ActionTrail, FlowMonk, ChronoMind, PeakPulse). Only MindLyst (KMP) and LocalMemGPT (SQLite) deferred. +**Sprint 3 deliverable:** 10 product backends encrypted (LysnrAI, JarvisJr, NoteLett, NomGap, ActionTrail, FlowMonk, ChronoMind, PeakPulse, MindLyst). Only LocalMemGPT (SQLite) deferred to Sprint 5. --- @@ -449,36 +452,29 @@ Week 1-2 Week 3-4 Week 5-6 Week 7-8 Week 9-10 Week 11-14 #### 6.2 Encryption Migration CLI -- [ ] **6.2.1** Create `scripts/encrypt-migrate.ts` — CLI tool for batch encrypting existing plaintext +- [x] **6.2.1** Create `scripts/encrypt-migrate.ts` — CLI tool for batch encrypting existing plaintext ```bash - npx tsx scripts/encrypt-migrate.ts \ - --product lysnrai \ - --container transcripts \ - --field transcriptText \ - --partition-key /userId \ - --batch-size 100 \ - --dry-run + npx tsx scripts/encrypt-migrate.ts --product lysnrai --dry-run + npx tsx scripts/encrypt-migrate.ts --product all --dry-run + npx tsx scripts/encrypt-migrate.ts --product chronomind --container timers + npx tsx scripts/encrypt-migrate.ts --product mindlyst --batch-size 50 --verbose ``` - - Uses `@bytelyst/field-encrypt` migration helpers - - Progress bar, resumable (tracks last processed ID) - - Dry run mode (report count without modifying) + - Uses `@bytelyst/field-encrypt` `migrateDocuments()` helper (idempotent via `__encrypted` sentinel) + - Connects directly to Cosmos DB via `@azure/cosmos` + - Per-product configs embedded: 10 products, 20 containers, 40+ fields + - Dry-run mode (count without modifying), verbose per-doc progress + - Nested field support (e.g., `triageResult.summary`) + - JSON-serialized arrays/objects (e.g., reflections string arrays, brainSummaries Record) + - Per-field breakdown table in summary output -- [ ] **6.2.2** Create per-product migration scripts - ``` - scripts/migrations/ - ├── encrypt-lysnrai.sh - ├── encrypt-jarvisjr.sh - ├── encrypt-notelett.sh - ├── encrypt-mindlyst.sh - ├── encrypt-nomgap.sh - ├── encrypt-actiontrail.sh - ├── encrypt-flowmonk.sh - └── encrypt-localmemgpt.sh - ``` +- [x] **6.2.2** Embedded per-product migration configs (no separate shell scripts needed) + - Products: lysnrai, jarvisjr, notelett, nomgap, actiontrail, flowmonk, chronomind, peakpulse, mindlyst + - Each config specifies: productId, mekName, containers[], fields[] with context + userIdField + jsonSerialize + - Run `--product all --dry-run` for full ecosystem audit -**Commit:** `feat(devops): encryption migration CLI and per-product migration scripts` +**Commit:** `feat(devops): encryption migration CLI with embedded product configs` #### 6.3 Monitoring & Audit diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 708f8573..083a29fc 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -393,6 +393,12 @@ importers: packages/celebrations: {} + packages/client-encrypt: + devDependencies: + vitest: + specifier: ^3.0.0 + version: 3.2.4(@types/debug@4.1.12)(@types/node@22.19.11)(happy-dom@18.0.1)(jiti@2.6.1)(jsdom@28.0.0(@noble/hashes@1.8.0))(lightningcss@1.31.1)(msw@2.12.10(@types/node@22.19.11)(typescript@5.9.3))(terser@5.46.0)(tsx@4.21.0)(yaml@2.8.2) + packages/config: dependencies: zod: @@ -604,6 +610,25 @@ importers: specifier: ^3.0.0 version: 3.2.4(@types/debug@4.1.12)(@types/node@22.19.11)(happy-dom@18.0.1)(jiti@2.6.1)(jsdom@28.0.0(@noble/hashes@1.8.0))(lightningcss@1.31.1)(msw@2.12.10(@types/node@22.19.11)(typescript@5.9.3))(terser@5.46.0)(tsx@4.21.0)(yaml@2.8.2) + packages/field-encrypt: + dependencies: + '@azure/identity': + specifier: '>=4.0.0' + version: 4.13.0 + '@azure/keyvault-keys': + specifier: '>=4.8.0' + version: 4.10.0(@azure/core-client@1.10.1) + '@bytelyst/errors': + specifier: workspace:* + version: link:../errors + devDependencies: + vitest: + specifier: ^3.0.0 + version: 3.2.4(@types/debug@4.1.12)(@types/node@22.19.11)(happy-dom@18.0.1)(jiti@2.6.1)(jsdom@28.0.0(@noble/hashes@1.8.0))(lightningcss@1.31.1)(msw@2.12.10(@types/node@22.19.11)(typescript@5.9.3))(terser@5.46.0)(tsx@4.21.0)(yaml@2.8.2) + zod: + specifier: ^3.24.0 + version: 3.25.76 + packages/gentle-notifications: {} packages/kill-switch-client: {} @@ -701,6 +726,15 @@ importers: packages/referral-client: {} + packages/secure-storage-web: + devDependencies: + fake-indexeddb: + specifier: ^6.0.0 + version: 6.2.5 + vitest: + specifier: ^3.0.0 + version: 3.2.4(@types/debug@4.1.12)(@types/node@22.19.11)(happy-dom@18.0.1)(jiti@2.6.1)(jsdom@28.0.0(@noble/hashes@1.8.0))(lightningcss@1.31.1)(msw@2.12.10(@types/node@22.19.11)(typescript@5.9.3))(terser@5.46.0)(tsx@4.21.0)(yaml@2.8.2) + packages/speech: devDependencies: typescript: @@ -762,6 +796,25 @@ importers: packages/webhook-dispatch: {} + scripts: + dependencies: + '@azure/cosmos': + specifier: ^4.2.0 + version: 4.9.1(@azure/core-client@1.10.1) + '@bytelyst/field-encrypt': + specifier: workspace:* + version: link:../packages/field-encrypt + devDependencies: + '@types/node': + specifier: ^22.12.0 + version: 22.19.11 + tsx: + specifier: ^4.19.2 + version: 4.21.0 + typescript: + specifier: ^5.7.3 + version: 5.9.3 + services/extraction-service: dependencies: '@azure/cosmos': @@ -8082,6 +8135,13 @@ packages: integrity: sha512-UOiS2in6/Q0FK0R0q6UY9vYpQ21mr/Qn1KOnte7vsACuNJf514WvCCUHSRCPcgjPT2bAhNIJdlE6bVap1GKmeg==, } + fake-indexeddb@6.2.5: + resolution: + { + integrity: sha512-CGnyrvbhPlWYMngksqrSSUT1BAVP49dZocrHuK0SvtR0D5TMs5wP0o3j7jexDJW01KSadjBp1M/71o/KR3nD1w==, + } + engines: { node: '>=18' } + fast-decode-uri-component@1.0.1: resolution: { @@ -19522,6 +19582,8 @@ snapshots: extendable-error@0.1.7: {} + fake-indexeddb@6.2.5: {} + fast-decode-uri-component@1.0.1: {} fast-deep-equal@3.1.3: {} diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml index 95793266..2f5a7d69 100644 --- a/pnpm-workspace.yaml +++ b/pnpm-workspace.yaml @@ -2,3 +2,4 @@ packages: - 'packages/*' - 'services/*' - 'dashboards/*' + - 'scripts' diff --git a/scripts/encrypt-migrate.ts b/scripts/encrypt-migrate.ts new file mode 100644 index 00000000..23fa350a --- /dev/null +++ b/scripts/encrypt-migrate.ts @@ -0,0 +1,644 @@ +#!/usr/bin/env npx tsx +/* eslint-disable no-console */ +/** + * Encryption Migration CLI — Batch-encrypt existing plaintext Cosmos documents. + * + * Uses @bytelyst/field-encrypt migrateDocuments() to idempotently encrypt + * plaintext fields across all product backends. + * + * Usage: + * npx tsx scripts/encrypt-migrate.ts --product lysnrai --dry-run + * npx tsx scripts/encrypt-migrate.ts --product lysnrai --container transcripts + * npx tsx scripts/encrypt-migrate.ts --product all --dry-run + * npx tsx scripts/encrypt-migrate.ts --product chronomind --batch-size 50 + * + * Required env vars: + * COSMOS_ENDPOINT — Azure Cosmos DB endpoint + * COSMOS_KEY — Azure Cosmos DB key + * COSMOS_DATABASE — Database name (default: lysnrai) + * FIELD_ENCRYPT_KEY_PROVIDER — 'memory' | 'env' | 'akv' (default: memory) + * FIELD_ENCRYPT_KEY — Hex key (for 'env' provider) + * AZURE_KEYVAULT_URL — AKV URL (for 'akv' provider) + * + * Per-product MEK names are defined in migration configs. + */ + +import { CosmosClient, type Container } from '@azure/cosmos'; +import { + createFieldEncryptor, + migrateDocuments, + type MigrationResult, + type FieldEncryptor, + type EncryptedField, +} from '@bytelyst/field-encrypt'; + +// ── Types ────────────────────────────────────────────── + +interface FieldSpec { + /** The field path (dot-separated for nested, e.g. 'triageResult.summary'). */ + field: string; + /** Encryption context label. */ + context: string; + /** The property on the doc that holds the userId for DEK scoping. Default: 'userId'. */ + userIdField?: string; + /** If true, the field is a string[] or Record — JSON-serialize before encrypting. */ + jsonSerialize?: boolean; +} + +interface ContainerMigrationConfig { + /** Cosmos container name. */ + container: string; + /** Partition key path (e.g. '/userId'). */ + partitionKey: string; + /** Fields to encrypt in this container. */ + fields: FieldSpec[]; +} + +interface ProductMigrationConfig { + /** Product ID. */ + productId: string; + /** MEK name for this product. */ + mekName: string; + /** Containers to migrate. */ + containers: ContainerMigrationConfig[]; +} + +// ── Field accessor helpers ──────────────────────────── + +function getNestedField(doc: Record, path: string): unknown { + const parts = path.split('.'); + let current: unknown = doc; + for (const part of parts) { + if (current == null || typeof current !== 'object') return undefined; + current = (current as Record)[part]; + } + return current; +} + +function setNestedField( + doc: Record, + path: string, + value: unknown +): Record { + const parts = path.split('.'); + const result = { ...doc }; + + if (parts.length === 1) { + result[parts[0]] = value; + return result; + } + + // Deep clone the nested path + let current = result; + for (let i = 0; i < parts.length - 1; i++) { + const nested = current[parts[i]]; + if (nested == null || typeof nested !== 'object') return result; + current[parts[i]] = { ...(nested as Record) }; + current = current[parts[i]] as Record; + } + current[parts[parts.length - 1]] = value; + return result; +} + +// ── Product migration configs ───────────────────────── + +const PRODUCT_CONFIGS: ProductMigrationConfig[] = [ + // ── LysnrAI (port 4015) ── + { + productId: 'lysnrai', + mekName: 'lysnr-mek', + containers: [ + { + container: 'transcripts', + partitionKey: '/userId', + fields: [{ field: 'transcriptText', context: 'transcripts' }], + }, + { + container: 'sessions', + partitionKey: '/userId', + fields: [{ field: 'notes', context: 'sessions' }], + }, + ], + }, + // ── JarvisJr (port 4012) ── + { + productId: 'jarvisjr', + mekName: 'jarvisjr-mek', + containers: [ + { + container: 'jarvis_sessions', + partitionKey: '/userId', + fields: [{ field: 'notes', context: 'jarvis-sessions' }], + }, + { + container: 'jarvis_memory', + partitionKey: '/agentId', + fields: [{ field: 'content', context: 'jarvis-memory', userIdField: 'agentId' }], + }, + ], + }, + // ── NoteLett (port 4016) ── + { + productId: 'notelett', + mekName: 'notelett-mek', + containers: [ + { + container: 'notes', + partitionKey: '/workspaceId', + fields: [{ field: 'body', context: 'notes', userIdField: 'createdBy' }], + }, + ], + }, + // ── NomGap (port 4013) ── + { + productId: 'nomgap', + mekName: 'nomgap-mek', + containers: [ + { + container: 'meal_logs', + partitionKey: '/userId', + fields: [{ field: 'notes', context: 'meal-logs' }], + }, + { + container: 'fasting_sessions', + partitionKey: '/userId', + fields: [{ field: 'notes', context: 'fasting-sessions' }], + }, + { + container: 'weight_log', + partitionKey: '/userId', + fields: [{ field: 'notes', context: 'weight-log' }], + }, + ], + }, + // ── ActionTrail (port 4018) ── + { + productId: 'actiontrail', + mekName: 'actiontrail-mek', + containers: [ + { + container: 'trail_actions', + partitionKey: '/userId', + fields: [ + { field: 'before', context: 'trail-actions', jsonSerialize: true }, + { field: 'after', context: 'trail-actions-after', jsonSerialize: true }, + ], + }, + { + container: 'trail_reverts', + partitionKey: '/userId', + fields: [ + { field: 'beforeState', context: 'trail-reverts', jsonSerialize: true }, + { field: 'afterState', context: 'trail-reverts-after', jsonSerialize: true }, + { field: 'revertPayload', context: 'trail-reverts-payload', jsonSerialize: true }, + ], + }, + ], + }, + // ── FlowMonk (port 4017) ── + { + productId: 'flowmonk', + mekName: 'flowmonk-mek', + containers: [ + { + container: 'tasks', + partitionKey: '/userId', + fields: [{ field: 'description', context: 'tasks' }], + }, + ], + }, + // ── ChronoMind (port 4011) ── + { + productId: 'chronomind', + mekName: 'chronomind-mek', + containers: [ + { + container: 'timers', + partitionKey: '/userId', + fields: [{ field: 'description', context: 'timers' }], + }, + { + container: 'routines', + partitionKey: '/userId', + fields: [ + { field: 'description', context: 'routines' }, + // steps[].notes handled differently — see note below + ], + }, + { + container: 'shared_timers', + partitionKey: '/householdId', + fields: [{ field: 'description', context: 'shared-timers', userIdField: 'createdBy' }], + }, + { + container: 'webhook_subscriptions', + partitionKey: '/userId', + fields: [ + { field: 'secret', context: 'webhooks' }, + { field: 'description', context: 'webhook-desc' }, + ], + }, + ], + }, + // ── PeakPulse (port 4010) ── + { + productId: 'peakpulse', + mekName: 'peakpulse-mek', + containers: [ + { + container: 'peak_sessions', + partitionKey: '/userId', + fields: [{ field: 'notes', context: 'peak-sessions' }], + }, + ], + }, + // ── MindLyst (port 4014) ── + { + productId: 'mindlyst', + mekName: 'mindlyst-mek', + containers: [ + { + container: 'memory_items', + partitionKey: '/userId', + fields: [ + { field: 'rawContent', context: 'memory' }, + { field: 'triageResult.summary', context: 'memory-triage' }, + ], + }, + { + container: 'reflections', + partitionKey: '/userId', + fields: [ + { field: 'repeatedThemes', context: 'reflection-themes', jsonSerialize: true }, + { field: 'postponedItems', context: 'reflection-postponed', jsonSerialize: true }, + { field: 'roleImbalanceSignals', context: 'reflection-imbalance', jsonSerialize: true }, + { field: 'suggestedAdjustments', context: 'reflection-adjustments', jsonSerialize: true }, + { field: 'vsLastWeek.summary', context: 'reflection-vs' }, + ], + }, + { + container: 'daily_briefs', + partitionKey: '/userId', + fields: [ + { field: 'greeting', context: 'daily-briefs' }, + { field: 'priorityItems', context: 'brief-priorities', jsonSerialize: true }, + { field: 'brainSummaries', context: 'brief-summaries', jsonSerialize: true }, + { field: 'streakMessage', context: 'brief-streak' }, + { field: 'motivationalQuote', context: 'brief-quote' }, + ], + }, + { + container: 'brains', + partitionKey: '/userId', + fields: [{ field: 'rolePrompt', context: 'brains' }], + }, + ], + }, +]; + +// ── CLI argument parsing ────────────────────────────── + +interface CliArgs { + product: string; + container?: string; + batchSize: number; + dryRun: boolean; + verbose: boolean; +} + +function parseArgs(): CliArgs { + const args = process.argv.slice(2); + const result: CliArgs = { + product: '', + batchSize: 100, + dryRun: false, + verbose: false, + }; + + for (let i = 0; i < args.length; i++) { + switch (args[i]) { + case '--product': + case '-p': + result.product = args[++i] ?? ''; + break; + case '--container': + case '-c': + result.container = args[++i]; + break; + case '--batch-size': + case '-b': + result.batchSize = parseInt(args[++i] ?? '100', 10); + break; + case '--dry-run': + case '-n': + result.dryRun = true; + break; + case '--verbose': + case '-v': + result.verbose = true; + break; + case '--help': + case '-h': + printUsage(); + process.exit(0); + break; + default: + console.error(`Unknown argument: ${args[i]}`); + printUsage(); + process.exit(1); + } + } + + if (!result.product) { + console.error('Error: --product is required\n'); + printUsage(); + process.exit(1); + } + + return result; +} + +function printUsage(): void { + console.log(` +╔══════════════════════════════════════════════════════════════╗ +║ ByteLyst Encryption Migration CLI ║ +║ Batch-encrypt existing plaintext Cosmos documents ║ +╚══════════════════════════════════════════════════════════════╝ + +Usage: + npx tsx scripts/encrypt-migrate.ts --product [options] + +Options: + --product, -p Product ID or 'all' (required) + --container, -c Migrate only this container (optional) + --batch-size, -b Documents per batch (default: 100) + --dry-run, -n Count without encrypting + --verbose, -v Show per-document progress + --help, -h Show this help + +Products: ${PRODUCT_CONFIGS.map(c => c.productId).join(', ')} + +Environment: + COSMOS_ENDPOINT Azure Cosmos DB endpoint + COSMOS_KEY Azure Cosmos DB key + COSMOS_DATABASE Database name (default: lysnrai) + FIELD_ENCRYPT_KEY_PROVIDER 'memory' | 'env' | 'akv' (default: memory) + FIELD_ENCRYPT_KEY Hex key (for 'env' provider) + AZURE_KEYVAULT_URL AKV URL (for 'akv' provider) + +Examples: + npx tsx scripts/encrypt-migrate.ts --product lysnrai --dry-run + npx tsx scripts/encrypt-migrate.ts --product all --dry-run + npx tsx scripts/encrypt-migrate.ts --product chronomind --container timers + npx tsx scripts/encrypt-migrate.ts --product mindlyst --batch-size 50 +`); +} + +// ── Cosmos helpers ──────────────────────────────────── + +function createCosmosClient(): CosmosClient { + const endpoint = process.env.COSMOS_ENDPOINT; + const key = process.env.COSMOS_KEY; + + if (!endpoint || !key) { + console.error('Error: COSMOS_ENDPOINT and COSMOS_KEY environment variables are required'); + process.exit(1); + } + + return new CosmosClient({ endpoint, key }); +} + +function getPartitionKeyValue(doc: Record, partitionKeyPath: string): string { + const field = partitionKeyPath.replace(/^\//, ''); + return String(doc[field] ?? ''); +} + +// ── Migration engine ────────────────────────────────── + +interface MigrateFieldResult { + containerName: string; + field: string; + result: MigrationResult; +} + +async function migrateContainerField( + cosmosContainer: Container, + partitionKey: string, + fieldSpec: FieldSpec, + encryptor: FieldEncryptor, + batchSize: number, + dryRun: boolean, + verbose: boolean +): Promise { + const containerName = cosmosContainer.id; + const userIdProp = fieldSpec.userIdField ?? 'userId'; + + const result = await migrateDocuments>({ + batchSize, + dryRun, + + fetchBatch: async (offset: number, size: number) => { + const query = `SELECT * FROM c ORDER BY c._ts OFFSET ${offset} LIMIT ${size}`; + const { resources } = await cosmosContainer.items.query(query).fetchAll(); + return resources; + }, + + getId: (doc: Record) => String(doc.id ?? 'unknown'), + + getField: (doc: Record) => getNestedField(doc, fieldSpec.field), + + encryptValue: async (plaintext: string, doc: Record) => { + const userId = String(doc[userIdProp] ?? 'unknown'); + return encryptor.encrypt(plaintext, { + userId, + context: fieldSpec.context, + }); + }, + + writeBack: async (doc: Record, encrypted: EncryptedField) => { + const updated = setNestedField(doc, fieldSpec.field, encrypted); + const pk = getPartitionKeyValue(doc, partitionKey); + await cosmosContainer.item(String(doc.id), pk).replace(updated); + }, + + onProgress: (progress: MigrationResult) => { + if (verbose) { + const mode = dryRun ? '[DRY-RUN]' : '[LIVE]'; + process.stdout.write( + `\r ${mode} ${containerName}.${fieldSpec.field}: scanned=${progress.scanned} encrypted=${progress.encrypted} skipped=${progress.skipped} errors=${progress.errors}` + ); + } + }, + }); + + if (verbose) { + process.stdout.write('\n'); + } + + return { containerName, field: fieldSpec.field, result }; +} + +async function migrateProduct( + config: ProductMigrationConfig, + cosmosClient: CosmosClient, + args: CliArgs +): Promise { + const dbName = process.env.COSMOS_DATABASE || 'lysnrai'; + const database = cosmosClient.database(dbName); + + const encryptor = createFieldEncryptor({ + keyProvider: (process.env.FIELD_ENCRYPT_KEY_PROVIDER as 'memory' | 'env' | 'akv') ?? 'memory', + encryptionKey: process.env.FIELD_ENCRYPT_KEY || undefined, + keyVaultUrl: process.env.AZURE_KEYVAULT_URL || undefined, + mekName: config.mekName, + }); + + const results: MigrateFieldResult[] = []; + + for (const containerConfig of config.containers) { + // Skip if user specified a specific container + if (args.container && containerConfig.container !== args.container) { + continue; + } + + const cosmosContainer = database.container(containerConfig.container); + const mode = args.dryRun ? 'DRY-RUN' : 'LIVE'; + + console.log(`\n 📦 ${containerConfig.container} (${mode})`); + + for (const fieldSpec of containerConfig.fields) { + console.log(` 🔐 Field: ${fieldSpec.field} (context: ${fieldSpec.context})`); + + try { + const fieldResult = await migrateContainerField( + cosmosContainer, + containerConfig.partitionKey, + fieldSpec, + encryptor, + args.batchSize, + args.dryRun, + args.verbose + ); + + results.push(fieldResult); + + const r = fieldResult.result; + console.log( + ` scanned=${r.scanned} encrypted=${r.encrypted} skipped=${r.skipped} errors=${r.errors}` + ); + + if (r.errorDetails.length > 0) { + console.log(' Errors:'); + for (const err of r.errorDetails) { + console.log(` - ${err.id}: ${err.error}`); + } + } + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + console.error(` ERROR: ${message}`); + results.push({ + containerName: containerConfig.container, + field: fieldSpec.field, + result: { + scanned: 0, + encrypted: 0, + skipped: 0, + errors: 1, + errorDetails: [{ id: 'container', error: message }], + }, + }); + } + } + } + + return results; +} + +// ── Main ────────────────────────────────────────────── + +async function main(): Promise { + const args = parseArgs(); + const startTime = Date.now(); + + console.log('\n════════════════════════════════════════════════════'); + console.log(' ByteLyst Encryption Migration CLI'); + console.log('════════════════════════════════════════════════════'); + console.log(` Mode: ${args.dryRun ? 'DRY-RUN (no writes)' : 'LIVE (encrypting!)'}`); + console.log(` Product: ${args.product}`); + console.log(` Container: ${args.container ?? 'all'}`); + console.log(` Batch size: ${args.batchSize}`); + console.log(` Database: ${process.env.COSMOS_DATABASE || 'lysnrai'}`); + console.log(` Key provider: ${process.env.FIELD_ENCRYPT_KEY_PROVIDER || 'memory'}`); + console.log('════════════════════════════════════════════════════'); + + const cosmosClient = createCosmosClient(); + const allResults: MigrateFieldResult[] = []; + + const productsToMigrate = + args.product === 'all' + ? PRODUCT_CONFIGS + : PRODUCT_CONFIGS.filter(c => c.productId === args.product); + + if (productsToMigrate.length === 0) { + console.error(`\nError: Unknown product '${args.product}'`); + console.error(`Available: ${PRODUCT_CONFIGS.map(c => c.productId).join(', ')}`); + process.exit(1); + } + + for (const productConfig of productsToMigrate) { + console.log(`\n🏢 ${productConfig.productId} (MEK: ${productConfig.mekName})`); + const results = await migrateProduct(productConfig, cosmosClient, args); + allResults.push(...results); + } + + // ── Summary ── + const elapsed = ((Date.now() - startTime) / 1000).toFixed(1); + const totals = allResults.reduce( + (acc, r) => ({ + scanned: acc.scanned + r.result.scanned, + encrypted: acc.encrypted + r.result.encrypted, + skipped: acc.skipped + r.result.skipped, + errors: acc.errors + r.result.errors, + }), + { scanned: 0, encrypted: 0, skipped: 0, errors: 0 } + ); + + console.log('\n════════════════════════════════════════════════════'); + console.log(' Migration Summary'); + console.log('════════════════════════════════════════════════════'); + console.log(` Products: ${productsToMigrate.length}`); + console.log(` Fields: ${allResults.length}`); + console.log(` Scanned: ${totals.scanned}`); + console.log(` Encrypted: ${totals.encrypted}`); + console.log(` Skipped: ${totals.skipped}`); + console.log(` Errors: ${totals.errors}`); + console.log(` Duration: ${elapsed}s`); + console.log(` Mode: ${args.dryRun ? 'DRY-RUN' : 'LIVE'}`); + console.log('════════════════════════════════════════════════════\n'); + + // Per-field breakdown + if (allResults.length > 0) { + console.log(' Per-field breakdown:'); + console.log(' ┌─────────────────────────────┬────────┬───────────┬─────────┬────────┐'); + console.log(' │ Container.Field │ Scanned│ Encrypted │ Skipped │ Errors │'); + console.log(' ├─────────────────────────────┼────────┼───────────┼─────────┼────────┤'); + for (const r of allResults) { + const name = `${r.containerName}.${r.field}`.padEnd(29); + const sc = String(r.result.scanned).padStart(7); + const en = String(r.result.encrypted).padStart(9); + const sk = String(r.result.skipped).padStart(7); + const er = String(r.result.errors).padStart(6); + console.log(` │ ${name}│${sc} │${en} │${sk} │${er} │`); + } + console.log(' └─────────────────────────────┴────────┴───────────┴─────────┴────────┘'); + } + + if (totals.errors > 0) { + process.exit(1); + } +} + +main().catch(err => { + console.error('\nFatal error:', err instanceof Error ? err.message : err); + process.exit(1); +}); diff --git a/scripts/package.json b/scripts/package.json new file mode 100644 index 00000000..6ede7ec8 --- /dev/null +++ b/scripts/package.json @@ -0,0 +1,16 @@ +{ + "name": "@bytelyst/scripts", + "version": "0.1.0", + "private": true, + "type": "module", + "description": "DevOps scripts for the ByteLyst ecosystem (migration, audit, maintenance)", + "dependencies": { + "@azure/cosmos": "^4.2.0", + "@bytelyst/field-encrypt": "workspace:*" + }, + "devDependencies": { + "@types/node": "^22.12.0", + "tsx": "^4.19.2", + "typescript": "^5.7.3" + } +}