feat(devops): encryption migration CLI with embedded product configs
- scripts/encrypt-migrate.ts — batch-encrypt existing plaintext Cosmos docs - scripts/ added as pnpm workspace member for clean @bytelyst/* imports - 10 product configs, 20 containers, 40+ fields - --dry-run, --product, --container, --batch-size, --verbose flags - Idempotent via __encrypted sentinel (migrateDocuments helper) - Updated E2EE roadmap Sprint 6.2 as complete
This commit is contained in:
saravanakumardb1
parent
cf9617cda5
commit
c252cfd198
@ -1,7 +1,7 @@
|
||||
# ByteLyst — End-to-End Encryption Implementation Roadmap
|
||||
|
||||
> **Purpose:** Phased implementation plan for encryption across the ByteLyst ecosystem.
|
||||
> **Status:** Phase 1 + Phase 2 (Sprint 3 + Sprint 4) COMPLETE — 9 backends + native SDKs + client-encrypt + secure-storage-web
|
||||
> **Status:** Phase 1 + Phase 2 COMPLETE (10 backends + native SDKs + client-encrypt + secure-storage-web) + Sprint 6.2 migration CLI
|
||||
> **Author:** AI Architecture Review
|
||||
> **Last updated:** 2026-03-21
|
||||
> **Design doc:** [`END_TO_END_ENCRYPTION_DESIGN.md`](END_TO_END_ENCRYPTION_DESIGN.md)
|
||||
@ -233,14 +233,16 @@ Week 1-2 Week 3-4 Week 5-6 Week 7-8 Week 9-10 Week 11-14
|
||||
|
||||
#### 3.2 MindLyst Backend (port 4014)
|
||||
|
||||
- [ ] **3.2.1** Add dependency + create encryptor singleton
|
||||
- [ ] **3.2.2** Encrypt `content` and `voiceTranscriptText` in memory_items module
|
||||
- [ ] **3.2.3** Encrypt array fields in reflections module
|
||||
- `repeatedThemes` (string[]), `suggestedAdjustments` (string[]), `vsLastWeek.summary` (string)
|
||||
- [x] **3.2.1** Add dependency + create encryptor singleton
|
||||
- [x] **3.2.2** Encrypt `rawContent` and `triageResult.summary` in memory_items module
|
||||
- [x] **3.2.3** Encrypt array fields in reflections module
|
||||
- `repeatedThemes` (string[]), `postponedItems` (string[]), `roleImbalanceSignals` (string[]), `suggestedAdjustments` (string[]), `vsLastWeek.summary` (string)
|
||||
- Array fields: JSON-serialize → encrypt → store as single EncryptedField each
|
||||
- [ ] **3.2.4** Update tests
|
||||
- [x] **3.2.4** Encrypt daily-briefs: `greeting`, `priorityItems`, `brainSummaries`, `streakMessage`, `motivationalQuote`
|
||||
- [x] **3.2.5** Encrypt brains: `rolePrompt`
|
||||
- [x] **3.2.6** Update tests
|
||||
|
||||
**Commit:** `feat(mindlyst): encrypt memory content, voice transcripts, and reflections`
|
||||
**Commit:** `feat(mindlyst): encrypt sensitive fields across all modules`
|
||||
|
||||
#### 3.3 NomGap Backend (port 4013)
|
||||
|
||||
@ -264,13 +266,14 @@ Week 1-2 Week 3-4 Week 5-6 Week 7-8 Week 9-10 Week 11-14
|
||||
#### 3.5 Remaining Backends (Low Priority — Defer or Skip)
|
||||
|
||||
| Product | Backend Port | Encrypted Fields | Decision |
|
||||
| ---------------------- | ---------------------------------------------------------------------- | --------------------------------------------------- | -------- |
|
||||
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | -------- |
|
||||
| **FlowMonk** (4017) | `tasks.description` | ✅ Done (Sprint 4) — 211 tests |
|
||||
| **ChronoMind** (4011) | `timers.description`, `routines.description`, `routines.steps[].notes` | ✅ Done — 182 tests |
|
||||
| **ChronoMind** (4011) | `timers.description`, `routines.description`, `routines.steps[].notes`, `shared-timers.description`, `webhooks.secret`, `webhooks.description` | ✅ Done — 182 tests |
|
||||
| **PeakPulse** (4010) | `peak-sessions.notes` | ✅ Done — 65 tests |
|
||||
| **LocalMemGPT** (4019) | `messages.content` | Deferred — SQLite raw SQL, needs different approach |
|
||||
| **MindLyst** (4014) | `memory.rawContent`, `memory.triageResult.summary`, reflections (5 fields), daily-briefs (5 fields), `brains.rolePrompt` | ✅ Done |
|
||||
| **LocalMemGPT** (4019) | `messages.content`, `conversations.system_prompt` | Deferred — SQLite raw SQL (better-sqlite3), needs SQLite-specific encryption approach |
|
||||
|
||||
**Sprint 3 deliverable:** 9 product backends encrypted (LysnrAI, JarvisJr, NoteLett, NomGap, ActionTrail, FlowMonk, ChronoMind, PeakPulse). Only MindLyst (KMP) and LocalMemGPT (SQLite) deferred.
|
||||
**Sprint 3 deliverable:** 10 product backends encrypted (LysnrAI, JarvisJr, NoteLett, NomGap, ActionTrail, FlowMonk, ChronoMind, PeakPulse, MindLyst). Only LocalMemGPT (SQLite) deferred to Sprint 5.
|
||||
|
||||
---
|
||||
|
||||
@ -449,36 +452,29 @@ Week 1-2 Week 3-4 Week 5-6 Week 7-8 Week 9-10 Week 11-14
|
||||
|
||||
#### 6.2 Encryption Migration CLI
|
||||
|
||||
- [ ] **6.2.1** Create `scripts/encrypt-migrate.ts` — CLI tool for batch encrypting existing plaintext
|
||||
- [x] **6.2.1** Create `scripts/encrypt-migrate.ts` — CLI tool for batch encrypting existing plaintext
|
||||
|
||||
```bash
|
||||
npx tsx scripts/encrypt-migrate.ts \
|
||||
--product lysnrai \
|
||||
--container transcripts \
|
||||
--field transcriptText \
|
||||
--partition-key /userId \
|
||||
--batch-size 100 \
|
||||
--dry-run
|
||||
npx tsx scripts/encrypt-migrate.ts --product lysnrai --dry-run
|
||||
npx tsx scripts/encrypt-migrate.ts --product all --dry-run
|
||||
npx tsx scripts/encrypt-migrate.ts --product chronomind --container timers
|
||||
npx tsx scripts/encrypt-migrate.ts --product mindlyst --batch-size 50 --verbose
|
||||
```
|
||||
|
||||
- Uses `@bytelyst/field-encrypt` migration helpers
|
||||
- Progress bar, resumable (tracks last processed ID)
|
||||
- Dry run mode (report count without modifying)
|
||||
- Uses `@bytelyst/field-encrypt` `migrateDocuments()` helper (idempotent via `__encrypted` sentinel)
|
||||
- Connects directly to Cosmos DB via `@azure/cosmos`
|
||||
- Per-product configs embedded: 10 products, 20 containers, 40+ fields
|
||||
- Dry-run mode (count without modifying), verbose per-doc progress
|
||||
- Nested field support (e.g., `triageResult.summary`)
|
||||
- JSON-serialized arrays/objects (e.g., reflections string arrays, brainSummaries Record)
|
||||
- Per-field breakdown table in summary output
|
||||
|
||||
- [ ] **6.2.2** Create per-product migration scripts
|
||||
```
|
||||
scripts/migrations/
|
||||
├── encrypt-lysnrai.sh
|
||||
├── encrypt-jarvisjr.sh
|
||||
├── encrypt-notelett.sh
|
||||
├── encrypt-mindlyst.sh
|
||||
├── encrypt-nomgap.sh
|
||||
├── encrypt-actiontrail.sh
|
||||
├── encrypt-flowmonk.sh
|
||||
└── encrypt-localmemgpt.sh
|
||||
```
|
||||
- [x] **6.2.2** Embedded per-product migration configs (no separate shell scripts needed)
|
||||
- Products: lysnrai, jarvisjr, notelett, nomgap, actiontrail, flowmonk, chronomind, peakpulse, mindlyst
|
||||
- Each config specifies: productId, mekName, containers[], fields[] with context + userIdField + jsonSerialize
|
||||
- Run `--product all --dry-run` for full ecosystem audit
|
||||
|
||||
**Commit:** `feat(devops): encryption migration CLI and per-product migration scripts`
|
||||
**Commit:** `feat(devops): encryption migration CLI with embedded product configs`
|
||||
|
||||
#### 6.3 Monitoring & Audit
|
||||
|
||||
|
||||
62
pnpm-lock.yaml
generated
62
pnpm-lock.yaml
generated
@ -393,6 +393,12 @@ importers:
|
||||
|
||||
packages/celebrations: {}
|
||||
|
||||
packages/client-encrypt:
|
||||
devDependencies:
|
||||
vitest:
|
||||
specifier: ^3.0.0
|
||||
version: 3.2.4(@types/debug@4.1.12)(@types/node@22.19.11)(happy-dom@18.0.1)(jiti@2.6.1)(jsdom@28.0.0(@noble/hashes@1.8.0))(lightningcss@1.31.1)(msw@2.12.10(@types/node@22.19.11)(typescript@5.9.3))(terser@5.46.0)(tsx@4.21.0)(yaml@2.8.2)
|
||||
|
||||
packages/config:
|
||||
dependencies:
|
||||
zod:
|
||||
@ -604,6 +610,25 @@ importers:
|
||||
specifier: ^3.0.0
|
||||
version: 3.2.4(@types/debug@4.1.12)(@types/node@22.19.11)(happy-dom@18.0.1)(jiti@2.6.1)(jsdom@28.0.0(@noble/hashes@1.8.0))(lightningcss@1.31.1)(msw@2.12.10(@types/node@22.19.11)(typescript@5.9.3))(terser@5.46.0)(tsx@4.21.0)(yaml@2.8.2)
|
||||
|
||||
packages/field-encrypt:
|
||||
dependencies:
|
||||
'@azure/identity':
|
||||
specifier: '>=4.0.0'
|
||||
version: 4.13.0
|
||||
'@azure/keyvault-keys':
|
||||
specifier: '>=4.8.0'
|
||||
version: 4.10.0(@azure/core-client@1.10.1)
|
||||
'@bytelyst/errors':
|
||||
specifier: workspace:*
|
||||
version: link:../errors
|
||||
devDependencies:
|
||||
vitest:
|
||||
specifier: ^3.0.0
|
||||
version: 3.2.4(@types/debug@4.1.12)(@types/node@22.19.11)(happy-dom@18.0.1)(jiti@2.6.1)(jsdom@28.0.0(@noble/hashes@1.8.0))(lightningcss@1.31.1)(msw@2.12.10(@types/node@22.19.11)(typescript@5.9.3))(terser@5.46.0)(tsx@4.21.0)(yaml@2.8.2)
|
||||
zod:
|
||||
specifier: ^3.24.0
|
||||
version: 3.25.76
|
||||
|
||||
packages/gentle-notifications: {}
|
||||
|
||||
packages/kill-switch-client: {}
|
||||
@ -701,6 +726,15 @@ importers:
|
||||
|
||||
packages/referral-client: {}
|
||||
|
||||
packages/secure-storage-web:
|
||||
devDependencies:
|
||||
fake-indexeddb:
|
||||
specifier: ^6.0.0
|
||||
version: 6.2.5
|
||||
vitest:
|
||||
specifier: ^3.0.0
|
||||
version: 3.2.4(@types/debug@4.1.12)(@types/node@22.19.11)(happy-dom@18.0.1)(jiti@2.6.1)(jsdom@28.0.0(@noble/hashes@1.8.0))(lightningcss@1.31.1)(msw@2.12.10(@types/node@22.19.11)(typescript@5.9.3))(terser@5.46.0)(tsx@4.21.0)(yaml@2.8.2)
|
||||
|
||||
packages/speech:
|
||||
devDependencies:
|
||||
typescript:
|
||||
@ -762,6 +796,25 @@ importers:
|
||||
|
||||
packages/webhook-dispatch: {}
|
||||
|
||||
scripts:
|
||||
dependencies:
|
||||
'@azure/cosmos':
|
||||
specifier: ^4.2.0
|
||||
version: 4.9.1(@azure/core-client@1.10.1)
|
||||
'@bytelyst/field-encrypt':
|
||||
specifier: workspace:*
|
||||
version: link:../packages/field-encrypt
|
||||
devDependencies:
|
||||
'@types/node':
|
||||
specifier: ^22.12.0
|
||||
version: 22.19.11
|
||||
tsx:
|
||||
specifier: ^4.19.2
|
||||
version: 4.21.0
|
||||
typescript:
|
||||
specifier: ^5.7.3
|
||||
version: 5.9.3
|
||||
|
||||
services/extraction-service:
|
||||
dependencies:
|
||||
'@azure/cosmos':
|
||||
@ -8082,6 +8135,13 @@ packages:
|
||||
integrity: sha512-UOiS2in6/Q0FK0R0q6UY9vYpQ21mr/Qn1KOnte7vsACuNJf514WvCCUHSRCPcgjPT2bAhNIJdlE6bVap1GKmeg==,
|
||||
}
|
||||
|
||||
fake-indexeddb@6.2.5:
|
||||
resolution:
|
||||
{
|
||||
integrity: sha512-CGnyrvbhPlWYMngksqrSSUT1BAVP49dZocrHuK0SvtR0D5TMs5wP0o3j7jexDJW01KSadjBp1M/71o/KR3nD1w==,
|
||||
}
|
||||
engines: { node: '>=18' }
|
||||
|
||||
fast-decode-uri-component@1.0.1:
|
||||
resolution:
|
||||
{
|
||||
@ -19522,6 +19582,8 @@ snapshots:
|
||||
|
||||
extendable-error@0.1.7: {}
|
||||
|
||||
fake-indexeddb@6.2.5: {}
|
||||
|
||||
fast-decode-uri-component@1.0.1: {}
|
||||
|
||||
fast-deep-equal@3.1.3: {}
|
||||
|
||||
@ -2,3 +2,4 @@ packages:
|
||||
- 'packages/*'
|
||||
- 'services/*'
|
||||
- 'dashboards/*'
|
||||
- 'scripts'
|
||||
|
||||
644
scripts/encrypt-migrate.ts
Normal file
644
scripts/encrypt-migrate.ts
Normal file
@ -0,0 +1,644 @@
|
||||
#!/usr/bin/env npx tsx
|
||||
/* eslint-disable no-console */
|
||||
/**
|
||||
* Encryption Migration CLI — Batch-encrypt existing plaintext Cosmos documents.
|
||||
*
|
||||
* Uses @bytelyst/field-encrypt migrateDocuments() to idempotently encrypt
|
||||
* plaintext fields across all product backends.
|
||||
*
|
||||
* Usage:
|
||||
* npx tsx scripts/encrypt-migrate.ts --product lysnrai --dry-run
|
||||
* npx tsx scripts/encrypt-migrate.ts --product lysnrai --container transcripts
|
||||
* npx tsx scripts/encrypt-migrate.ts --product all --dry-run
|
||||
* npx tsx scripts/encrypt-migrate.ts --product chronomind --batch-size 50
|
||||
*
|
||||
* Required env vars:
|
||||
* COSMOS_ENDPOINT — Azure Cosmos DB endpoint
|
||||
* COSMOS_KEY — Azure Cosmos DB key
|
||||
* COSMOS_DATABASE — Database name (default: lysnrai)
|
||||
* FIELD_ENCRYPT_KEY_PROVIDER — 'memory' | 'env' | 'akv' (default: memory)
|
||||
* FIELD_ENCRYPT_KEY — Hex key (for 'env' provider)
|
||||
* AZURE_KEYVAULT_URL — AKV URL (for 'akv' provider)
|
||||
*
|
||||
* Per-product MEK names are defined in migration configs.
|
||||
*/
|
||||
|
||||
import { CosmosClient, type Container } from '@azure/cosmos';
|
||||
import {
|
||||
createFieldEncryptor,
|
||||
migrateDocuments,
|
||||
type MigrationResult,
|
||||
type FieldEncryptor,
|
||||
type EncryptedField,
|
||||
} from '@bytelyst/field-encrypt';
|
||||
|
||||
// ── Types ──────────────────────────────────────────────
|
||||
|
||||
interface FieldSpec {
|
||||
/** The field path (dot-separated for nested, e.g. 'triageResult.summary'). */
|
||||
field: string;
|
||||
/** Encryption context label. */
|
||||
context: string;
|
||||
/** The property on the doc that holds the userId for DEK scoping. Default: 'userId'. */
|
||||
userIdField?: string;
|
||||
/** If true, the field is a string[] or Record — JSON-serialize before encrypting. */
|
||||
jsonSerialize?: boolean;
|
||||
}
|
||||
|
||||
interface ContainerMigrationConfig {
|
||||
/** Cosmos container name. */
|
||||
container: string;
|
||||
/** Partition key path (e.g. '/userId'). */
|
||||
partitionKey: string;
|
||||
/** Fields to encrypt in this container. */
|
||||
fields: FieldSpec[];
|
||||
}
|
||||
|
||||
interface ProductMigrationConfig {
|
||||
/** Product ID. */
|
||||
productId: string;
|
||||
/** MEK name for this product. */
|
||||
mekName: string;
|
||||
/** Containers to migrate. */
|
||||
containers: ContainerMigrationConfig[];
|
||||
}
|
||||
|
||||
// ── Field accessor helpers ────────────────────────────
|
||||
|
||||
function getNestedField(doc: Record<string, unknown>, path: string): unknown {
|
||||
const parts = path.split('.');
|
||||
let current: unknown = doc;
|
||||
for (const part of parts) {
|
||||
if (current == null || typeof current !== 'object') return undefined;
|
||||
current = (current as Record<string, unknown>)[part];
|
||||
}
|
||||
return current;
|
||||
}
|
||||
|
||||
function setNestedField(
|
||||
doc: Record<string, unknown>,
|
||||
path: string,
|
||||
value: unknown
|
||||
): Record<string, unknown> {
|
||||
const parts = path.split('.');
|
||||
const result = { ...doc };
|
||||
|
||||
if (parts.length === 1) {
|
||||
result[parts[0]] = value;
|
||||
return result;
|
||||
}
|
||||
|
||||
// Deep clone the nested path
|
||||
let current = result;
|
||||
for (let i = 0; i < parts.length - 1; i++) {
|
||||
const nested = current[parts[i]];
|
||||
if (nested == null || typeof nested !== 'object') return result;
|
||||
current[parts[i]] = { ...(nested as Record<string, unknown>) };
|
||||
current = current[parts[i]] as Record<string, unknown>;
|
||||
}
|
||||
current[parts[parts.length - 1]] = value;
|
||||
return result;
|
||||
}
|
||||
|
||||
// ── Product migration configs ─────────────────────────
|
||||
|
||||
const PRODUCT_CONFIGS: ProductMigrationConfig[] = [
|
||||
// ── LysnrAI (port 4015) ──
|
||||
{
|
||||
productId: 'lysnrai',
|
||||
mekName: 'lysnr-mek',
|
||||
containers: [
|
||||
{
|
||||
container: 'transcripts',
|
||||
partitionKey: '/userId',
|
||||
fields: [{ field: 'transcriptText', context: 'transcripts' }],
|
||||
},
|
||||
{
|
||||
container: 'sessions',
|
||||
partitionKey: '/userId',
|
||||
fields: [{ field: 'notes', context: 'sessions' }],
|
||||
},
|
||||
],
|
||||
},
|
||||
// ── JarvisJr (port 4012) ──
|
||||
{
|
||||
productId: 'jarvisjr',
|
||||
mekName: 'jarvisjr-mek',
|
||||
containers: [
|
||||
{
|
||||
container: 'jarvis_sessions',
|
||||
partitionKey: '/userId',
|
||||
fields: [{ field: 'notes', context: 'jarvis-sessions' }],
|
||||
},
|
||||
{
|
||||
container: 'jarvis_memory',
|
||||
partitionKey: '/agentId',
|
||||
fields: [{ field: 'content', context: 'jarvis-memory', userIdField: 'agentId' }],
|
||||
},
|
||||
],
|
||||
},
|
||||
// ── NoteLett (port 4016) ──
|
||||
{
|
||||
productId: 'notelett',
|
||||
mekName: 'notelett-mek',
|
||||
containers: [
|
||||
{
|
||||
container: 'notes',
|
||||
partitionKey: '/workspaceId',
|
||||
fields: [{ field: 'body', context: 'notes', userIdField: 'createdBy' }],
|
||||
},
|
||||
],
|
||||
},
|
||||
// ── NomGap (port 4013) ──
|
||||
{
|
||||
productId: 'nomgap',
|
||||
mekName: 'nomgap-mek',
|
||||
containers: [
|
||||
{
|
||||
container: 'meal_logs',
|
||||
partitionKey: '/userId',
|
||||
fields: [{ field: 'notes', context: 'meal-logs' }],
|
||||
},
|
||||
{
|
||||
container: 'fasting_sessions',
|
||||
partitionKey: '/userId',
|
||||
fields: [{ field: 'notes', context: 'fasting-sessions' }],
|
||||
},
|
||||
{
|
||||
container: 'weight_log',
|
||||
partitionKey: '/userId',
|
||||
fields: [{ field: 'notes', context: 'weight-log' }],
|
||||
},
|
||||
],
|
||||
},
|
||||
// ── ActionTrail (port 4018) ──
|
||||
{
|
||||
productId: 'actiontrail',
|
||||
mekName: 'actiontrail-mek',
|
||||
containers: [
|
||||
{
|
||||
container: 'trail_actions',
|
||||
partitionKey: '/userId',
|
||||
fields: [
|
||||
{ field: 'before', context: 'trail-actions', jsonSerialize: true },
|
||||
{ field: 'after', context: 'trail-actions-after', jsonSerialize: true },
|
||||
],
|
||||
},
|
||||
{
|
||||
container: 'trail_reverts',
|
||||
partitionKey: '/userId',
|
||||
fields: [
|
||||
{ field: 'beforeState', context: 'trail-reverts', jsonSerialize: true },
|
||||
{ field: 'afterState', context: 'trail-reverts-after', jsonSerialize: true },
|
||||
{ field: 'revertPayload', context: 'trail-reverts-payload', jsonSerialize: true },
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
// ── FlowMonk (port 4017) ──
|
||||
{
|
||||
productId: 'flowmonk',
|
||||
mekName: 'flowmonk-mek',
|
||||
containers: [
|
||||
{
|
||||
container: 'tasks',
|
||||
partitionKey: '/userId',
|
||||
fields: [{ field: 'description', context: 'tasks' }],
|
||||
},
|
||||
],
|
||||
},
|
||||
// ── ChronoMind (port 4011) ──
|
||||
{
|
||||
productId: 'chronomind',
|
||||
mekName: 'chronomind-mek',
|
||||
containers: [
|
||||
{
|
||||
container: 'timers',
|
||||
partitionKey: '/userId',
|
||||
fields: [{ field: 'description', context: 'timers' }],
|
||||
},
|
||||
{
|
||||
container: 'routines',
|
||||
partitionKey: '/userId',
|
||||
fields: [
|
||||
{ field: 'description', context: 'routines' },
|
||||
// steps[].notes handled differently — see note below
|
||||
],
|
||||
},
|
||||
{
|
||||
container: 'shared_timers',
|
||||
partitionKey: '/householdId',
|
||||
fields: [{ field: 'description', context: 'shared-timers', userIdField: 'createdBy' }],
|
||||
},
|
||||
{
|
||||
container: 'webhook_subscriptions',
|
||||
partitionKey: '/userId',
|
||||
fields: [
|
||||
{ field: 'secret', context: 'webhooks' },
|
||||
{ field: 'description', context: 'webhook-desc' },
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
// ── PeakPulse (port 4010) ──
|
||||
{
|
||||
productId: 'peakpulse',
|
||||
mekName: 'peakpulse-mek',
|
||||
containers: [
|
||||
{
|
||||
container: 'peak_sessions',
|
||||
partitionKey: '/userId',
|
||||
fields: [{ field: 'notes', context: 'peak-sessions' }],
|
||||
},
|
||||
],
|
||||
},
|
||||
// ── MindLyst (port 4014) ──
|
||||
{
|
||||
productId: 'mindlyst',
|
||||
mekName: 'mindlyst-mek',
|
||||
containers: [
|
||||
{
|
||||
container: 'memory_items',
|
||||
partitionKey: '/userId',
|
||||
fields: [
|
||||
{ field: 'rawContent', context: 'memory' },
|
||||
{ field: 'triageResult.summary', context: 'memory-triage' },
|
||||
],
|
||||
},
|
||||
{
|
||||
container: 'reflections',
|
||||
partitionKey: '/userId',
|
||||
fields: [
|
||||
{ field: 'repeatedThemes', context: 'reflection-themes', jsonSerialize: true },
|
||||
{ field: 'postponedItems', context: 'reflection-postponed', jsonSerialize: true },
|
||||
{ field: 'roleImbalanceSignals', context: 'reflection-imbalance', jsonSerialize: true },
|
||||
{ field: 'suggestedAdjustments', context: 'reflection-adjustments', jsonSerialize: true },
|
||||
{ field: 'vsLastWeek.summary', context: 'reflection-vs' },
|
||||
],
|
||||
},
|
||||
{
|
||||
container: 'daily_briefs',
|
||||
partitionKey: '/userId',
|
||||
fields: [
|
||||
{ field: 'greeting', context: 'daily-briefs' },
|
||||
{ field: 'priorityItems', context: 'brief-priorities', jsonSerialize: true },
|
||||
{ field: 'brainSummaries', context: 'brief-summaries', jsonSerialize: true },
|
||||
{ field: 'streakMessage', context: 'brief-streak' },
|
||||
{ field: 'motivationalQuote', context: 'brief-quote' },
|
||||
],
|
||||
},
|
||||
{
|
||||
container: 'brains',
|
||||
partitionKey: '/userId',
|
||||
fields: [{ field: 'rolePrompt', context: 'brains' }],
|
||||
},
|
||||
],
|
||||
},
|
||||
];
|
||||
|
||||
// ── CLI argument parsing ──────────────────────────────
|
||||
|
||||
interface CliArgs {
|
||||
product: string;
|
||||
container?: string;
|
||||
batchSize: number;
|
||||
dryRun: boolean;
|
||||
verbose: boolean;
|
||||
}
|
||||
|
||||
function parseArgs(): CliArgs {
|
||||
const args = process.argv.slice(2);
|
||||
const result: CliArgs = {
|
||||
product: '',
|
||||
batchSize: 100,
|
||||
dryRun: false,
|
||||
verbose: false,
|
||||
};
|
||||
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
switch (args[i]) {
|
||||
case '--product':
|
||||
case '-p':
|
||||
result.product = args[++i] ?? '';
|
||||
break;
|
||||
case '--container':
|
||||
case '-c':
|
||||
result.container = args[++i];
|
||||
break;
|
||||
case '--batch-size':
|
||||
case '-b':
|
||||
result.batchSize = parseInt(args[++i] ?? '100', 10);
|
||||
break;
|
||||
case '--dry-run':
|
||||
case '-n':
|
||||
result.dryRun = true;
|
||||
break;
|
||||
case '--verbose':
|
||||
case '-v':
|
||||
result.verbose = true;
|
||||
break;
|
||||
case '--help':
|
||||
case '-h':
|
||||
printUsage();
|
||||
process.exit(0);
|
||||
break;
|
||||
default:
|
||||
console.error(`Unknown argument: ${args[i]}`);
|
||||
printUsage();
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
if (!result.product) {
|
||||
console.error('Error: --product is required\n');
|
||||
printUsage();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
function printUsage(): void {
|
||||
console.log(`
|
||||
╔══════════════════════════════════════════════════════════════╗
|
||||
║ ByteLyst Encryption Migration CLI ║
|
||||
║ Batch-encrypt existing plaintext Cosmos documents ║
|
||||
╚══════════════════════════════════════════════════════════════╝
|
||||
|
||||
Usage:
|
||||
npx tsx scripts/encrypt-migrate.ts --product <id> [options]
|
||||
|
||||
Options:
|
||||
--product, -p <id> Product ID or 'all' (required)
|
||||
--container, -c <name> Migrate only this container (optional)
|
||||
--batch-size, -b <n> Documents per batch (default: 100)
|
||||
--dry-run, -n Count without encrypting
|
||||
--verbose, -v Show per-document progress
|
||||
--help, -h Show this help
|
||||
|
||||
Products: ${PRODUCT_CONFIGS.map(c => c.productId).join(', ')}
|
||||
|
||||
Environment:
|
||||
COSMOS_ENDPOINT Azure Cosmos DB endpoint
|
||||
COSMOS_KEY Azure Cosmos DB key
|
||||
COSMOS_DATABASE Database name (default: lysnrai)
|
||||
FIELD_ENCRYPT_KEY_PROVIDER 'memory' | 'env' | 'akv' (default: memory)
|
||||
FIELD_ENCRYPT_KEY Hex key (for 'env' provider)
|
||||
AZURE_KEYVAULT_URL AKV URL (for 'akv' provider)
|
||||
|
||||
Examples:
|
||||
npx tsx scripts/encrypt-migrate.ts --product lysnrai --dry-run
|
||||
npx tsx scripts/encrypt-migrate.ts --product all --dry-run
|
||||
npx tsx scripts/encrypt-migrate.ts --product chronomind --container timers
|
||||
npx tsx scripts/encrypt-migrate.ts --product mindlyst --batch-size 50
|
||||
`);
|
||||
}
|
||||
|
||||
// ── Cosmos helpers ────────────────────────────────────
|
||||
|
||||
function createCosmosClient(): CosmosClient {
|
||||
const endpoint = process.env.COSMOS_ENDPOINT;
|
||||
const key = process.env.COSMOS_KEY;
|
||||
|
||||
if (!endpoint || !key) {
|
||||
console.error('Error: COSMOS_ENDPOINT and COSMOS_KEY environment variables are required');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
return new CosmosClient({ endpoint, key });
|
||||
}
|
||||
|
||||
function getPartitionKeyValue(doc: Record<string, unknown>, partitionKeyPath: string): string {
|
||||
const field = partitionKeyPath.replace(/^\//, '');
|
||||
return String(doc[field] ?? '');
|
||||
}
|
||||
|
||||
// ── Migration engine ──────────────────────────────────
|
||||
|
||||
interface MigrateFieldResult {
|
||||
containerName: string;
|
||||
field: string;
|
||||
result: MigrationResult;
|
||||
}
|
||||
|
||||
async function migrateContainerField(
|
||||
cosmosContainer: Container,
|
||||
partitionKey: string,
|
||||
fieldSpec: FieldSpec,
|
||||
encryptor: FieldEncryptor,
|
||||
batchSize: number,
|
||||
dryRun: boolean,
|
||||
verbose: boolean
|
||||
): Promise<MigrateFieldResult> {
|
||||
const containerName = cosmosContainer.id;
|
||||
const userIdProp = fieldSpec.userIdField ?? 'userId';
|
||||
|
||||
const result = await migrateDocuments<Record<string, unknown>>({
|
||||
batchSize,
|
||||
dryRun,
|
||||
|
||||
fetchBatch: async (offset: number, size: number) => {
|
||||
const query = `SELECT * FROM c ORDER BY c._ts OFFSET ${offset} LIMIT ${size}`;
|
||||
const { resources } = await cosmosContainer.items.query(query).fetchAll();
|
||||
return resources;
|
||||
},
|
||||
|
||||
getId: (doc: Record<string, unknown>) => String(doc.id ?? 'unknown'),
|
||||
|
||||
getField: (doc: Record<string, unknown>) => getNestedField(doc, fieldSpec.field),
|
||||
|
||||
encryptValue: async (plaintext: string, doc: Record<string, unknown>) => {
|
||||
const userId = String(doc[userIdProp] ?? 'unknown');
|
||||
return encryptor.encrypt(plaintext, {
|
||||
userId,
|
||||
context: fieldSpec.context,
|
||||
});
|
||||
},
|
||||
|
||||
writeBack: async (doc: Record<string, unknown>, encrypted: EncryptedField) => {
|
||||
const updated = setNestedField(doc, fieldSpec.field, encrypted);
|
||||
const pk = getPartitionKeyValue(doc, partitionKey);
|
||||
await cosmosContainer.item(String(doc.id), pk).replace(updated);
|
||||
},
|
||||
|
||||
onProgress: (progress: MigrationResult) => {
|
||||
if (verbose) {
|
||||
const mode = dryRun ? '[DRY-RUN]' : '[LIVE]';
|
||||
process.stdout.write(
|
||||
`\r ${mode} ${containerName}.${fieldSpec.field}: scanned=${progress.scanned} encrypted=${progress.encrypted} skipped=${progress.skipped} errors=${progress.errors}`
|
||||
);
|
||||
}
|
||||
},
|
||||
});
|
||||
|
||||
if (verbose) {
|
||||
process.stdout.write('\n');
|
||||
}
|
||||
|
||||
return { containerName, field: fieldSpec.field, result };
|
||||
}
|
||||
|
||||
async function migrateProduct(
|
||||
config: ProductMigrationConfig,
|
||||
cosmosClient: CosmosClient,
|
||||
args: CliArgs
|
||||
): Promise<MigrateFieldResult[]> {
|
||||
const dbName = process.env.COSMOS_DATABASE || 'lysnrai';
|
||||
const database = cosmosClient.database(dbName);
|
||||
|
||||
const encryptor = createFieldEncryptor({
|
||||
keyProvider: (process.env.FIELD_ENCRYPT_KEY_PROVIDER as 'memory' | 'env' | 'akv') ?? 'memory',
|
||||
encryptionKey: process.env.FIELD_ENCRYPT_KEY || undefined,
|
||||
keyVaultUrl: process.env.AZURE_KEYVAULT_URL || undefined,
|
||||
mekName: config.mekName,
|
||||
});
|
||||
|
||||
const results: MigrateFieldResult[] = [];
|
||||
|
||||
for (const containerConfig of config.containers) {
|
||||
// Skip if user specified a specific container
|
||||
if (args.container && containerConfig.container !== args.container) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const cosmosContainer = database.container(containerConfig.container);
|
||||
const mode = args.dryRun ? 'DRY-RUN' : 'LIVE';
|
||||
|
||||
console.log(`\n 📦 ${containerConfig.container} (${mode})`);
|
||||
|
||||
for (const fieldSpec of containerConfig.fields) {
|
||||
console.log(` 🔐 Field: ${fieldSpec.field} (context: ${fieldSpec.context})`);
|
||||
|
||||
try {
|
||||
const fieldResult = await migrateContainerField(
|
||||
cosmosContainer,
|
||||
containerConfig.partitionKey,
|
||||
fieldSpec,
|
||||
encryptor,
|
||||
args.batchSize,
|
||||
args.dryRun,
|
||||
args.verbose
|
||||
);
|
||||
|
||||
results.push(fieldResult);
|
||||
|
||||
const r = fieldResult.result;
|
||||
console.log(
|
||||
` scanned=${r.scanned} encrypted=${r.encrypted} skipped=${r.skipped} errors=${r.errors}`
|
||||
);
|
||||
|
||||
if (r.errorDetails.length > 0) {
|
||||
console.log(' Errors:');
|
||||
for (const err of r.errorDetails) {
|
||||
console.log(` - ${err.id}: ${err.error}`);
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
console.error(` ERROR: ${message}`);
|
||||
results.push({
|
||||
containerName: containerConfig.container,
|
||||
field: fieldSpec.field,
|
||||
result: {
|
||||
scanned: 0,
|
||||
encrypted: 0,
|
||||
skipped: 0,
|
||||
errors: 1,
|
||||
errorDetails: [{ id: 'container', error: message }],
|
||||
},
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
// ── Main ──────────────────────────────────────────────
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const args = parseArgs();
|
||||
const startTime = Date.now();
|
||||
|
||||
console.log('\n════════════════════════════════════════════════════');
|
||||
console.log(' ByteLyst Encryption Migration CLI');
|
||||
console.log('════════════════════════════════════════════════════');
|
||||
console.log(` Mode: ${args.dryRun ? 'DRY-RUN (no writes)' : 'LIVE (encrypting!)'}`);
|
||||
console.log(` Product: ${args.product}`);
|
||||
console.log(` Container: ${args.container ?? 'all'}`);
|
||||
console.log(` Batch size: ${args.batchSize}`);
|
||||
console.log(` Database: ${process.env.COSMOS_DATABASE || 'lysnrai'}`);
|
||||
console.log(` Key provider: ${process.env.FIELD_ENCRYPT_KEY_PROVIDER || 'memory'}`);
|
||||
console.log('════════════════════════════════════════════════════');
|
||||
|
||||
const cosmosClient = createCosmosClient();
|
||||
const allResults: MigrateFieldResult[] = [];
|
||||
|
||||
const productsToMigrate =
|
||||
args.product === 'all'
|
||||
? PRODUCT_CONFIGS
|
||||
: PRODUCT_CONFIGS.filter(c => c.productId === args.product);
|
||||
|
||||
if (productsToMigrate.length === 0) {
|
||||
console.error(`\nError: Unknown product '${args.product}'`);
|
||||
console.error(`Available: ${PRODUCT_CONFIGS.map(c => c.productId).join(', ')}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
for (const productConfig of productsToMigrate) {
|
||||
console.log(`\n🏢 ${productConfig.productId} (MEK: ${productConfig.mekName})`);
|
||||
const results = await migrateProduct(productConfig, cosmosClient, args);
|
||||
allResults.push(...results);
|
||||
}
|
||||
|
||||
// ── Summary ──
|
||||
const elapsed = ((Date.now() - startTime) / 1000).toFixed(1);
|
||||
const totals = allResults.reduce(
|
||||
(acc, r) => ({
|
||||
scanned: acc.scanned + r.result.scanned,
|
||||
encrypted: acc.encrypted + r.result.encrypted,
|
||||
skipped: acc.skipped + r.result.skipped,
|
||||
errors: acc.errors + r.result.errors,
|
||||
}),
|
||||
{ scanned: 0, encrypted: 0, skipped: 0, errors: 0 }
|
||||
);
|
||||
|
||||
console.log('\n════════════════════════════════════════════════════');
|
||||
console.log(' Migration Summary');
|
||||
console.log('════════════════════════════════════════════════════');
|
||||
console.log(` Products: ${productsToMigrate.length}`);
|
||||
console.log(` Fields: ${allResults.length}`);
|
||||
console.log(` Scanned: ${totals.scanned}`);
|
||||
console.log(` Encrypted: ${totals.encrypted}`);
|
||||
console.log(` Skipped: ${totals.skipped}`);
|
||||
console.log(` Errors: ${totals.errors}`);
|
||||
console.log(` Duration: ${elapsed}s`);
|
||||
console.log(` Mode: ${args.dryRun ? 'DRY-RUN' : 'LIVE'}`);
|
||||
console.log('════════════════════════════════════════════════════\n');
|
||||
|
||||
// Per-field breakdown
|
||||
if (allResults.length > 0) {
|
||||
console.log(' Per-field breakdown:');
|
||||
console.log(' ┌─────────────────────────────┬────────┬───────────┬─────────┬────────┐');
|
||||
console.log(' │ Container.Field │ Scanned│ Encrypted │ Skipped │ Errors │');
|
||||
console.log(' ├─────────────────────────────┼────────┼───────────┼─────────┼────────┤');
|
||||
for (const r of allResults) {
|
||||
const name = `${r.containerName}.${r.field}`.padEnd(29);
|
||||
const sc = String(r.result.scanned).padStart(7);
|
||||
const en = String(r.result.encrypted).padStart(9);
|
||||
const sk = String(r.result.skipped).padStart(7);
|
||||
const er = String(r.result.errors).padStart(6);
|
||||
console.log(` │ ${name}│${sc} │${en} │${sk} │${er} │`);
|
||||
}
|
||||
console.log(' └─────────────────────────────┴────────┴───────────┴─────────┴────────┘');
|
||||
}
|
||||
|
||||
if (totals.errors > 0) {
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
main().catch(err => {
|
||||
console.error('\nFatal error:', err instanceof Error ? err.message : err);
|
||||
process.exit(1);
|
||||
});
|
||||
16
scripts/package.json
Normal file
16
scripts/package.json
Normal file
@ -0,0 +1,16 @@
|
||||
{
|
||||
"name": "@bytelyst/scripts",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"description": "DevOps scripts for the ByteLyst ecosystem (migration, audit, maintenance)",
|
||||
"dependencies": {
|
||||
"@azure/cosmos": "^4.2.0",
|
||||
"@bytelyst/field-encrypt": "workspace:*"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^22.12.0",
|
||||
"tsx": "^4.19.2",
|
||||
"typescript": "^5.7.3"
|
||||
}
|
||||
}
|
||||
Loading…
Reference in New Issue
Block a user