feat(devops): encryption migration CLI with embedded product configs

- scripts/encrypt-migrate.ts — batch-encrypt existing plaintext Cosmos docs
- scripts/ added as pnpm workspace member for clean @bytelyst/* imports
- 10 product configs, 20 containers, 40+ fields
- --dry-run, --product, --container, --batch-size, --verbose flags
- Idempotent via __encrypted sentinel (migrateDocuments helper)
- Updated E2EE roadmap Sprint 6.2 as complete
This commit is contained in:
saravanakumardb1 2026-03-21 13:19:55 -07:00
parent cf9617cda5
commit c252cfd198
5 changed files with 757 additions and 38 deletions

View File

@ -1,7 +1,7 @@
# ByteLyst — End-to-End Encryption Implementation Roadmap # ByteLyst — End-to-End Encryption Implementation Roadmap
> **Purpose:** Phased implementation plan for encryption across the ByteLyst ecosystem. > **Purpose:** Phased implementation plan for encryption across the ByteLyst ecosystem.
> **Status:** Phase 1 + Phase 2 (Sprint 3 + Sprint 4) COMPLETE — 9 backends + native SDKs + client-encrypt + secure-storage-web > **Status:** Phase 1 + Phase 2 COMPLETE (10 backends + native SDKs + client-encrypt + secure-storage-web) + Sprint 6.2 migration CLI
> **Author:** AI Architecture Review > **Author:** AI Architecture Review
> **Last updated:** 2026-03-21 > **Last updated:** 2026-03-21
> **Design doc:** [`END_TO_END_ENCRYPTION_DESIGN.md`](END_TO_END_ENCRYPTION_DESIGN.md) > **Design doc:** [`END_TO_END_ENCRYPTION_DESIGN.md`](END_TO_END_ENCRYPTION_DESIGN.md)
@ -233,14 +233,16 @@ Week 1-2 Week 3-4 Week 5-6 Week 7-8 Week 9-10 Week 11-14
#### 3.2 MindLyst Backend (port 4014) #### 3.2 MindLyst Backend (port 4014)
- [ ] **3.2.1** Add dependency + create encryptor singleton - [x] **3.2.1** Add dependency + create encryptor singleton
- [ ] **3.2.2** Encrypt `content` and `voiceTranscriptText` in memory_items module - [x] **3.2.2** Encrypt `rawContent` and `triageResult.summary` in memory_items module
- [ ] **3.2.3** Encrypt array fields in reflections module - [x] **3.2.3** Encrypt array fields in reflections module
- `repeatedThemes` (string[]), `suggestedAdjustments` (string[]), `vsLastWeek.summary` (string) - `repeatedThemes` (string[]), `postponedItems` (string[]), `roleImbalanceSignals` (string[]), `suggestedAdjustments` (string[]), `vsLastWeek.summary` (string)
- Array fields: JSON-serialize → encrypt → store as single EncryptedField each - Array fields: JSON-serialize → encrypt → store as single EncryptedField each
- [ ] **3.2.4** Update tests - [x] **3.2.4** Encrypt daily-briefs: `greeting`, `priorityItems`, `brainSummaries`, `streakMessage`, `motivationalQuote`
- [x] **3.2.5** Encrypt brains: `rolePrompt`
- [x] **3.2.6** Update tests
**Commit:** `feat(mindlyst): encrypt memory content, voice transcripts, and reflections` **Commit:** `feat(mindlyst): encrypt sensitive fields across all modules`
#### 3.3 NomGap Backend (port 4013) #### 3.3 NomGap Backend (port 4013)
@ -263,14 +265,15 @@ Week 1-2 Week 3-4 Week 5-6 Week 7-8 Week 9-10 Week 11-14
#### 3.5 Remaining Backends (Low Priority — Defer or Skip) #### 3.5 Remaining Backends (Low Priority — Defer or Skip)
| Product | Backend Port | Encrypted Fields | Decision | | Product | Backend Port | Encrypted Fields | Decision |
| ---------------------- | ---------------------------------------------------------------------- | --------------------------------------------------- | -------- | | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | -------- |
| **FlowMonk** (4017) | `tasks.description` | ✅ Done (Sprint 4) — 211 tests | | **FlowMonk** (4017) | `tasks.description` | ✅ Done (Sprint 4) — 211 tests |
| **ChronoMind** (4011) | `timers.description`, `routines.description`, `routines.steps[].notes` | ✅ Done — 182 tests | | **ChronoMind** (4011) | `timers.description`, `routines.description`, `routines.steps[].notes`, `shared-timers.description`, `webhooks.secret`, `webhooks.description` | ✅ Done — 182 tests |
| **PeakPulse** (4010) | `peak-sessions.notes` | ✅ Done — 65 tests | | **PeakPulse** (4010) | `peak-sessions.notes` | ✅ Done — 65 tests |
| **LocalMemGPT** (4019) | `messages.content` | Deferred — SQLite raw SQL, needs different approach | | **MindLyst** (4014) | `memory.rawContent`, `memory.triageResult.summary`, reflections (5 fields), daily-briefs (5 fields), `brains.rolePrompt` | ✅ Done |
| **LocalMemGPT** (4019) | `messages.content`, `conversations.system_prompt` | Deferred — SQLite raw SQL (better-sqlite3), needs SQLite-specific encryption approach |
**Sprint 3 deliverable:** 9 product backends encrypted (LysnrAI, JarvisJr, NoteLett, NomGap, ActionTrail, FlowMonk, ChronoMind, PeakPulse). Only MindLyst (KMP) and LocalMemGPT (SQLite) deferred. **Sprint 3 deliverable:** 10 product backends encrypted (LysnrAI, JarvisJr, NoteLett, NomGap, ActionTrail, FlowMonk, ChronoMind, PeakPulse, MindLyst). Only LocalMemGPT (SQLite) deferred to Sprint 5.
--- ---
@ -449,36 +452,29 @@ Week 1-2 Week 3-4 Week 5-6 Week 7-8 Week 9-10 Week 11-14
#### 6.2 Encryption Migration CLI #### 6.2 Encryption Migration CLI
- [ ] **6.2.1** Create `scripts/encrypt-migrate.ts` — CLI tool for batch encrypting existing plaintext - [x] **6.2.1** Create `scripts/encrypt-migrate.ts` — CLI tool for batch encrypting existing plaintext
```bash ```bash
npx tsx scripts/encrypt-migrate.ts \ npx tsx scripts/encrypt-migrate.ts --product lysnrai --dry-run
--product lysnrai \ npx tsx scripts/encrypt-migrate.ts --product all --dry-run
--container transcripts \ npx tsx scripts/encrypt-migrate.ts --product chronomind --container timers
--field transcriptText \ npx tsx scripts/encrypt-migrate.ts --product mindlyst --batch-size 50 --verbose
--partition-key /userId \
--batch-size 100 \
--dry-run
``` ```
- Uses `@bytelyst/field-encrypt` migration helpers - Uses `@bytelyst/field-encrypt` `migrateDocuments()` helper (idempotent via `__encrypted` sentinel)
- Progress bar, resumable (tracks last processed ID) - Connects directly to Cosmos DB via `@azure/cosmos`
- Dry run mode (report count without modifying) - Per-product configs embedded: 10 products, 20 containers, 40+ fields
- Dry-run mode (count without modifying), verbose per-doc progress
- Nested field support (e.g., `triageResult.summary`)
- JSON-serialized arrays/objects (e.g., reflections string arrays, brainSummaries Record)
- Per-field breakdown table in summary output
- [ ] **6.2.2** Create per-product migration scripts - [x] **6.2.2** Embedded per-product migration configs (no separate shell scripts needed)
``` - Products: lysnrai, jarvisjr, notelett, nomgap, actiontrail, flowmonk, chronomind, peakpulse, mindlyst
scripts/migrations/ - Each config specifies: productId, mekName, containers[], fields[] with context + userIdField + jsonSerialize
├── encrypt-lysnrai.sh - Run `--product all --dry-run` for full ecosystem audit
├── encrypt-jarvisjr.sh
├── encrypt-notelett.sh
├── encrypt-mindlyst.sh
├── encrypt-nomgap.sh
├── encrypt-actiontrail.sh
├── encrypt-flowmonk.sh
└── encrypt-localmemgpt.sh
```
**Commit:** `feat(devops): encryption migration CLI and per-product migration scripts` **Commit:** `feat(devops): encryption migration CLI with embedded product configs`
#### 6.3 Monitoring & Audit #### 6.3 Monitoring & Audit

62
pnpm-lock.yaml generated
View File

@ -393,6 +393,12 @@ importers:
packages/celebrations: {} packages/celebrations: {}
packages/client-encrypt:
devDependencies:
vitest:
specifier: ^3.0.0
version: 3.2.4(@types/debug@4.1.12)(@types/node@22.19.11)(happy-dom@18.0.1)(jiti@2.6.1)(jsdom@28.0.0(@noble/hashes@1.8.0))(lightningcss@1.31.1)(msw@2.12.10(@types/node@22.19.11)(typescript@5.9.3))(terser@5.46.0)(tsx@4.21.0)(yaml@2.8.2)
packages/config: packages/config:
dependencies: dependencies:
zod: zod:
@ -604,6 +610,25 @@ importers:
specifier: ^3.0.0 specifier: ^3.0.0
version: 3.2.4(@types/debug@4.1.12)(@types/node@22.19.11)(happy-dom@18.0.1)(jiti@2.6.1)(jsdom@28.0.0(@noble/hashes@1.8.0))(lightningcss@1.31.1)(msw@2.12.10(@types/node@22.19.11)(typescript@5.9.3))(terser@5.46.0)(tsx@4.21.0)(yaml@2.8.2) version: 3.2.4(@types/debug@4.1.12)(@types/node@22.19.11)(happy-dom@18.0.1)(jiti@2.6.1)(jsdom@28.0.0(@noble/hashes@1.8.0))(lightningcss@1.31.1)(msw@2.12.10(@types/node@22.19.11)(typescript@5.9.3))(terser@5.46.0)(tsx@4.21.0)(yaml@2.8.2)
packages/field-encrypt:
dependencies:
'@azure/identity':
specifier: '>=4.0.0'
version: 4.13.0
'@azure/keyvault-keys':
specifier: '>=4.8.0'
version: 4.10.0(@azure/core-client@1.10.1)
'@bytelyst/errors':
specifier: workspace:*
version: link:../errors
devDependencies:
vitest:
specifier: ^3.0.0
version: 3.2.4(@types/debug@4.1.12)(@types/node@22.19.11)(happy-dom@18.0.1)(jiti@2.6.1)(jsdom@28.0.0(@noble/hashes@1.8.0))(lightningcss@1.31.1)(msw@2.12.10(@types/node@22.19.11)(typescript@5.9.3))(terser@5.46.0)(tsx@4.21.0)(yaml@2.8.2)
zod:
specifier: ^3.24.0
version: 3.25.76
packages/gentle-notifications: {} packages/gentle-notifications: {}
packages/kill-switch-client: {} packages/kill-switch-client: {}
@ -701,6 +726,15 @@ importers:
packages/referral-client: {} packages/referral-client: {}
packages/secure-storage-web:
devDependencies:
fake-indexeddb:
specifier: ^6.0.0
version: 6.2.5
vitest:
specifier: ^3.0.0
version: 3.2.4(@types/debug@4.1.12)(@types/node@22.19.11)(happy-dom@18.0.1)(jiti@2.6.1)(jsdom@28.0.0(@noble/hashes@1.8.0))(lightningcss@1.31.1)(msw@2.12.10(@types/node@22.19.11)(typescript@5.9.3))(terser@5.46.0)(tsx@4.21.0)(yaml@2.8.2)
packages/speech: packages/speech:
devDependencies: devDependencies:
typescript: typescript:
@ -762,6 +796,25 @@ importers:
packages/webhook-dispatch: {} packages/webhook-dispatch: {}
scripts:
dependencies:
'@azure/cosmos':
specifier: ^4.2.0
version: 4.9.1(@azure/core-client@1.10.1)
'@bytelyst/field-encrypt':
specifier: workspace:*
version: link:../packages/field-encrypt
devDependencies:
'@types/node':
specifier: ^22.12.0
version: 22.19.11
tsx:
specifier: ^4.19.2
version: 4.21.0
typescript:
specifier: ^5.7.3
version: 5.9.3
services/extraction-service: services/extraction-service:
dependencies: dependencies:
'@azure/cosmos': '@azure/cosmos':
@ -8082,6 +8135,13 @@ packages:
integrity: sha512-UOiS2in6/Q0FK0R0q6UY9vYpQ21mr/Qn1KOnte7vsACuNJf514WvCCUHSRCPcgjPT2bAhNIJdlE6bVap1GKmeg==, integrity: sha512-UOiS2in6/Q0FK0R0q6UY9vYpQ21mr/Qn1KOnte7vsACuNJf514WvCCUHSRCPcgjPT2bAhNIJdlE6bVap1GKmeg==,
} }
fake-indexeddb@6.2.5:
resolution:
{
integrity: sha512-CGnyrvbhPlWYMngksqrSSUT1BAVP49dZocrHuK0SvtR0D5TMs5wP0o3j7jexDJW01KSadjBp1M/71o/KR3nD1w==,
}
engines: { node: '>=18' }
fast-decode-uri-component@1.0.1: fast-decode-uri-component@1.0.1:
resolution: resolution:
{ {
@ -19522,6 +19582,8 @@ snapshots:
extendable-error@0.1.7: {} extendable-error@0.1.7: {}
fake-indexeddb@6.2.5: {}
fast-decode-uri-component@1.0.1: {} fast-decode-uri-component@1.0.1: {}
fast-deep-equal@3.1.3: {} fast-deep-equal@3.1.3: {}

View File

@ -2,3 +2,4 @@ packages:
- 'packages/*' - 'packages/*'
- 'services/*' - 'services/*'
- 'dashboards/*' - 'dashboards/*'
- 'scripts'

644
scripts/encrypt-migrate.ts Normal file
View File

@ -0,0 +1,644 @@
#!/usr/bin/env npx tsx
/* eslint-disable no-console */
/**
* Encryption Migration CLI Batch-encrypt existing plaintext Cosmos documents.
*
* Uses @bytelyst/field-encrypt migrateDocuments() to idempotently encrypt
* plaintext fields across all product backends.
*
* Usage:
* npx tsx scripts/encrypt-migrate.ts --product lysnrai --dry-run
* npx tsx scripts/encrypt-migrate.ts --product lysnrai --container transcripts
* npx tsx scripts/encrypt-migrate.ts --product all --dry-run
* npx tsx scripts/encrypt-migrate.ts --product chronomind --batch-size 50
*
* Required env vars:
* COSMOS_ENDPOINT Azure Cosmos DB endpoint
* COSMOS_KEY Azure Cosmos DB key
* COSMOS_DATABASE Database name (default: lysnrai)
* FIELD_ENCRYPT_KEY_PROVIDER 'memory' | 'env' | 'akv' (default: memory)
* FIELD_ENCRYPT_KEY Hex key (for 'env' provider)
* AZURE_KEYVAULT_URL AKV URL (for 'akv' provider)
*
* Per-product MEK names are defined in migration configs.
*/
import { CosmosClient, type Container } from '@azure/cosmos';
import {
createFieldEncryptor,
migrateDocuments,
type MigrationResult,
type FieldEncryptor,
type EncryptedField,
} from '@bytelyst/field-encrypt';
// ── Types ──────────────────────────────────────────────
interface FieldSpec {
/** The field path (dot-separated for nested, e.g. 'triageResult.summary'). */
field: string;
/** Encryption context label. */
context: string;
/** The property on the doc that holds the userId for DEK scoping. Default: 'userId'. */
userIdField?: string;
/** If true, the field is a string[] or Record — JSON-serialize before encrypting. */
jsonSerialize?: boolean;
}
interface ContainerMigrationConfig {
/** Cosmos container name. */
container: string;
/** Partition key path (e.g. '/userId'). */
partitionKey: string;
/** Fields to encrypt in this container. */
fields: FieldSpec[];
}
interface ProductMigrationConfig {
/** Product ID. */
productId: string;
/** MEK name for this product. */
mekName: string;
/** Containers to migrate. */
containers: ContainerMigrationConfig[];
}
// ── Field accessor helpers ────────────────────────────
function getNestedField(doc: Record<string, unknown>, path: string): unknown {
const parts = path.split('.');
let current: unknown = doc;
for (const part of parts) {
if (current == null || typeof current !== 'object') return undefined;
current = (current as Record<string, unknown>)[part];
}
return current;
}
function setNestedField(
doc: Record<string, unknown>,
path: string,
value: unknown
): Record<string, unknown> {
const parts = path.split('.');
const result = { ...doc };
if (parts.length === 1) {
result[parts[0]] = value;
return result;
}
// Deep clone the nested path
let current = result;
for (let i = 0; i < parts.length - 1; i++) {
const nested = current[parts[i]];
if (nested == null || typeof nested !== 'object') return result;
current[parts[i]] = { ...(nested as Record<string, unknown>) };
current = current[parts[i]] as Record<string, unknown>;
}
current[parts[parts.length - 1]] = value;
return result;
}
// ── Product migration configs ─────────────────────────
const PRODUCT_CONFIGS: ProductMigrationConfig[] = [
// ── LysnrAI (port 4015) ──
{
productId: 'lysnrai',
mekName: 'lysnr-mek',
containers: [
{
container: 'transcripts',
partitionKey: '/userId',
fields: [{ field: 'transcriptText', context: 'transcripts' }],
},
{
container: 'sessions',
partitionKey: '/userId',
fields: [{ field: 'notes', context: 'sessions' }],
},
],
},
// ── JarvisJr (port 4012) ──
{
productId: 'jarvisjr',
mekName: 'jarvisjr-mek',
containers: [
{
container: 'jarvis_sessions',
partitionKey: '/userId',
fields: [{ field: 'notes', context: 'jarvis-sessions' }],
},
{
container: 'jarvis_memory',
partitionKey: '/agentId',
fields: [{ field: 'content', context: 'jarvis-memory', userIdField: 'agentId' }],
},
],
},
// ── NoteLett (port 4016) ──
{
productId: 'notelett',
mekName: 'notelett-mek',
containers: [
{
container: 'notes',
partitionKey: '/workspaceId',
fields: [{ field: 'body', context: 'notes', userIdField: 'createdBy' }],
},
],
},
// ── NomGap (port 4013) ──
{
productId: 'nomgap',
mekName: 'nomgap-mek',
containers: [
{
container: 'meal_logs',
partitionKey: '/userId',
fields: [{ field: 'notes', context: 'meal-logs' }],
},
{
container: 'fasting_sessions',
partitionKey: '/userId',
fields: [{ field: 'notes', context: 'fasting-sessions' }],
},
{
container: 'weight_log',
partitionKey: '/userId',
fields: [{ field: 'notes', context: 'weight-log' }],
},
],
},
// ── ActionTrail (port 4018) ──
{
productId: 'actiontrail',
mekName: 'actiontrail-mek',
containers: [
{
container: 'trail_actions',
partitionKey: '/userId',
fields: [
{ field: 'before', context: 'trail-actions', jsonSerialize: true },
{ field: 'after', context: 'trail-actions-after', jsonSerialize: true },
],
},
{
container: 'trail_reverts',
partitionKey: '/userId',
fields: [
{ field: 'beforeState', context: 'trail-reverts', jsonSerialize: true },
{ field: 'afterState', context: 'trail-reverts-after', jsonSerialize: true },
{ field: 'revertPayload', context: 'trail-reverts-payload', jsonSerialize: true },
],
},
],
},
// ── FlowMonk (port 4017) ──
{
productId: 'flowmonk',
mekName: 'flowmonk-mek',
containers: [
{
container: 'tasks',
partitionKey: '/userId',
fields: [{ field: 'description', context: 'tasks' }],
},
],
},
// ── ChronoMind (port 4011) ──
{
productId: 'chronomind',
mekName: 'chronomind-mek',
containers: [
{
container: 'timers',
partitionKey: '/userId',
fields: [{ field: 'description', context: 'timers' }],
},
{
container: 'routines',
partitionKey: '/userId',
fields: [
{ field: 'description', context: 'routines' },
// steps[].notes handled differently — see note below
],
},
{
container: 'shared_timers',
partitionKey: '/householdId',
fields: [{ field: 'description', context: 'shared-timers', userIdField: 'createdBy' }],
},
{
container: 'webhook_subscriptions',
partitionKey: '/userId',
fields: [
{ field: 'secret', context: 'webhooks' },
{ field: 'description', context: 'webhook-desc' },
],
},
],
},
// ── PeakPulse (port 4010) ──
{
productId: 'peakpulse',
mekName: 'peakpulse-mek',
containers: [
{
container: 'peak_sessions',
partitionKey: '/userId',
fields: [{ field: 'notes', context: 'peak-sessions' }],
},
],
},
// ── MindLyst (port 4014) ──
{
productId: 'mindlyst',
mekName: 'mindlyst-mek',
containers: [
{
container: 'memory_items',
partitionKey: '/userId',
fields: [
{ field: 'rawContent', context: 'memory' },
{ field: 'triageResult.summary', context: 'memory-triage' },
],
},
{
container: 'reflections',
partitionKey: '/userId',
fields: [
{ field: 'repeatedThemes', context: 'reflection-themes', jsonSerialize: true },
{ field: 'postponedItems', context: 'reflection-postponed', jsonSerialize: true },
{ field: 'roleImbalanceSignals', context: 'reflection-imbalance', jsonSerialize: true },
{ field: 'suggestedAdjustments', context: 'reflection-adjustments', jsonSerialize: true },
{ field: 'vsLastWeek.summary', context: 'reflection-vs' },
],
},
{
container: 'daily_briefs',
partitionKey: '/userId',
fields: [
{ field: 'greeting', context: 'daily-briefs' },
{ field: 'priorityItems', context: 'brief-priorities', jsonSerialize: true },
{ field: 'brainSummaries', context: 'brief-summaries', jsonSerialize: true },
{ field: 'streakMessage', context: 'brief-streak' },
{ field: 'motivationalQuote', context: 'brief-quote' },
],
},
{
container: 'brains',
partitionKey: '/userId',
fields: [{ field: 'rolePrompt', context: 'brains' }],
},
],
},
];
// ── CLI argument parsing ──────────────────────────────
interface CliArgs {
product: string;
container?: string;
batchSize: number;
dryRun: boolean;
verbose: boolean;
}
function parseArgs(): CliArgs {
const args = process.argv.slice(2);
const result: CliArgs = {
product: '',
batchSize: 100,
dryRun: false,
verbose: false,
};
for (let i = 0; i < args.length; i++) {
switch (args[i]) {
case '--product':
case '-p':
result.product = args[++i] ?? '';
break;
case '--container':
case '-c':
result.container = args[++i];
break;
case '--batch-size':
case '-b':
result.batchSize = parseInt(args[++i] ?? '100', 10);
break;
case '--dry-run':
case '-n':
result.dryRun = true;
break;
case '--verbose':
case '-v':
result.verbose = true;
break;
case '--help':
case '-h':
printUsage();
process.exit(0);
break;
default:
console.error(`Unknown argument: ${args[i]}`);
printUsage();
process.exit(1);
}
}
if (!result.product) {
console.error('Error: --product is required\n');
printUsage();
process.exit(1);
}
return result;
}
function printUsage(): void {
console.log(`
ByteLyst Encryption Migration CLI
Batch-encrypt existing plaintext Cosmos documents
Usage:
npx tsx scripts/encrypt-migrate.ts --product <id> [options]
Options:
--product, -p <id> Product ID or 'all' (required)
--container, -c <name> Migrate only this container (optional)
--batch-size, -b <n> Documents per batch (default: 100)
--dry-run, -n Count without encrypting
--verbose, -v Show per-document progress
--help, -h Show this help
Products: ${PRODUCT_CONFIGS.map(c => c.productId).join(', ')}
Environment:
COSMOS_ENDPOINT Azure Cosmos DB endpoint
COSMOS_KEY Azure Cosmos DB key
COSMOS_DATABASE Database name (default: lysnrai)
FIELD_ENCRYPT_KEY_PROVIDER 'memory' | 'env' | 'akv' (default: memory)
FIELD_ENCRYPT_KEY Hex key (for 'env' provider)
AZURE_KEYVAULT_URL AKV URL (for 'akv' provider)
Examples:
npx tsx scripts/encrypt-migrate.ts --product lysnrai --dry-run
npx tsx scripts/encrypt-migrate.ts --product all --dry-run
npx tsx scripts/encrypt-migrate.ts --product chronomind --container timers
npx tsx scripts/encrypt-migrate.ts --product mindlyst --batch-size 50
`);
}
// ── Cosmos helpers ────────────────────────────────────
function createCosmosClient(): CosmosClient {
const endpoint = process.env.COSMOS_ENDPOINT;
const key = process.env.COSMOS_KEY;
if (!endpoint || !key) {
console.error('Error: COSMOS_ENDPOINT and COSMOS_KEY environment variables are required');
process.exit(1);
}
return new CosmosClient({ endpoint, key });
}
function getPartitionKeyValue(doc: Record<string, unknown>, partitionKeyPath: string): string {
const field = partitionKeyPath.replace(/^\//, '');
return String(doc[field] ?? '');
}
// ── Migration engine ──────────────────────────────────
interface MigrateFieldResult {
containerName: string;
field: string;
result: MigrationResult;
}
async function migrateContainerField(
cosmosContainer: Container,
partitionKey: string,
fieldSpec: FieldSpec,
encryptor: FieldEncryptor,
batchSize: number,
dryRun: boolean,
verbose: boolean
): Promise<MigrateFieldResult> {
const containerName = cosmosContainer.id;
const userIdProp = fieldSpec.userIdField ?? 'userId';
const result = await migrateDocuments<Record<string, unknown>>({
batchSize,
dryRun,
fetchBatch: async (offset: number, size: number) => {
const query = `SELECT * FROM c ORDER BY c._ts OFFSET ${offset} LIMIT ${size}`;
const { resources } = await cosmosContainer.items.query(query).fetchAll();
return resources;
},
getId: (doc: Record<string, unknown>) => String(doc.id ?? 'unknown'),
getField: (doc: Record<string, unknown>) => getNestedField(doc, fieldSpec.field),
encryptValue: async (plaintext: string, doc: Record<string, unknown>) => {
const userId = String(doc[userIdProp] ?? 'unknown');
return encryptor.encrypt(plaintext, {
userId,
context: fieldSpec.context,
});
},
writeBack: async (doc: Record<string, unknown>, encrypted: EncryptedField) => {
const updated = setNestedField(doc, fieldSpec.field, encrypted);
const pk = getPartitionKeyValue(doc, partitionKey);
await cosmosContainer.item(String(doc.id), pk).replace(updated);
},
onProgress: (progress: MigrationResult) => {
if (verbose) {
const mode = dryRun ? '[DRY-RUN]' : '[LIVE]';
process.stdout.write(
`\r ${mode} ${containerName}.${fieldSpec.field}: scanned=${progress.scanned} encrypted=${progress.encrypted} skipped=${progress.skipped} errors=${progress.errors}`
);
}
},
});
if (verbose) {
process.stdout.write('\n');
}
return { containerName, field: fieldSpec.field, result };
}
async function migrateProduct(
config: ProductMigrationConfig,
cosmosClient: CosmosClient,
args: CliArgs
): Promise<MigrateFieldResult[]> {
const dbName = process.env.COSMOS_DATABASE || 'lysnrai';
const database = cosmosClient.database(dbName);
const encryptor = createFieldEncryptor({
keyProvider: (process.env.FIELD_ENCRYPT_KEY_PROVIDER as 'memory' | 'env' | 'akv') ?? 'memory',
encryptionKey: process.env.FIELD_ENCRYPT_KEY || undefined,
keyVaultUrl: process.env.AZURE_KEYVAULT_URL || undefined,
mekName: config.mekName,
});
const results: MigrateFieldResult[] = [];
for (const containerConfig of config.containers) {
// Skip if user specified a specific container
if (args.container && containerConfig.container !== args.container) {
continue;
}
const cosmosContainer = database.container(containerConfig.container);
const mode = args.dryRun ? 'DRY-RUN' : 'LIVE';
console.log(`\n 📦 ${containerConfig.container} (${mode})`);
for (const fieldSpec of containerConfig.fields) {
console.log(` 🔐 Field: ${fieldSpec.field} (context: ${fieldSpec.context})`);
try {
const fieldResult = await migrateContainerField(
cosmosContainer,
containerConfig.partitionKey,
fieldSpec,
encryptor,
args.batchSize,
args.dryRun,
args.verbose
);
results.push(fieldResult);
const r = fieldResult.result;
console.log(
` scanned=${r.scanned} encrypted=${r.encrypted} skipped=${r.skipped} errors=${r.errors}`
);
if (r.errorDetails.length > 0) {
console.log(' Errors:');
for (const err of r.errorDetails) {
console.log(` - ${err.id}: ${err.error}`);
}
}
} catch (err) {
const message = err instanceof Error ? err.message : String(err);
console.error(` ERROR: ${message}`);
results.push({
containerName: containerConfig.container,
field: fieldSpec.field,
result: {
scanned: 0,
encrypted: 0,
skipped: 0,
errors: 1,
errorDetails: [{ id: 'container', error: message }],
},
});
}
}
}
return results;
}
// ── Main ──────────────────────────────────────────────
async function main(): Promise<void> {
const args = parseArgs();
const startTime = Date.now();
console.log('\n════════════════════════════════════════════════════');
console.log(' ByteLyst Encryption Migration CLI');
console.log('════════════════════════════════════════════════════');
console.log(` Mode: ${args.dryRun ? 'DRY-RUN (no writes)' : 'LIVE (encrypting!)'}`);
console.log(` Product: ${args.product}`);
console.log(` Container: ${args.container ?? 'all'}`);
console.log(` Batch size: ${args.batchSize}`);
console.log(` Database: ${process.env.COSMOS_DATABASE || 'lysnrai'}`);
console.log(` Key provider: ${process.env.FIELD_ENCRYPT_KEY_PROVIDER || 'memory'}`);
console.log('════════════════════════════════════════════════════');
const cosmosClient = createCosmosClient();
const allResults: MigrateFieldResult[] = [];
const productsToMigrate =
args.product === 'all'
? PRODUCT_CONFIGS
: PRODUCT_CONFIGS.filter(c => c.productId === args.product);
if (productsToMigrate.length === 0) {
console.error(`\nError: Unknown product '${args.product}'`);
console.error(`Available: ${PRODUCT_CONFIGS.map(c => c.productId).join(', ')}`);
process.exit(1);
}
for (const productConfig of productsToMigrate) {
console.log(`\n🏢 ${productConfig.productId} (MEK: ${productConfig.mekName})`);
const results = await migrateProduct(productConfig, cosmosClient, args);
allResults.push(...results);
}
// ── Summary ──
const elapsed = ((Date.now() - startTime) / 1000).toFixed(1);
const totals = allResults.reduce(
(acc, r) => ({
scanned: acc.scanned + r.result.scanned,
encrypted: acc.encrypted + r.result.encrypted,
skipped: acc.skipped + r.result.skipped,
errors: acc.errors + r.result.errors,
}),
{ scanned: 0, encrypted: 0, skipped: 0, errors: 0 }
);
console.log('\n════════════════════════════════════════════════════');
console.log(' Migration Summary');
console.log('════════════════════════════════════════════════════');
console.log(` Products: ${productsToMigrate.length}`);
console.log(` Fields: ${allResults.length}`);
console.log(` Scanned: ${totals.scanned}`);
console.log(` Encrypted: ${totals.encrypted}`);
console.log(` Skipped: ${totals.skipped}`);
console.log(` Errors: ${totals.errors}`);
console.log(` Duration: ${elapsed}s`);
console.log(` Mode: ${args.dryRun ? 'DRY-RUN' : 'LIVE'}`);
console.log('════════════════════════════════════════════════════\n');
// Per-field breakdown
if (allResults.length > 0) {
console.log(' Per-field breakdown:');
console.log(' ┌─────────────────────────────┬────────┬───────────┬─────────┬────────┐');
console.log(' │ Container.Field │ Scanned│ Encrypted │ Skipped │ Errors │');
console.log(' ├─────────────────────────────┼────────┼───────────┼─────────┼────────┤');
for (const r of allResults) {
const name = `${r.containerName}.${r.field}`.padEnd(29);
const sc = String(r.result.scanned).padStart(7);
const en = String(r.result.encrypted).padStart(9);
const sk = String(r.result.skipped).padStart(7);
const er = String(r.result.errors).padStart(6);
console.log(`${name}${sc}${en}${sk}${er}`);
}
console.log(' └─────────────────────────────┴────────┴───────────┴─────────┴────────┘');
}
if (totals.errors > 0) {
process.exit(1);
}
}
main().catch(err => {
console.error('\nFatal error:', err instanceof Error ? err.message : err);
process.exit(1);
});

16
scripts/package.json Normal file
View File

@ -0,0 +1,16 @@
{
"name": "@bytelyst/scripts",
"version": "0.1.0",
"private": true,
"type": "module",
"description": "DevOps scripts for the ByteLyst ecosystem (migration, audit, maintenance)",
"dependencies": {
"@azure/cosmos": "^4.2.0",
"@bytelyst/field-encrypt": "workspace:*"
},
"devDependencies": {
"@types/node": "^22.12.0",
"tsx": "^4.19.2",
"typescript": "^5.7.3"
}
}