# 7. Multi-GPU Workloads (Future Path)

> **RTX 5090:** Your first serious CUDA GPU — a stepping stone to multi-GPU and cloud GPU workflows
> **Why it matters:** The skills, code, and workflows you build on one GPU translate directly to multi-GPU and cloud infrastructure

---

## Why Think About Multi-GPU Now?

You don't need multiple GPUs today. But learning on a single RTX 5090 builds skills that scale directly:

```
┌──────────────────────────────────────────────────────────────────────┐
│ GPU Scaling Path                                                     │
│                                                                      │
│  TODAY                                                               │
│  ┌─────────────────────────────────┐                                │
│  │ RTX 5090 Laptop (24 GB VRAM)   │ ← You are here                 │
│  │ Single GPU, local inference     │                                │
│  │ Fine-tuning up to 13B models   │                                │
│  └────────────────┬────────────────┘                                │
│                   │                                                  │
│  NEAR FUTURE      ▼                                                  │
│  ┌─────────────────────────────────┐                                │
│  │ Desktop + eGPU or 2× GPU tower │                                │
│  │ 48–96 GB total VRAM            │                                │
│  │ Fine-tune 70B models           │                                │
│  │ Run 2–3 models simultaneously  │                                │
│  └────────────────┬────────────────┘                                │
│                   │                                                  │
│  CLOUD / PROD     ▼                                                  │
│  ┌─────────────────────────────────┐                                │
│  │ Cloud GPU instances             │                                │
│  │ A100/H100 × 2–8 (80–640 GB)   │                                │
│  │ Train large models              │                                │
│  │ Serve at scale                  │                                │
│  └─────────────────────────────────┘                                │
│                                                                      │
│  SAME CUDA code works at every level ↑                              │
│                                                                      │
└──────────────────────────────────────────────────────────────────────┘
```

---

## What Multi-GPU Enables

| Capability            | Single GPU (24 GB) | 2× GPU (48 GB)  | 4× GPU (96 GB)  | Cloud (640 GB) |
| --------------------- | ------------------ | --------------- | --------------- | -------------- |
| 7B inference          | ✅ Very fast       | ✅ + concurrent | ✅ + concurrent | ✅ at scale    |
| 32B inference         | ✅ Fits in VRAM    | ✅ Very fast    | ✅ Very fast    | ✅ at scale    |
| 70B inference         | ⚠️ Partial GPU     | ✅ Full GPU     | ✅ Very fast    | ✅ at scale    |
| 7B fine-tune (QLoRA)  | ✅ Comfortable     | ✅ Faster       | ✅ Faster       | ✅ Fastest     |
| 13B fine-tune (QLoRA) | ✅ Fits            | ✅ Comfortable  | ✅ Fast         | ✅ Fastest     |
| 70B fine-tune (QLoRA) | ❌ OOM             | ⚠️ Tight        | ✅ Fits         | ✅ Comfortable |
| 7B full fine-tune     | ❌ OOM             | ⚠️ Tight        | ✅ Fits         | ✅ Comfortable |
| Serve 3+ models       | ❌ VRAM limit      | ✅ Yes          | ✅ Yes          | ✅ Yes         |
| SDXL + LLM concurrent | ⚠️ Tight           | ✅ Yes          | ✅ Yes          | ✅ Yes         |

---

## Expansion Options

### Option 1: eGPU (Thunderbolt/USB4)

Connect an external GPU to your Razer Blade via Thunderbolt 4:

```
┌──────────────────────────────────┐     Thunderbolt 4     ┌──────────────────────┐
│ Razer Blade 18                   │◄═══(~40 Gbps)════════►│ eGPU Enclosure       │
│ RTX 5090 (24 GB) — internal     │                        │ RTX 4090 (24 GB)     │
│                                  │                        │ or RTX 5090 (24 GB)  │
└──────────────────────────────────┘                        └──────────────────────┘

Total VRAM: 48 GB (24 + 24)
Limitation: Thunderbolt bandwidth (~40 Gbps) is slower than PCIe 5.0 (~128 Gbps)
Best for: Model serving (latency-tolerant), not training (bandwidth-sensitive)
```

**Recommended eGPU enclosures:**
| Enclosure | Price | GPU Support |
|-----------|-------|-------------|
| Razer Core X | ~$300 | Full-length desktop GPUs |
| Sonnet Breakaway | ~$250 | Most desktop GPUs |
| Akitio Node | ~$200 | Compact form factor |

### Option 2: Desktop Build (Maximum Performance)

Build a dedicated GPU workstation:

```
┌──────────────────────────────────────────────────────────────────────┐
│ Desktop GPU Workstation                                              │
│                                                                      │
│  Motherboard: ASUS/MSI with 2–4× PCIe 5.0 x16 slots               │
│  CPU: Intel i9 or AMD Ryzen 9 (enough PCIe lanes)                  │
│  RAM: 128 GB DDR5                                                    │
│  GPU 1: RTX 5090 (24 GB GDDR7)                                     │
│  GPU 2: RTX 5090 (24 GB GDDR7)                                     │
│  Total VRAM: 48 GB                                                   │
│  PSU: 1200W+ (two 5090s draw ~600W under load)                     │
│                                                                      │
│  Cost: ~$5,000–7,000                                                │
│  Performance: Near-datacenter for most workloads                    │
│                                                                      │
└──────────────────────────────────────────────────────────────────────┘
```

### Option 3: Cloud GPU (On-Demand)

Rent GPU time when you need it:

| Provider    | GPU       | VRAM   | Price/Hour | Best For             |
| ----------- | --------- | ------ | ---------- | -------------------- |
| Lambda Labs | A100 80GB | 80 GB  | ~$1.10     | Training             |
| RunPod      | A100 80GB | 80 GB  | ~$1.64     | Training + inference |
| Vast.ai     | RTX 4090  | 24 GB  | ~$0.30     | Budget inference     |
| AWS (p4d)   | A100 ×8   | 640 GB | ~$32       | Large-scale training |
| Together AI | H100      | 80 GB  | ~$2.50     | Fine-tuning API      |

**Your RTX 5090 code runs identically on cloud GPUs** — same PyTorch, same CUDA.

---

## How to Prepare Now (Single GPU)

### 1. Write GPU-Agnostic Code

Structure your code so it works with any number of GPUs:

```python
"""GPU-agnostic model loading — works with 1 or N GPUs."""
import torch

def get_device():
    """Select the best available device."""
    if torch.cuda.is_available():
        # Multi-GPU: use DataParallel or DistributedDataParallel
        if torch.cuda.device_count() > 1:
            print(f"Using {torch.cuda.device_count()} GPUs")
            return "cuda"  # PyTorch handles multi-GPU distribution
        return "cuda:0"
    elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
        return "mps"
    return "cpu"

device = get_device()
model = MyModel().to(device)

# Wrap for multi-GPU (no-op on single GPU)
if torch.cuda.device_count() > 1:
    model = torch.nn.DataParallel(model)
```

### 2. Learn Model Parallelism Concepts

```python
"""Pipeline parallelism — split model layers across GPUs."""
# Example: split a large model across 2 GPUs

# GPU 0: layers 0–15
# GPU 1: layers 16–31

# With Hugging Face Accelerate:
from accelerate import init_empty_weights, load_checkpoint_and_dispatch

with init_empty_weights():
    model = AutoModelForCausalLM.from_config(config)

model = load_checkpoint_and_dispatch(
    model,
    checkpoint="./model-weights",
    device_map="auto",  # Automatically distributes across available GPUs
)
```

### 3. Ollama Multi-GPU (Already Supported)

Ollama can split a single large model across multiple GPUs:

```bash
# When you have 2 GPUs, Ollama auto-detects and splits
# A 70B model: 24 GB on GPU 0, 16 GB on GPU 1, rest in RAM

# Or manually control GPU assignment
CUDA_VISIBLE_DEVICES=0,1 ollama serve

# Check GPU allocation
nvidia-smi  # Shows VRAM usage per GPU
```

### 4. vLLM (High-Throughput Inference Server)

```bash
# vLLM supports tensor parallelism across GPUs
pip install vllm

# Serve a model across 2 GPUs
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Meta-Llama-3.1-70B-Instruct \
    --tensor-parallel-size 2 \
    --gpu-memory-utilization 0.9

# API compatible with OpenAI format
curl http://localhost:8000/v1/completions -d '{
    "model": "meta-llama/Meta-Llama-3.1-70B-Instruct",
    "prompt": "Hello",
    "max_tokens": 100
}'
```

---

## Scaling Scenarios for Your Projects

### Scenario 1: LysnrAI at Scale

```
TODAY (1× RTX 5090):
  - 1 user, local inference
  - Whisper + Ollama + TTS, one at a time

FUTURE (2× GPU desktop):
  - GPU 0: Ollama (always-on coding model)
  - GPU 1: Whisper + TTS (dedicated)
  - Run all 3 workloads concurrently

PRODUCTION (Cloud):
  - vLLM serving on A100
  - Whisper on dedicated GPU
  - TTS on dedicated GPU
  - Handles 100+ concurrent users
```

### Scenario 2: Fine-Tuning Larger Models

```
TODAY (24 GB VRAM):
  - QLoRA 7B–13B models
  - Training time: 1–4 hours

FUTURE (48 GB VRAM):
  - QLoRA 70B models
  - LoRA FP16 32B models
  - Training time: 4–12 hours

CLOUD (80+ GB VRAM):
  - Full fine-tune 7B–13B models
  - QLoRA 70B+ models
  - Training time: hours with H100
```

### Scenario 3: Image + Text Generation Pipeline

```
TODAY (1× GPU):
  - SDXL OR LLM, not both at once (VRAM constraint)

FUTURE (2× GPU):
  - GPU 0: LLM (Ollama, 32B model, ~19 GB)
  - GPU 1: SDXL + ControlNet (~10 GB)
  - Generate images guided by LLM descriptions

PRODUCTION:
  - Automated content pipeline:
    LLM writes description → SDXL generates image → Whisper adds voiceover
```

---

## Benefits of Starting Single-GPU

| Benefit                  | Description                                                         |
| ------------------------ | ------------------------------------------------------------------- |
| **Code portability**     | CUDA code runs the same on 1, 2, or 100 GPUs                        |
| **Skill foundation**     | Memory management, profiling, optimization skills transfer directly |
| **Cost efficiency**      | Learn on local hardware ($0) before renting cloud ($$$)             |
| **Workflow development** | Build training pipelines, inference servers, batch scripts now      |
| **Hardware literacy**    | Understand VRAM limits, bandwidth, PCIe bottlenecks                 |

---

## Skills You'll Build Toward

| Skill                    | Single GPU (Now)   | Multi-GPU (Future)           | Career Impact           |
| ------------------------ | ------------------ | ---------------------------- | ----------------------- |
| **CUDA programming**     | Kernels, memory    | NCCL, all-reduce             | ML Infrastructure       |
| **Model parallelism**    | Understand concept | Implement tensor/pipeline    | Senior ML Engineer      |
| **Distributed training** | Data loading       | Multi-node coordination      | ML Platform Engineer    |
| **Inference serving**    | Ollama, local API  | vLLM, Triton, load balancing | MLOps / Production      |
| **GPU monitoring**       | nvidia-smi, nvtop  | Cluster monitoring           | DevOps / SRE            |
| **Cost optimization**    | VRAM budget        | Spot instances, right-sizing | FinOps / ML Engineering |
| **Batch scheduling**     | Cron jobs          | Kubernetes, Slurm            | ML Platform             |

### Learning Path

```
┌──────────────────────────────────────────────────────────────────────┐
│ Skills Progression                                                   │
│                                                                      │
│  Level 1 (Now — RTX 5090 Single GPU)                                │
│  ├── PyTorch + CUDA basics                                          │
│  ├── Ollama model serving                                           │
│  ├── QLoRA fine-tuning 7B models                                    │
│  ├── nvidia-smi monitoring                                          │
│  └── TensorRT basic optimization                                    │
│                                                                      │
│  Level 2 (6 months — Same GPU, deeper skills)                       │
│  ├── Custom Triton kernels                                          │
│  ├── vLLM inference server                                          │
│  ├── Advanced quantization (AWQ, GPTQ)                              │
│  ├── Profiling and optimization                                     │
│  └── Model merging and adapter stacking                             │
│                                                                      │
│  Level 3 (12 months — Multi-GPU or Cloud)                           │
│  ├── Multi-GPU inference (tensor parallelism)                       │
│  ├── Distributed training (DDP, FSDP)                               │
│  ├── Cloud GPU workflow (Lambda, RunPod)                             │
│  ├── Production serving with autoscaling                            │
│  └── NCCL and multi-node communication                              │
│                                                                      │
│  Level 4 (Future — Production ML)                                    │
│  ├── Kubernetes + GPU scheduling                                    │
│  ├── Model serving at scale (thousands of requests/sec)             │
│  ├── Training pipelines with experiment tracking                    │
│  ├── Custom model architectures                                     │
│  └── Open-source ML contributions                                   │
│                                                                      │
└──────────────────────────────────────────────────────────────────────┘
```

---

## Cost Planning

### Single GPU (Current)

| Item                               | Cost        | Status    |
| ---------------------------------- | ----------- | --------- |
| Razer Blade 18 RTX 5090            | $5,200      | Purchased |
| Electricity (~200W avg, 8 hrs/day) | ~$15/month  | Ongoing   |
| **Total first year**               | **~$5,380** |           |

### Desktop Upgrade (Future)

| Item                       | Estimated Cost |
| -------------------------- | -------------- |
| Desktop tower + PSU + RAM  | ~$1,500        |
| RTX 5090 desktop GPU       | ~$2,000        |
| Second RTX 5090 (optional) | ~$2,000        |
| **Total (2× GPU desktop)** | **~$5,500**    |

### Cloud Alternative (Per-Use)

| Usage Pattern           | Monthly Cost |
| ----------------------- | ------------ |
| 10 hours/month on A100  | ~$11         |
| 40 hours/month on A100  | ~$44         |
| 160 hours/month on A100 | ~$176        |
| Always-on A100          | ~$792        |

**Break-even vs desktop:** ~12–18 months at heavy usage (40+ hours/month).

---

## Next Steps

- [ ] Write all training and inference scripts to be GPU-count agnostic
- [ ] Install and test vLLM on single GPU with Llama 3.1 8B
- [ ] Practice monitoring GPU memory and compute utilization
- [ ] Try model offloading: run a 70B model with partial CPU/GPU split
- [ ] Explore Lambda Labs or RunPod for a cloud GPU test ($5–10 experiment)
- [ ] Benchmark single GPU throughput to establish a baseline for comparison
