# 1. Raw GPU Inference Speed

> **RTX 5090:** 2–4× faster inference on all models ≤32B compared to Mac M4 Pro
> **Why it matters:** Faster coding suggestions, faster conversations, faster iteration

---

## What Is GPU Inference?

When you run a model like `qwen2.5-coder:32b` through Ollama, the GPU does the heavy lifting — multiplying billions of numbers (matrix operations) to generate each token of the response. The speed of this process depends on:

1. **VRAM bandwidth** — how fast data moves within the GPU
2. **Compute cores** — how many operations run in parallel
3. **VRAM capacity** — whether the full model fits without spilling to CPU RAM

```
┌──────────────────────────────────────────────────────────────────────┐
│ Token Generation Pipeline                                            │
│                                                                      │
│  Prompt → [Tokenize] → [GPU: Matrix Multiply] → [Sample] → Token   │
│                              ▲                                       │
│                              │                                       │
│                    This is the bottleneck.                           │
│                    RTX 5090 does this 2–4× faster.                  │
└──────────────────────────────────────────────────────────────────────┘
```

---

## Performance: Mac vs Razer

| Model                     | Mac M4 Pro (MPS) | Razer RTX 5090 (CUDA) | Speedup |
| ------------------------- | ---------------- | --------------------- | ------- |
| llama3.1:8b (4.9 GB)      | ~50–70 tok/s     | ~100–150 tok/s        | ~2×     |
| qwen2.5-coder:7b (4.7 GB) | ~40–60 tok/s     | ~80–120 tok/s         | ~2×     |
| qwen2.5-coder:32b (19 GB) | ~15–25 tok/s     | ~40–60 tok/s          | ~2.5×   |
| deepseek-r1:32b (19 GB)   | ~15–25 tok/s     | ~40–60 tok/s          | ~2.5×   |
| sematre/orpheus:en (4 GB) | ~realtime        | ~2–3× realtime        | ~2.5×   |

### Why the RTX 5090 Is Faster

```
┌─────────────────────────┬──────────────────────┬──────────────────────────┐
│ Metric                  │ Mac M4 Pro           │ RTX 5090                 │
├─────────────────────────┼──────────────────────┼──────────────────────────┤
│ GPU memory bandwidth    │ ~273 GB/s (shared)   │ ~1,000+ GB/s (GDDR7)    │
│ Compute cores           │ 20 Metal cores       │ ~18,000 CUDA cores       │
│ Tensor cores            │ None (Neural Engine) │ 5th/6th gen Tensor cores │
│ FP16 throughput         │ ~25 TFLOPS           │ ~200+ TFLOPS             │
│ Model in memory?        │ Yes (unified 48 GB)  │ Yes (24 GB VRAM)         │
└─────────────────────────┴──────────────────────┴──────────────────────────┘
```

The RTX 5090's GDDR7 bandwidth is ~4× higher, and it has ~8× the raw FP16 compute throughput. The actual speedup is 2–4× (not 8×) because inference is mostly **memory-bandwidth bound**, not compute-bound — the GPU spends most of its time reading model weights, not computing.

---

## How to Use It

### Basic: Ollama (Already Set Up)

Ollama runs natively on Windows and uses CUDA automatically. No extra config needed.

```bash
# From WSL2 or Windows terminal
ollama run qwen2.5-coder:32b "Write a Fastify route that validates input with Zod"
```

### Interactive Coding Assistant

```bash
# Start a conversation with the 32B coding model
ollama run qwen2.5-coder:32b

# Or use the 7B model for quick questions (faster response start)
ollama run qwen2.5-coder:7b
```

### From the Dashboard

```bash
cd ~/code/mygh/learning_ai_common_plat/__LOCAL_LLMs
bash start-dashboard.sh
# Open http://localhost:3000 — model status + inference visible
```

### Benchmark Your Actual Speed

```bash
# Quick benchmark — measure tokens per second
time ollama run qwen2.5-coder:7b "Write a Python function that implements binary search" --verbose 2>&1 | tail -5

# Compare models
for model in llama3.1:8b qwen2.5-coder:7b qwen2.5-coder:32b; do
    echo "=== $model ==="
    ollama run "$model" "Hello world" --verbose 2>&1 | grep "eval rate"
done
```

### API Access (for Scripts/Apps)

```bash
# Ollama exposes a REST API at localhost:11434
curl -s http://localhost:11434/api/generate -d '{
  "model": "qwen2.5-coder:32b",
  "prompt": "Explain CUDA memory hierarchy in 3 sentences",
  "stream": false
}' | python3 -c "import sys,json; print(json.load(sys.stdin)['response'])"
```

---

## Benefits

### For Your LysnrAI / MindLyst Projects

- **Faster code generation** — 32B model responses in ~0.5–1.5s vs ~2–4s on Mac
- **More context in less time** — can process longer prompts without waiting
- **Better for agentic workflows** — LangGraph agents that call LLMs multiple times per step run 2–4× faster end-to-end
- **Batch processing** — generate embeddings, summaries, or classifications for hundreds of items quickly

### For Daily Coding

- **Near-instant small model responses** — 7B at 80–120 tok/s feels like reading speed
- **Viable 32B coding assistant** — 40–60 tok/s is fast enough for real-time pair programming
- **DeepSeek-R1 reasoning** — chain-of-thought at 40–60 tok/s makes complex reasoning practical

---

## Skills You'll Learn

| Skill                      | What You'll Learn                                                    | Why It Matters                         |
| -------------------------- | -------------------------------------------------------------------- | -------------------------------------- |
| **GPU memory management**  | How VRAM allocation works, model offloading, quantization trade-offs | Core ML engineering skill              |
| **CUDA profiling**         | Using `nvidia-smi`, `nvtop`, watching GPU utilization                | Essential for optimizing AI workloads  |
| **Quantization**           | Q4 vs Q8 vs FP16 — speed/quality trade-offs                          | Industry-standard model deployment     |
| **Inference optimization** | Batch size, context length, KV cache tuning                          | Key for production AI systems          |
| **Model selection**        | When to use 7B vs 32B vs 70B for different tasks                     | Practical AI engineering judgment      |
| **REST API integration**   | Building apps that call local LLM APIs                               | Directly applicable to LysnrAI backend |

---

## Advanced: Tuning Ollama for Performance

```bash
# Set number of GPU layers (default: all)
# Useful if you want to run 2 models with partial GPU offload
OLLAMA_NUM_GPU=999 ollama serve

# Monitor GPU during inference
watch -n 0.5 nvidia-smi

# Or install nvtop for a richer GPU monitor
sudo apt install nvtop
nvtop
```

### Ollama Environment Variables

| Variable                   | Default | Description                             |
| -------------------------- | ------- | --------------------------------------- |
| `OLLAMA_NUM_PARALLEL`      | 1       | Concurrent request slots                |
| `OLLAMA_MAX_LOADED_MODELS` | 1       | Models kept in VRAM simultaneously      |
| `OLLAMA_FLASH_ATTENTION`   | true    | Use flash attention (faster, less VRAM) |
| `OLLAMA_GPU_OVERHEAD`      | 0       | Reserve VRAM (bytes) for other apps     |

---

## Next Steps

- [ ] Benchmark all 5 models on the Razer and record actual tok/s
- [ ] Try `OLLAMA_NUM_PARALLEL=4` for concurrent requests
- [ ] Experiment with `OLLAMA_MAX_LOADED_MODELS=2` to keep 7B + 32B hot
- [ ] Build a simple script that compares Mac vs Razer inference times
