# 4. Fine-Tuning / Training

> **RTX 5090:** 24 GB VRAM enables LoRA fine-tuning of 7B–13B models locally
> **Why it matters:** Customize models for your specific use cases — coding style, domain knowledge, voice commands

---

## What Is Fine-Tuning?

Fine-tuning takes a pre-trained model (like Llama 3.1 8B) and trains it further on your own data to specialize its behavior. Instead of training from scratch (which costs millions), you adjust a small fraction of the model's weights.

```
┌──────────────────────────────────────────────────────────────────────┐
│ Fine-Tuning vs Prompting                                             │
│                                                                      │
│  Prompting:    "You are a coding assistant for TypeScript..."       │
│                Works OK, but limited by context window              │
│                Model doesn't truly "learn" your preferences         │
│                                                                      │
│  Fine-Tuning:  Train on 1000s of your code examples                │
│                Model internalizes your coding patterns              │
│                Better quality, no prompt overhead, faster inference  │
│                                                                      │
│  LoRA:         Fine-tune only ~1–5% of parameters                  │
│                Needs 16–24 GB VRAM (fits RTX 5090)                  │
│                Training time: hours, not days                        │
│                                                                      │
└──────────────────────────────────────────────────────────────────────┘
```

### Why Not on Mac?

| Aspect            | Mac M4 Pro (MPS)               | RTX 5090 (CUDA)    |
| ----------------- | ------------------------------ | ------------------ |
| Training support  | Limited MPS support            | Full CUDA + cuDNN  |
| Framework support | PyTorch MPS (some ops missing) | Full PyTorch CUDA  |
| VRAM for training | ~30 GB usable (shared)         | 24 GB dedicated    |
| Memory bandwidth  | ~273 GB/s                      | ~1,000+ GB/s       |
| Training speed    | 5–10× slower than CUDA         | Baseline           |
| LoRA libraries    | Partial compatibility          | Full compatibility |

**Training is compute AND memory bandwidth intensive** — the RTX 5090's ~1 TB/s VRAM bandwidth makes it 5–10× faster than MPS for gradient computation.

---

## Fine-Tuning Methods

### LoRA (Low-Rank Adaptation) — Recommended

Trains only small "adapter" matrices (~1–5% of model parameters). The base model stays frozen.

```
┌──────────────────────────────────────────────────────────────────────┐
│ LoRA Architecture                                                    │
│                                                                      │
│  Base Model (frozen, ~16 GB)                                         │
│  ┌─────────────────────────────────────────────┐                    │
│  │ Layer 1: [Attention] [FFN]                   │                    │
│  │ Layer 2: [Attention] [FFN]                   │ ← Not modified    │
│  │ ...                                          │                    │
│  │ Layer N: [Attention] [FFN]                   │                    │
│  └─────────────────────────────────────────────┘                    │
│       ↕ small adapters (rank 8–64)                                   │
│  ┌─────────────────────────────────────────────┐                    │
│  │ LoRA Adapter A (64 KB per layer)             │ ← Trainable       │
│  │ LoRA Adapter B (64 KB per layer)             │ ← Trainable       │
│  └─────────────────────────────────────────────┘                    │
│                                                                      │
│  Total trainable params: ~10–50 MB (vs 8–16 GB base)               │
│  VRAM needed: ~18–22 GB for 7B model                                │
│  Training time: ~1–4 hours for 1000 examples                        │
│                                                                      │
└──────────────────────────────────────────────────────────────────────┘
```

### QLoRA (Quantized LoRA) — For Larger Models

Loads the base model in 4-bit quantization, trains LoRA adapters in FP16. Halves memory requirements.

| Method            | 7B Model VRAM | 13B Model VRAM | 32B Model VRAM |
| ----------------- | ------------- | -------------- | -------------- |
| Full fine-tune    | ~56 GB        | ~104 GB        | ~256 GB        |
| LoRA (FP16)       | ~18 GB        | ~32 GB         | ~72 GB         |
| **QLoRA (4-bit)** | **~8 GB**     | **~14 GB**     | **~22 GB**     |

**QLoRA at 32B fits in 24 GB VRAM** — this is remarkable for a laptop.

---

## How to Use It

### Setup: Install Training Libraries

```bash
# Activate the existing TTS venv (or create a new one)
cd ~/code/mygh/learning_ai_common_plat/__LOCAL_LLMs
python3.12 -m venv .venv-training
source .venv-training/bin/activate

# Install training stack
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
pip install transformers datasets accelerate peft bitsandbytes trl
pip install wandb  # optional: experiment tracking
```

### Example 1: Fine-Tune Llama 3.1 8B for Your Coding Style

```python
"""Fine-tune Llama 3.1 8B with LoRA on your coding examples."""
from datasets import Dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from peft import LoraConfig, get_peft_model
from trl import SFTTrainer

# 1. Prepare your data
# Format: instruction → response pairs from YOUR code
training_data = [
    {
        "instruction": "Write a Fastify route that creates a user with Zod validation",
        "response": """import { z } from 'zod';
import type { FastifyInstance } from 'fastify';

const CreateUserSchema = z.object({
  email: z.string().email(),
  name: z.string().min(1).max(100),
  productId: z.string(),
});

export async function userRoutes(app: FastifyInstance) {
  app.post('/users', async (req, reply) => {
    const body = CreateUserSchema.parse(req.body);
    const user = await req.server.userRepository.create(body);
    return reply.status(201).send(user);
  });
}"""
    },
    # Add 100–1000 more examples from your actual codebase
]

dataset = Dataset.from_list([
    {"text": f"### Instruction:\n{d['instruction']}\n\n### Response:\n{d['response']}"}
    for d in training_data
])

# 2. Load model in 4-bit (QLoRA)
model_name = "meta-llama/Meta-Llama-3.1-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    load_in_4bit=True,
    device_map="auto",
)

# 3. Configure LoRA
lora_config = LoraConfig(
    r=16,                # Rank (higher = more capacity, more VRAM)
    lora_alpha=32,       # Scaling factor
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: ~10M / total: ~8B (0.13%)

# 4. Train
training_args = TrainingArguments(
    output_dir="./lora-llama-coding",
    num_train_epochs=3,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    warmup_steps=10,
    logging_steps=10,
    save_steps=100,
    fp16=True,
)

trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    args=training_args,
    tokenizer=tokenizer,
    dataset_text_field="text",
    max_seq_length=2048,
)

trainer.train()

# 5. Save the adapter (small file, ~50 MB)
model.save_pretrained("./lora-llama-coding")
tokenizer.save_pretrained("./lora-llama-coding")
```

### Example 2: Fine-Tune for LysnrAI Voice Commands

```python
# Train a model to understand your specific voice command patterns
training_data = [
    {"instruction": "Parse: remind me to call john tomorrow at 3pm",
     "response": '{"action": "reminder", "contact": "john", "time": "tomorrow 3pm", "task": "call"}'},
    {"instruction": "Parse: add milk to my grocery list",
     "response": '{"action": "add_item", "list": "grocery", "item": "milk"}'},
    {"instruction": "Parse: summarize my meeting notes from yesterday",
     "response": '{"action": "summarize", "source": "meeting_notes", "date": "yesterday"}'},
    # ... hundreds more examples
]
```

### Example 3: Convert LoRA to Ollama Model

After training, merge the LoRA adapter and convert to GGUF for Ollama:

```bash
# Merge LoRA adapter back into base model
python -c "
from peft import AutoPeftModelForCausalLM

model = AutoPeftModelForCausalLM.from_pretrained('./lora-llama-coding')
merged = model.merge_and_unload()
merged.save_pretrained('./merged-llama-coding')
"

# Convert to GGUF (requires llama.cpp)
cd ~/llama.cpp
python convert_hf_to_gguf.py ../merged-llama-coding --outtype q4_k_m

# Create Ollama model
cat > Modelfile <<EOF
FROM ./merged-llama-coding.gguf
SYSTEM "You are a TypeScript coding assistant specialized in Fastify, Zod, and Azure Cosmos DB."
EOF

ollama create my-coding-model -f Modelfile
ollama run my-coding-model
```

---

## What Models Can You Fine-Tune on 24 GB VRAM?

| Model         | Method      | VRAM Usage | Feasible?      | Training Time (1K examples) |
| ------------- | ----------- | ---------- | -------------- | --------------------------- |
| Llama 3.1 8B  | QLoRA 4-bit | ~8 GB      | ✅ Comfortable | ~1–2 hours                  |
| Qwen 2.5 7B   | QLoRA 4-bit | ~8 GB      | ✅ Comfortable | ~1–2 hours                  |
| Llama 3.1 8B  | LoRA FP16   | ~18 GB     | ✅ Fits        | ~2–3 hours                  |
| Mistral 7B    | QLoRA 4-bit | ~8 GB      | ✅ Comfortable | ~1–2 hours                  |
| Llama 3.1 13B | QLoRA 4-bit | ~14 GB     | ✅ Fits        | ~3–5 hours                  |
| CodeLlama 34B | QLoRA 4-bit | ~22 GB     | ⚠️ Tight       | ~6–10 hours                 |
| Llama 3.1 70B | QLoRA 4-bit | ~40 GB     | ❌ Too large   | Need multi-GPU              |

---

## Benefits

| Benefit              | Description                                                         |
| -------------------- | ------------------------------------------------------------------- |
| **Personalization**  | Model learns YOUR coding style, patterns, and conventions           |
| **Domain expertise** | Train on your project's specific API patterns, schemas, terminology |
| **Smaller + faster** | A fine-tuned 7B can outperform a generic 32B on your specific tasks |
| **Privacy**          | Your training data never leaves your machine                        |
| **Cost**             | $0 vs $100–1000s for cloud fine-tuning (OpenAI, Together AI)        |
| **Iteration speed**  | Train → test → adjust in hours, not days                            |
| **Portable output**  | Export to GGUF/Ollama — runs anywhere                               |

---

## Skills You'll Learn

| Skill                      | What You'll Learn                                        | Career Value                                  |
| -------------------------- | -------------------------------------------------------- | --------------------------------------------- |
| **LoRA / QLoRA**           | Parameter-efficient fine-tuning techniques               | Top ML engineering skill (2024–2026 standard) |
| **Hugging Face ecosystem** | transformers, datasets, peft, trl, accelerate            | Industry-standard ML tooling                  |
| **Training loops**         | Loss functions, learning rates, gradient accumulation    | Core ML fundamentals                          |
| **Data preparation**       | Curating, cleaning, formatting training datasets         | Critical for any ML project                   |
| **Quantization**           | 4-bit, 8-bit, FP16 — trade-offs and techniques           | Essential for deployment                      |
| **Model evaluation**       | Perplexity, human eval, A/B testing fine-tuned vs base   | ML product development                        |
| **VRAM management**        | Gradient checkpointing, mixed precision, batch sizing    | GPU optimization                              |
| **Model merging**          | Merging LoRA adapters, converting to GGUF                | ML deployment pipeline                        |
| **Experiment tracking**    | Weights & Biases, training curves, hyperparameter tuning | Professional ML workflow                      |

---

## Training Data Sources for Your Projects

| Source                 | Data Type              | Fine-Tune Goal           |
| ---------------------- | ---------------------- | ------------------------ |
| Your GitHub repos      | TypeScript/Python code | Coding style model       |
| LysnrAI voice commands | Command → JSON pairs   | Voice command parser     |
| MindLyst triage logs   | Input → categorization | Content triage model     |
| Your commit messages   | Diff → message pairs   | Commit message generator |
| Your code reviews      | Code → feedback pairs  | Code review assistant    |
| Slack/Teams messages   | Conversations          | Writing style model      |

### Extracting Training Data from Your Repos

```bash
# Extract all TypeScript functions as training examples
find ~/code/mygh/learning_ai_common_plat -name "*.ts" -exec grep -l "export" {} \; | head -20

# Extract commit messages paired with diffs
cd ~/code/mygh/learning_ai_common_plat
git log --oneline -100 --format="%H %s" | while read hash msg; do
    echo "=== $msg ==="
    git diff "$hash^" "$hash" --stat
    echo ""
done
```

---

## Next Steps

- [ ] Install training libraries in a dedicated venv
- [ ] Collect 100+ instruction/response pairs from your codebase
- [ ] Run a QLoRA fine-tune of Llama 3.1 8B on your coding examples
- [ ] Evaluate: compare fine-tuned model vs base model on 20 test prompts
- [ ] Convert to GGUF and serve through Ollama
- [ ] Fine-tune a voice command parser for LysnrAI
- [ ] Experiment with different LoRA ranks (8, 16, 32, 64) and measure quality
