# Tesla V100 32GB on Pop!_OS — Complete Setup & Tuning Guide

A from-scratch, gotcha-by-gotcha guide to running a **PCIe Tesla V100 32GB** alongside a consumer GPU (RTX 3080 in this build) on Pop!_OS, serving LLMs via llama.cpp with MTP and DFlash2 speculative decoding. Everything here was learned the hard way on one rig — this document exists so the next rig goes faster.

## Table of Contents
1. [Hardware](#hardware)
2. [Power Delivery — Read This First](#power-delivery--read-this-first)
3. [Driver Installation](#driver-installation)
4. [The CUDA Device Order "Hang" — The #1 Time-Waster](#the-cuda-device-order-hang--the-1-time-waster)
5. [Build Toolchain](#build-toolchain)
6. [Thermal Safety](#thermal-safety)
7. [Fan Control](#fan-control)
8. [Power Limit, ECC, and Clock Locking](#power-limit-ecc-and-clock-locking)
9. [Serving Models: MTP and DFlash2](#serving-models-mtp-and-dflash2)
10. [The `--fit` Auto-Placement Trap](#the---fit-auto-placement-trap)
11. [MoE Models on Limited VRAM](#moe-models-on-limited-vram)
12. [Reasoning / Thinking Model Gotchas](#reasoning--thinking-model-gotchas)
13. [Remote Access via Tailscale](#remote-access-via-tailscale)
14. [Benchmark Results](#benchmark-results)
15. [Quick Reference Commands](#quick-reference-commands)

---

## Hardware

- **GPU 0**: **Tesla V100 32GB, PCIe** (Volta, sm_70). Worth knowing: it identifies itself as `Tesla PG500-216` in `nvidia-smi` and as `GV100GL [Tesla PG500-216]` in `lspci`, not as "V100" — so don't grep for the string "V100" in your scripts.
- **GPU 1**: NVIDIA RTX 3080 10GB (consumer card, sm_86)
- **CPU**: AMD Ryzen 7 2700 — 8 cores / 16 threads
- **Motherboard**: Gigabyte X470 AORUS ULTRA GAMING (-CF), BIOS F6 (2019-01-25), PCIe Gen 3
- **RAM**: 48GB DDR4 (mixed 8+16+8+16 kit, running at 2133 MT/s)
- **Storage**: 512GB Intel NVMe (models) + 8TB WD HDD
- **OS**: Pop!_OS 22.04 LTS, kernel 6.16.3 (systemd-boot, not GRUB — matters for kernel parameter changes)
- **Driver / CUDA**: nvidia-driver 580.173.02, CUDA 11.5.119

---

## Power Delivery — Read This First

**You will need to buy an adapter cable. Your PSU cannot power this card directly.**

The PCIe V100 has a single 8-pin power socket, and it is **not** a standard PCIe 8-pin — you cannot plug a normal PCIe lead into it. You need an aftermarket adapter that converts the card's 8-pin input into **two PCIe 8-pin connections**, which you then feed from two separate PSU cables.

The adapter used in this build:
> **COMeap NVIDIA Graphics Card Power Adapter Cable** — 8-pin to dual PCIe 8-pin (6+2)
> https://www.amazon.com/dp/B07M9X68DS

Order this at the same time as the card. It is easy to overlook and it will block your entire build if it doesn't arrive with everything else.

### Why two cables and not one

The card pulls up to **250W**. A single PCIe 8-pin is rated for roughly **150W** continuous. That gap is the whole reason the adapter has two inputs.

Feed the adapter from **two separate PSU cables**. Do not power it through:
- A single pigtail / Y-splitter hanging off one PSU cable
- A motherboard-sourced auxiliary power feed

Both can cause voltage sag under load, and — more seriously — sustained high current through a connector not rated for it is how connectors melt.

> In this build the card was initially run from a single motherboard-sourced pigtail before being corrected to dedicated PSU runs. If you see unexplained throttling, or performance that refuses to respond to power-limit changes, check your power path before you start tuning software.

---

## Driver Installation

Volta (V100) is **not supported by the open-source `nvidia` kernel driver** — it lacks GSP (GPU System Processor) firmware support for this generation. You must use the proprietary driver.

```bash
sudo apt install nvidia-driver-580   # or latest available proprietary driver
sudo reboot
```

After reboot, confirm both GPUs are visible:
```bash
nvidia-smi --query-gpu=index,name,pci.bus_id,memory.total --format=csv
```

---

## The CUDA Device Order "Hang" — The #1 Time-Waster

**Symptom:** `llama-server` (or any CUDA app) appears to hang for several minutes on startup with no error, no output, nothing — looks completely frozen.

**Root cause:** `nvidia-smi`'s device index order (by PCI bus ID) does not necessarily match CUDA's default device enumeration order. If you build a binary for one GPU's architecture (e.g., `sm_70` only, for the V100) and `CUDA_VISIBLE_DEVICES=0` or a hardcoded device index silently resolves to a *different* GPU (e.g., the RTX 3080), CUDA has to JIT-compile PTX for that unexpected architecture on the fly — a one-time process that can take several minutes and produces zero console output while it happens. It looks exactly like a hang.

**Fix — set this environment variable for every launch:**
```bash
export CUDA_DEVICE_ORDER=PCI_BUS_ID
```
This forces CUDA's device index order to match `nvidia-smi`'s (PCI bus order), so `CUDA_VISIBLE_DEVICES=N` reliably targets the GPU you think it does.

**Better yet**, build with both architectures baked in so a mismatch is never fatal, just slightly wasteful:
```bash
-DCMAKE_CUDA_ARCHITECTURES="70;86"   # 70 = V100 (Volta), 86 = RTX 3080 (Ampere)
```
A dual-arch build costs more compile time (flash-attention and mmf template instances roughly double) but means you never hit the JIT-recompile trap and can freely run either GPU with the same binary.

**Don't chase false leads first** (we did, so you don't have to): this is *not* a compiler ABI mismatch, *not* an llama.cpp version issue. Check `CUDA_DEVICE_ORDER` before anything else if a CUDA app "hangs" on first launch after adding a second GPU.

---

## Build Toolchain

CUDA 11.5's headers are incompatible with gcc-11's `std::function` implementation (template errors). Use gcc-10 for the **entire** build, not just as the CUDA host compiler:

```bash
sudo apt install gcc-10 g++-10

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES="70;86" \
  -DCMAKE_C_COMPILER=/usr/bin/gcc-10 \
  -DCMAKE_CXX_COMPILER=/usr/bin/g++-10 \
  -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++-10 \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)
```

A dual-arch build with a modern llama.cpp (flash-attention, DFlash2, MTP, all the template-instance kernels) takes a genuinely long time — expect 20-40+ minutes on a modest CPU. Budget for it.

---

## Thermal Safety

A passively-cooled datacenter GPU in a desktop chassis has **no fan of its own** — it depends entirely on case airflow forced through its heatsink. Do not run one unmonitored the first several times.

### The watchdog pattern that works
```bash
#!/bin/bash
SAFE_MAX=80
TICK=0
while true; do
  READ=$(nvidia-smi --query-gpu=index,name,temperature.gpu,power.draw,memory.used --format=csv,noheader,nounits)
  V100_TEMP=$(echo "$READ" | awk -F', ' '$1==0 {print $3}')
  if [ -n "$V100_TEMP" ] && [ "$V100_TEMP" -ge "$SAFE_MAX" ]; then
    echo "!!! V100 HIT ${V100_TEMP}C - KILLING SERVER !!!"
    pkill -9 -f "build/bin/llama-server"
  fi
  [ $((TICK % 12)) -eq 0 ] && echo "temps: $READ"
  TICK=$((TICK+1))
  sleep 5
done
```

**Lessons learned building this:**
- **Use an absolute cutoff only.** An early version tried a "rate of rise ≥8°C per tick" heuristic to catch runaway thermal events early — this produced a false-positive kill on a completely normal idle→load transition (52°C→61°C is just what happens when a GPU starts working). Stick to a hard ceiling.
- **The script must not `exit` after a kill if you plan to restart the server.** A single-shot watchdog that exits after firing once leaves you with zero protection on the next launch — this bit us twice in one afternoon during real testing. Loop forever; just keep re-checking.
- **80°C is a conservative software-imposed margin, not the V100's real limit** (the chip's actual throttle/shutdown thresholds are higher). Treat a kill at 80°C as "working as intended," not a crisis — the card is fine, just cool it down before the next run.
- **`pkill -f "llama-server"` can false-positive-match its own wrapper's command text** if you check `pgrep -af` output from inside the same tool/shell session that's constructing similarly-worded diagnostic commands. Use a more specific pattern like `pkill -f "build/bin/llama-server"` and verify with a clean, isolated `pgrep -af` call, not one chained after other commands.

### BIOS-level backstop
Set the motherboard's own fan-curve system to ramp aggressively (e.g., to 100%) past 80°C on whatever slot-adjacent temperature sensor is available (commonly labeled "PCIEX16 slot temp" or similar). This is a hardware-level safety net independent of the software watchdog — good belt-and-suspenders. **Important caveat**: this sensor reads the motherboard/slot area, not the GPU die itself, and typically reads noticeably cooler than the die under load. Don't trust a curve trigger point calibrated against this sensor without correlating it against real `nvidia-smi` die-temperature readings first.

---

## Fan Control

### What we tried and what actually worked
| Fan | Result |
|---|---|
| Static pressure case fan (quiet-tuned) | Insufficient even at 100% — moves too little air for a passive datacenter heatsink |
| Noctua (quiet, but lower CFM than a mining fan) | Comfortable on short bursts (~60°C), but **failed on sustained loads** — climbed steadily past 80°C within 3-4 minutes under real generation, tripping the watchdog |
| High-static-pressure ASIC mining fan | The only option that held comfortably under sustained heavy load. At 30% duty on a lighter (MoE/sparse) workload: rock-steady 35-41°C. At 15-23% duty under a heavier dense-model + speculative-decoding workload at full power: climbed to 78-80°C and tripped the watchdog multiple times — **fan speed needs matter a lot more for compute-heavy dense-model workloads than for sparse MoE workloads.** |

**Bottom line: use a genuine high-static-pressure fan (mining-rig style), not a "quiet" consumer fan, if you're running this at meaningful power/sustained load.** Expect to need real airflow, not just "some" airflow — a passively-cooled datacenter GPU has a dense fin stack that needs pressure to push air through, not just volume.

### Software fan control on this board (Gigabyte X470)
The mainline in-kernel `it87` driver didn't recognize this board's Super I/O chip:
```
it87: Unsupported chip (DEVID=0x8686)
```
Fix: build the out-of-tree fork with DKMS instead of the in-kernel driver:
```bash
git clone https://github.com/frankcrawford/it87.git
cd it87
sudo ./dkms-install.sh
```

If it then reports "No such device" even after DKMS install, ACPI is likely reserving the Super I/O's I/O ports. Add a boot parameter. **Pop!_OS uses systemd-boot, not GRUB**, so:
```bash
sudo kernelstub -a "acpi_enforce_resources=lax"
sudo reboot
```

**Known hard limitation on this board**: not every fan header is software-controllable this way. A dedicated "PUMP" header on this Gigabyte board was swept through every detected PWM channel with no response — it's wired through Gigabyte's separate EC/WMI path, not exposed to Linux `it87`. The `t-8ch/linux-gigabyte-wmi-driver` project exposes read-only temp sensors via WMI but not fan control. For headers like this, set a fixed speed in the BIOS instead of trying to control it from Linux.

---

## Power Limit, ECC, and Clock Locking

These three settings, done together, were worth a genuine **~30% additional throughput** on a demanding workload (dense model + DFlash2 speculative decoding) over the 150W default-ECC state, at a much smaller cost in heat than "just running at full power with no other changes" would suggest.

### 1. Power limit
```bash
sudo nvidia-smi -i 0 -pl 250   # this card's range was 100-250W; 250W = full TDP
```
**Important finding**: whether power-limiting costs you anything depends heavily on the workload. On a sparse MoE model that never pushed the card near its power ceiling anyway, 150W and 250W produced *identical* throughput — capping power was completely free. On a dense model with speculative decoding (MTP or DFlash2), which drives real sustained high utilization, cutting from 250W to 150W cost a genuine 15-30%+ of throughput. **Test your specific workload before assuming a power cap is "free."**

### 2. ECC (Tesla-specific)
Tesla-class GPUs run ECC on HBM2 by default — a real bandwidth/latency tax on every memory read. Disabling it is a legitimate, well-known Volta tuning trick:
```bash
sudo nvidia-smi -i 0 -e 0
sudo reboot   # required to take effect
```
Once disabled, **it stays disabled across future reboots** (it's stored in the GPU's non-volatile state) — you don't need to redo this every boot.

**Interesting asymmetry we measured**: ECC-off benefited MTP (a lighter, more memory-bandwidth-sensitive draft mechanism) far more than DFlash2 (a heavier, more compute-bound block-drafting mechanism) — roughly +12.5% vs +4.3% relative to their 150W/ECC-on baselines. If your workload is more compute-bound, don't expect ECC-off alone to be a silver bullet; you'll want the power headroom too.

### 3. Clock locking
```bash
# find the actual max supported graphics clock first:
nvidia-smi -i 0 -q -d SUPPORTED_CLOCKS | grep Graphics | head -1
sudo nvidia-smi -i 0 -lgc 1380   # replace with your card's actual max — 1380MHz on this one
```
Locks the clock near boost so the GPU doesn't spend time "hunting" through DVFS states between requests — small but real consistency win, no measured downside.

### Persisting all of this across reboots
None of the above (except ECC, which is non-volatile) survives a reboot on its own. Use a systemd service targeting the GPU by **UUID** (not index — index can shift if you change PCIe slots):

```bash
nvidia-smi -i 0 --query-gpu=uuid --format=csv,noheader   # get the UUID first
```

`/etc/systemd/system/nvidia-v100-powerlimit.service`:
```ini
[Unit]
Description=Set Tesla V100 power limit + clock lock
After=multi-user.target nvidia-persistenced.service
Wants=nvidia-persistenced.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/nvidia-smi -i GPU-<your-uuid-here> -pm 1
ExecStart=/usr/bin/nvidia-smi -i GPU-<your-uuid-here> -pl 250
ExecStart=/usr/bin/nvidia-smi -i GPU-<your-uuid-here> -lgc 1380

[Install]
WantedBy=multi-user.target
```
```bash
sudo systemctl daemon-reload
sudo systemctl enable --now nvidia-v100-powerlimit.service
```
Verified working: reboot the machine, then check `nvidia-smi --query-gpu=power.limit,clocks.gr,ecc.mode.current,persistence_mode --format=csv` — all four should already reflect your target state with zero manual steps.

---

## Serving Models: MTP and DFlash2

Both are speculative decoding methods in llama.cpp — a small "draft" model proposes several tokens ahead, and the full "target" model verifies them in one batched pass, which is cheaper than generating every token autoregressively when the draft is right often enough.

### MTP (Multi-Token Prediction, self-speculative)
```bash
CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0 \
./build/bin/llama-server \
  -m target-model.gguf \
  -md mtp-sidecar-model.gguf \
  -ngl 99 -ngld 99 \
  --spec-type draft-mtp \
  -c <context> -fa on --jinja -b 512 -ub 256 \
  --host 127.0.0.1 --port 8080
```
MTP sidecar GGUF files are usually a small file (~1.5-6GB depending on quant) published alongside the main model, often named `mtp-<model-name>-<quant>.gguf`.

### DFlash2 (block-diffusion drafting with candidate selection)
As of writing (Aug 2026), DFlash2 support merged into `ggml-org/llama.cpp` master via **PR #27342** — no need to build from a feature branch, just pull latest master. (If it's not yet merged when you read this, check for the PR or the `ikawrakow/ik_llama.cpp` fork, which sometimes carries ports of unmerged features.)

```bash
CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0 \
./build/bin/llama-server \
  -m target-model.gguf \
  -md dflash2-draft-model.gguf \
  -ngl 99 -ngld 99 \
  --spec-type draft-dflash --spec-draft-n-max 4 \
  -c <context> -fa on --jinja -b 512 -ub 256 \
  --host 127.0.0.1 --port 8080
```
Check `./build/bin/llama-server --help | grep spec-type` to confirm your build actually has `draft-dflash` listed before assuming it's available.

**Benign warnings you'll see on every DFlash2 load — do not panic:**
```
E llama_init_from_model: failed to initialize the context: dflash requires ctx_other to be set (this warning is normal during memory fitting)
W operator(): failed to measure the memory of the extra model, fitting without it...
```
These are part of DFlash2's internal memory-fitting probe and appear even on a completely successful load. Wait for `model loaded` / `listening on` before concluding anything's actually wrong.

**Which one is faster depends heavily on the workload — test both, don't assume:**
- MTP tended to win on general prose/reasoning-style prompts
- DFlash2 tended to win on code generation, and pulled further ahead as power/ECC constraints were lifted (it's the more power-hungry of the two)
- At full power with ECC off, DFlash2 won across the board on this hardware/model — see the [Benchmark Results](#benchmark-results) section

---

## The `--fit` Auto-Placement Trap

Modern llama.cpp has an automatic device-memory-fitting system (`-fit on`, the default) that reasons about your GPU(s)' free memory and automatically computes `n_gpu_layers`, `--tensor-split`, and MoE CPU/GPU tensor overrides *together, as one holistic plan* — including across multiple GPUs.

**The trap**: this system explicitly refuses to run if you manually set **any** of the following, and silently falls back to a naive single-device placement that can blow past a single GPU's VRAM even when your total across multiple GPUs would have been plenty:
- `-ncmoe` / `--n-cpu-moe`
- `-ot` / `--override-tensor`
- `--tensor-split`
- `-ngl` (when combined with the above)

If you manually set any of these and see this in your logs:
```
common_fit_params: failed to fit params to free device memory: ... already set by user, abort
```
...followed by an OOM on device 0 alone, **that's why**. The fix is almost always to simply remove your manual override and let `-fit` (on by default) handle everything itself — it's specifically designed to reason about multi-GPU placement, and doing it manually with these older-style flags actively defeats that.

Related flags for tuning the auto-fit behavior itself (rather than fighting it):
```
-fit on|off          # default on
-fitt MiB,MiB,...    # target free-memory margin per device (default 1024)
-fitc N              # minimum context size fit will settle for if squeezed (default 4096)
```

---

## MoE Models on Limited VRAM

For a Mixture-of-Experts model too large to fit fully in VRAM, `-ncmoe N` keeps the MoE expert weights of the first N layers on CPU RAM while everything else stays on GPU — this is the standard way to run an oversized MoE model on modest VRAM. Tune N empirically: too high and you waste GPU capacity, too low and you OOM. Watch the error carefully — a `cudaMalloc failed: out of memory` on the compute-buffer allocation step means back off by a few layers, not necessarily a huge jump.

For an *extremely* oversized embedding table specifically (some newer architectures use huge n-gram/hash-based embedding tables that can be tens of GB on their own), keep it CPU/SSD-resident explicitly rather than trying to fit it in VRAM:
```
-ot "ple_ngram_embd=CPU"   # or whatever the specific tensor name pattern is for your architecture
```
combined with the default `--tensor-read-lazy auto` behavior (lazy-reads large tensors from disk rather than fully loading into RAM). This lets you technically run models well above your documented VRAM+RAM minimums, at a real throughput cost — useful for experimentation, not for anything latency-sensitive. **Note**: this technique conflicts with `--fit` per the section above — you're deliberately overriding placement here, so `--fit` won't help you, and you'll need to tune `-ncmoe` manually alongside it.

---

## Reasoning / Thinking Model Gotchas

- **`--reasoning-format` defaults to `auto`**, which for many chat templates resolves to `none` — meaning `<think>` tags stay raw and unparsed inside the regular `content` field rather than being split into a separate `reasoning_content` field for UI display. If your reasoning isn't showing up as a distinct "thinking" section in a chat UI, explicitly pass `--reasoning-format deepseek`.
- **`--reasoning-effort` accepted values are chat-template-specific.** Don't assume `low`/`medium`/`high` are universal — one model in this session only accepted `xhigh` (its default/highest), `medium`, and `low`; passing `high` threw a Jinja template exception. Check the model's actual chat template or just try it and read the error.
- **A per-request `reasoning_budget` JSON field is not guaranteed to be honored**, especially on very new/experimental architecture branches. We hit a case where `reasoning_budget: 0`, a numeric cap, and `"high"` were all silently ignored on one branch, burning the entire token budget on unbounded reasoning and returning empty final content every time. The fix that actually worked was the **server-launch-level** `--reasoning off` flag (a different code path — it changes chat-template application, not a runtime request parameter). If a request-level reasoning control seems to do nothing, try the server-level flag instead.
- **Reasoning content is measurably slower to speculatively decode than final answer content — for both MTP and DFlash2, roughly equally.** We measured a 45-55% throughput drop during active "thinking" versus final-answer generation on the same model/hardware (e.g., ~55 tok/s non-reasoning down to ~25-30 tok/s while reasoning). This isn't a flaw in one method vs the other — it's because a draft model's value depends on correctly *predicting* what the target will say, and reasoning traces are inherently more exploratory/self-correcting ("wait, let me reconsider...") than polished output, so draft acceptance rates drop for any speculative method. If you're benchmarking "speed," always specify whether you mean with or without reasoning enabled — they're not comparable numbers.
- **Small `max_tokens` + a heavy reasoning budget can silently produce empty output.** If `finish_reason: "length"` comes back with an empty `content` field, check `reasoning_content` — it's very likely the entire budget went to thinking with none left for the actual answer. Fix: raise `max_tokens` substantially (some models want 15,000-20,000+ for genuinely complete responses on complex prompts), or cap/disable reasoning for that specific request if you don't need it.
- **The generation's context ceiling (`-c` at server launch) always wins over a larger `max_tokens` in the request.** If `total_tokens` in the response `usage` field exactly equals your launch-time `-c` value, that's the real cause of a truncated/empty response — not the `max_tokens` you requested.

---

## Remote Access via Tailscale

To serve the web UI to other devices (phone, laptop) without exposing it to the LAN or public internet:
```bash
tailscale ip -4   # get this machine's tailnet IP
./build/bin/llama-server ... --host <tailnet-ip> --port 8080   # NOT 0.0.0.0
```
Binding specifically to the Tailscale interface IP (rather than `0.0.0.0`) means the server is only reachable from devices actually in your tailnet — not your home LAN, not the internet. No port forwarding, no extra firewall rules needed if your local firewall (e.g., `ufw`) is inactive or already permits the tailnet interface. There's no API key by default (`llama-server` logs a CORS warning about this) — fine within a private tailnet, worth adding if you ever plan to expose more broadly.

---

## Benchmark Results

All results below: Qwen3.8-27B dense model, Q4_K_M target quant, on the V100 alone (RTX 3080 idle), same two test prompts (a short general/coding prompt, and a short pure-code prompt), non-reasoning unless noted.

### Power / ECC state comparison (same model, same prompts)
| Config | MTP general/code | DFlash2 general/code |
|---|---|---|
| 150W, ECC on | 48.3 / — tok/s | 40.5 / — tok/s |
| 180W, ECC off, clocks locked | 54.3 / 52.8 tok/s | 42.3 / 54.5 tok/s |
| 250W, ECC on | 56.2 / 55.1 tok/s | 52.2 / 58.1 tok/s |
| **250W, ECC off, clocks locked** | 52.8 / 57.5 tok/s | **60.3 / 60.8 tok/s** |

**Takeaway**: at full power with ECC off, DFlash2 became the clear winner on this hardware — a genuine reversal from every power-capped comparison, where MTP had generally led. If you're only going to tune one thing, tune power+ECC before picking a method.

### Spec-decode depth / batch-size micro-tuning (250W, ECC on, clocks locked)
| Knob | Best observed |
|---|---|
| MTP `-b 2048 -ub 512` | 58.8 tok/s (general) — cheap, easy win |
| DFlash2 + `unsloth`'s `UD-Q4_K_XL` target quant (vs plain `Q4_K_M`) | 61.6 tok/s (code) — best single result of the whole sweep |
| `--spec-draft-n-max` sweeps for either method | No clean monotonic trend — treat single-run results here as noisy; repeat trials before trusting |

### Reasoning-on penalty (both methods, same hardware)
| | Non-reasoning peak | During active reasoning |
|---|---|---|
| MTP | 48-58 tok/s | 27-35 tok/s |
| DFlash2 | 50-60 tok/s | 24-32 tok/s |

### Context ceiling (DFlash2, target 18.97GB + draft 1.14GB weights, 32GB card)
| Context | VRAM used | Fits? |
|---|---|---|
| 8,192 | ~23GB | yes, comfortable |
| 65,536 | 26.6GB | yes, comfortable |
| 114,688 | 29.7GB | yes, ~2.3GB margin |
| 131,072 | 30.7GB | yes, ~1.3GB margin |
| 147,456 | 31.76GB | yes, but only ~1GB margin — **too tight for production**, risk of generation-time OOM |

KV cache scaling was remarkably efficient on this model (looks like a low-KV-head GQA architecture) — going from 8K to 128K context only cost ~7.5GB. Don't assume this is universal; test your specific model.

---

## Quick Reference Commands

```bash
# --- One-time setup ---
sudo apt install nvidia-driver-580 gcc-10 g++-10
export CUDA_DEVICE_ORDER=PCI_BUS_ID   # put this in your shell profile

# --- Build (dual arch, V100 + Ampere consumer GPU) ---
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="70;86" \
  -DCMAKE_C_COMPILER=/usr/bin/gcc-10 -DCMAKE_CXX_COMPILER=/usr/bin/g++-10 \
  -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++-10 -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)

# --- Best-known production launch (DFlash2, full power/ECC-off tuned) ---
CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0 \
./build/bin/llama-server \
  -m target-model-Q4_K_M.gguf \
  -md dflash2-draft-model.gguf \
  -ngl 99 -ngld 99 \
  --spec-type draft-dflash --spec-draft-n-max 4 \
  -c 114688 -np 1 -fa on --jinja -b 512 -ub 256 \
  --reasoning on --reasoning-format deepseek --reasoning-effort low \
  --host <tailnet-ip> --port 8080

# --- Thermal watchdog (run alongside, always) ---
while true; do
  T=$(nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader,nounits -i 0)
  [ "$T" -ge 80 ] && pkill -9 -f "build/bin/llama-server" && echo "KILLED at ${T}C"
  sleep 5
done
```

---

*Compiled from an extended hands-on session getting a V100+RTX3080 rig fully tuned on Pop!_OS — every gotcha in here was hit for real, not theorized. If something here doesn't match your exact hardware/model, treat the methodology (test both configs, watch temps, verify claims before trusting them) as the real takeaway.*