Cheap Volta,
60 tokens a second.

What's in the box

The card is a PCIe Tesla V100 32GB (Volta, sm_70). One thing worth knowing straight away: it reports itself as Tesla PG500-216 in nvidia-smi and GV100GL [Tesla PG500-216] in lspci — not as "V100". Don't grep for the string "V100" in your scripts and expect a match.

It shared the box with an RTX 3080 10GB on a Gigabyte X470 AORUS ULTRA GAMING (BIOS F6, PCIe Gen 3) with a Ryzen 7 2700 and 48 GB of DDR4, running Pop!_OS 22.04. Two of those details matter more than they look: Pop!_OS boots with systemd-boot rather than GRUB, so every "add a kernel parameter" instruction you'll find online needs translating — and the CPU has 8 physical cores, which is exactly why -t 8 beat -t 16 when we tested thread counts.

The adapter used in this build:

COMeap NVIDIA Graphics Card Power Adapter Cable
8-pin  ->  dual PCIe 8-pin (6+2)

https://www.amazon.com/dp/B07M9X68DS

Order it at the same time as the card. It's easy to overlook, and without it the build simply stops until it turns up.

Why it takes two cables

The card pulls up to 250 W. A single PCIe 8-pin is rated for roughly 150 W continuous. That gap is the entire reason the adapter has two inputs — and why both of them need to come from separate PSU cables, not one cable with a pigtail on the end, and not a motherboard-sourced auxiliary feed.

Two things go wrong otherwise. The mild one is voltage sag under load, which makes the card throttle defensively and produces baffling benchmark results — you change the power limit and nothing moves, because the card was never actually reaching that limit. The serious one is that sustained high current through a connector not rated for it is how connectors melt.

This build initially ran the card from a single motherboard-sourced pigtail before that was corrected. If you see unexplained throttling, or performance that refuses to respond to power-limit changes, check your power path before you start tuning software.

Volta is not supported by the open-source NVIDIA kernel module — it lacks the GSP firmware support that generation needs. If you're on the open driver, the card simply will not appear. There is no configuration that fixes this; you need the proprietary driver.

sudo apt install nvidia-driver-580   # or current proprietary
sudo reboot

# then confirm both cards are visible
nvidia-smi --query-gpu=index,name,pci.bus_id,memory.total --format=csv

Symptom

llama-server starts and then does nothing. No error, no progress, no output at all, for minutes. Every instinct says it has deadlocked.

Cause

nvidia-smi orders devices by PCI bus ID. CUDA, by default, does not. In a mixed-GPU machine those two orderings can disagree — so CUDA_VISIBLE_DEVICES=0 may quietly hand you a different card than the one you were looking at in nvidia-smi.

If your binary was compiled for only one architecture — say sm_70 for the V100 — and it lands on the RTX 3080 instead, CUDA has to JIT-compile PTX for the unexpected architecture at runtime. That is a one-time, multi-minute, completely silent operation. It is indistinguishable from a hang unless you already know to look for it.

False leads, so you can skip them

We first suspected a regression in llama.cpp's HTTP layer and checked out an older tag. The hang persisted. We then suspected a compiler ABI mismatch and rebuilt the entire project with gcc-10. The hang persisted. Both were plausible, both were wrong, and both cost hours. Check device ordering first whenever a CUDA program appears to hang shortly after you add a second GPU to a machine.

CUDA 11.5's headers do not compile against gcc-11 — you'll get template errors deep inside std::function. Use gcc-10, and use it for the entire build, not just as the CUDA host compiler.

sudo apt install gcc-10 g++-10

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp

cmake -B build \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES="70;86" \
  -DCMAKE_C_COMPILER=/usr/bin/gcc-10 \
  -DCMAKE_CXX_COMPILER=/usr/bin/g++-10 \
  -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++-10 \
  -DCMAKE_BUILD_TYPE=Release

cmake --build build -j$(nproc)

Budget real time for this. A modern llama.cpp with flash-attention and all the template-instance kernels, compiled for two architectures, took 20–40+ minutes on a modest CPU. You can verify afterwards that both architectures actually made it into the binary:

cuobjdump --list-elf build/bin/libggml-cuda.so | grep -oE "sm_[0-9]+" | sort -u
# expect: sm_70  sm_86

Run a watchdog every single time until you have proven your cooling under a sustained, real workload — not a 30-second benchmark. Ours killed the server three separate times during testing, each time correctly.

#!/bin/bash
SAFE_MAX=80
while true; do
  T=$(nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader,nounits -i 0)
  if [ -n "$T" ] && [ "$T" -ge "$SAFE_MAX" ]; then
    echo "!!! V100 HIT ${T}C — KILLING SERVER !!!"
    pkill -9 -f "build/bin/llama-server"
  fi
  sleep 5
done

Four things we got wrong building that

A hardware backstop

Set your motherboard's own fan curve to ramp hard past 80°C on whatever slot-adjacent sensor it exposes (often labelled something like "PCIEX16 slot temp"). That's a safety net that works even if your software watchdog isn't running.

But calibrate it against reality first. That sensor reads the board area near the slot, not the GPU die, and it runs noticeably cooler under load. A curve that waits for the slot to hit 60°C may not engage until the die is far hotter than you intended. Compare it against nvidia-smi before trusting the trigger point.

The nuance worth carrying away: required fan speed depends enormously on workload type. The mining fan at 30% held a sparse MoE model at a flat 40°C indefinitely. That same fan at 15–23%, running a dense model with speculative decoding at full power, climbed to 80°C and tripped the watchdog — repeatedly. Sparse MoE inference leaves the GPU idle far more of the time than dense inference with a draft model does.

Software fan control on Gigabyte boards

The in-kernel it87 driver didn't recognise this board's Super I/O chip:

it87: Unsupported chip (DEVID=0x8686)

The fix is the out-of-tree fork, installed via DKMS so it survives kernel updates:

git clone https://github.com/frankcrawford/it87.git
cd it87
sudo ./dkms-install.sh

If it then reports No such device despite installing cleanly, ACPI is reserving the Super I/O's ports. You need a kernel parameter — and here's where Pop!_OS differs from most guides you'll find, because it uses systemd-boot rather than GRUB:

sudo kernelstub -a "acpi_enforce_resources=lax"
sudo reboot

ECC off

Tesla-class cards run ECC on their HBM2 by default. That is a genuine bandwidth and latency cost on every memory read, and for a homelab inference box the tradeoff rarely makes sense. Disabling it requires one reboot — and then it stays disabled permanently, because it lives in the GPU's non-volatile state rather than being reapplied at boot.

Power limit

Our card's range was 100–250W. Whether capping costs you anything is entirely workload-dependent, which was one of the more surprising results of the whole exercise: on sparse MoE inference, 150W and 250W were identical. On dense inference with a draft model, the cap cost 15–30%.

Clock lock

Find your card's actual maximum first rather than copying a number from a guide — ours topped out at 1380 MHz. Locking it stops the card hunting through DVFS states between requests. Small, but free.

# find your card's real max graphics clock
nvidia-smi -i 0 -q -d SUPPORTED_CLOCKS | grep Graphics | head -1

sudo nvidia-smi -i 0 -pm 1      # persistence mode
sudo nvidia-smi -i 0 -e 0       # ECC off — needs one reboot, then permanent
sudo nvidia-smi -i 0 -pl 250    # full TDP
sudo nvidia-smi -i 0 -lgc 1380  # your card's max, not necessarily 1380

Making it survive reboots

Everything except ECC resets on reboot. A systemd unit fixes that — and note it targets the GPU by UUID rather than index, so it keeps working if you ever move cards between slots:

# get your UUID first:
#   nvidia-smi -i 0 --query-gpu=uuid --format=csv,noheader

[Unit]
Description=Tesla V100 power limit + clock lock
After=multi-user.target nvidia-persistenced.service
Wants=nvidia-persistenced.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/nvidia-smi -i GPU-<your-uuid> -pm 1
ExecStart=/usr/bin/nvidia-smi -i GPU-<your-uuid> -pl 250
ExecStart=/usr/bin/nvidia-smi -i GPU-<your-uuid> -lgc 1380

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now nvidia-v100-tune.service

# after a reboot, all four should already be correct:
nvidia-smi --query-gpu=power.limit,clocks.gr,ecc.mode.current,persistence_mode --format=csv

DFlash2

DFlash2 landed in llama.cpp mainline on 27 Aug 2026 via PR #27342 — so there's no branch juggling required, just build current master. Confirm your binary has it before downloading gigabytes of weights:

./build/bin/llama-server --help | grep spec-type
# you want "draft-dflash" in the list of allowed values

CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0 \
./build/bin/llama-server \
  -m target.gguf -md dflash2-draft.gguf \
  -ngl 99 -ngld 99 \
  --spec-type draft-dflash --spec-draft-n-max 4 \
  -c 114688 -np 1 -fa on --jinja -b 512 -ub 256

E llama_init_from_model: failed to initialize the context: dflash requires
  ctx_other to be set (this warning is normal during memory fitting)
W operator(): failed to measure the memory of the extra model, fitting without it

MTP

Multi-token prediction uses a small sidecar model, usually published alongside the main weights as mtp-<model>-<quant>.gguf:

./build/bin/llama-server \
  -m target.gguf -md mtp-sidecar.gguf \
  -ngl 99 -ngld 99 \
  --spec-type draft-mtp \
  -c 114688 -np 1 -fa on --jinja

Modern llama.cpp has an automatic memory-fitting system (-fit on, the default) that computes n_gpu_layers, tensor split, and MoE overrides together, as one plan, across all your GPUs. It is good at this.

It also refuses to run at all if you have manually set any of -ncmoe, -ot, --tensor-split, or -ngl alongside them — and then falls back to naive placement that can dump everything onto device 0 and OOM, even when your combined VRAM was more than sufficient.

W common_fit_params: failed to fit params to free device memory:
    model_params::tensor_split already set by user, abort
E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 32277.02 MiB
    on device 0: cudaMalloc failed: out of memory

When you do need manual control: oversized MoE

For a Mixture-of-Experts model too big for VRAM, -ncmoe N keeps the expert weights of the first N layers in system RAM. Tune N empirically — a cudaMalloc failed on the compute buffer means back off a couple of layers, not a dozen.

For architectures with enormous n-gram or hash embedding tables (tens of GB on their own), pin that specific tensor to CPU and let it stay memory-mapped from SSD:

-ncmoe 46 -ot "ple_ngram_embd=CPU"
# --tensor-read-lazy defaults to auto: tensors >4GB are read lazily from disk

This will run models far above your nominal VRAM+RAM budget, at a real throughput cost. It's an experimentation tool, not a production configuration — and remember it's exactly the manual override that disables auto-fit, so you're tuning by hand from that point on.

Serving it to your other devices

Bind to your Tailscale interface IP rather than 0.0.0.0. The server is then reachable from your phone and laptop, and from nothing else — not your LAN, not the internet. No port forwarding, no firewall rules.

tailscale ip -4          # e.g. 100.x.y.z
./build/bin/llama-server ... --host 100.x.y.z --port 8080

There's no API key by default — fine inside a private tailnet, worth adding before you expose it any further.

Power & ECC state

All figures tok/s. The bottom row is the configuration in the recipe. Note how DFlash2 climbs steadily as constraints lift while MTP largely plateaus — that divergence is the whole story.

Micro-tuning · 250 W, ECC on, clocks locked

Reasoning penalty

Context ceiling · 32 GB card, 18.97 GB target + 1.14 GB draft

Loading successfully is not the same as running safely — the last two rows load fine but leave little room for compute buffers once generation starts allocating.