PYTORCH_DEVICE_MISMATCH warning RuntimeError: Expected all tensors on the same device
Mixing CPU and GPU tensors in a forward pass, often from partial CPU offload during quantized inference.
Symptom
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
Typically appears mid-inference, not at model load.
Cause
Some layers or tensors ended up on CPU while others are on GPU. Common
with partial GPU offload (e.g. llama.cpp-style --n-gpu-layers, or a
device_map="auto" split in transformers/accelerate that spills
onto CPU RAM when VRAM runs out) — the split itself isn’t the bug, but a
custom hook, input tensor, or cache object created without .to(device)
can land on the wrong device and trip this error.
Fix
Confirm every tensor you construct manually (inputs, attention masks, KV
cache buffers) is explicitly moved with .to(model.device) rather than
assuming default placement. If using accelerate’s device_map="auto",
check model.hf_device_map to see which layers actually landed on CPU —
if that’s unintentional, it usually means the model doesn’t fit in VRAM
at your current quantization and needs a lower bit-width or fewer
GPU-offloaded layers.
Frequently asked questions
Does this mean my model doesn't fit on my GPU?
Sometimes, but not always. If accelerate split layers onto CPU because VRAM ran out, check model.hf_device_map — if that split was unintentional, you need a lower quantization or fewer GPU-offloaded layers. But it can also come from a single manually-created tensor that wasn't moved with .to(device), with no VRAM shortage at all.
Why does this only appear mid-inference instead of at model load?
Partial-offload setups often load successfully because the initial forward pass exercises a code path where devices happen to line up. A custom input, cache object, or hook created later in the session is what actually trips the mismatch, which is why it surfaces mid-inference rather than immediately.