Documentation
Start here if you're setting up local inference for the first time, or jump to a specific topic using the sidebar. For guides on quantization, VRAM budgeting, and diagnosing common local inference errors, see the blog.
Getting started
Set up your first local inference environment and run a model end to end.
Quantization basics
What quantization is, the common formats (GGUF, AWQ, GPTQ), and how to pick one.
CPU Offloading & System RAM Utilization
Learn how to offload LLM layers to system RAM, calculate performance trade-offs, and optimize hybrid CPU-GPU inference speed.
VRAM Math: Sizing Hardware for Local LLMs
How LocalNodeOps estimates VRAM for local LLMs: weights, KV cache and 10% overhead, with a worked example for a 32B model on a 24GB GPU.