Understanding llm gpu vram requirements is the single most critical step when planning open-weight model deployment. Running out of video memory results in Out-Of-Memory (OOM) crashes or severe performance drops when execution spills over to CPU system RAM.
GPU Picks aggregates hardware specifications, provider pricing, and architectural memory footprints according to our strict editorial methodology. This guide outlines the mathematical formulas used to calculate VRAM capacity for Large Language Models (LLMs), breaks down quantization precision trade-offs, and maps modern open-weight architectures (such as DeepSeek V4/R1, Qwen 3.7, GLM-5.2, and Llama 4) to specific GPU cloud configurations.
Key takeaways
- Full FP16 precision requires approximately 2GB of VRAM per 1 billion parameters.[source] FP8 precision halves this to 1GB per billion,[source] while 4-bit quantization (INT4/GGUF/AWQ) reduces model weight footprints to ~0.55GB per billion parameters.[source]
- Memory sizing must account for the Key-Value (KV) cache, which grows linearly with context length and batch size.[source] A 128K context window can require 20GB to 80GB of additional VRAM beyond raw model weights.[source]
- Mixture of Experts (MoE) models like DeepSeek V3/R1 use active parameter routing per token (37B active out of 671B total), but the entire 671B weight set must remain resident in VRAM to maintain real-time throughput.[source]
- Single 24GB GPUs run quantized 32B models. Enterprise 70B models require dual 48GB or 80GB cards, while 671B MoE models require multi-node clusters with 8x H100 80GB or 4x H200 141GB GPUs.[source]
How to calculate LLM GPU VRAM requirements
To estimate the minimum VRAM required to load and serve an open-weight LLM for inference, use the three-part calculation:
- Model Weight Memory: Multiply model parameters by the bytes per parameter based on precision format.[source]
- KV Cache Overhead: Calculate memory allocated to maintain user conversation history across active context windows.[source]
- CUDA and Framework Overhead: Add a 20 percent buffer for PyTorch/CUDA runtime allocations, activation memory, and serving engine buffers.[source]
For long-context models supporting 128K to 1M tokens, KV cache usage often equals or exceeds the memory required for the model weights.[source] Serving engines like vLLM optimize this using PagedAttention to prevent memory fragmentation.[source]
3. Activation and CUDA runtime overhead
When serving inputs, activation tensors created between transformer layers require additional memory.[source] PyTorch and CUDA runtime context buffers allocate an additional 1GB to 3GB of VRAM upon initialization.[source] Adding a 20 percent buffer onto raw weight and cache totals prevents sudden OOM crashes under peak concurrency.[source]
To optimize deployment costs across different hardware setups, review our strategies for how to optimize GPU cloud costs.
2026 SOTA model VRAM requirements matrix
The table below outlines real-world VRAM requirements for prominent open-weight models across FP16, FP8, and INT4 precision levels.
| Model Architecture & Scale | Total Parameters | Precision Format | Base Weight Memory | Recommended VRAM (with 32K Context) | Minimum GPU Hardware Needed |
|---|---|---|---|---|---|
| DeepSeek R1 | 671B MoE (37B active) | FP8 (Native) | ~671 GB | ~780 GB | 8x H100 80GB or 4x H200 141GB |
| DeepSeek V3 | 671B MoE (37B active) | FP8 (Native) | ~671 GB | ~780 GB | 8x H100 80GB or 4x H200 141GB |
| DeepSeek-V4-Flash | 284B MoE (13B active) | FP8 / INT4 | ~160-284 GB | ~175-310 GB | 2x H200 141GB or 2x A100 80GB |
| Llama 3.1 | 405B Dense | FP16 | ~810 GB | ~920 GB | 8x H100 80GB or 8x A100 80GB |
| Qwen 2.5-72B | 72.7B Dense | FP16 | ~145 GB | ~165 GB | 2x A100 80GB or 1x H200 141GB |
| Qwen 2.5-72B | 72.7B Dense | INT4 / AWQ | ~39 GB | ~48 GB | 2x RTX 4090 24GB or 1x L40S 48GB |
| Qwen 2.5-Coder-32B | 32.5B Dense | FP16 | ~65 GB | ~75 GB | 1x A100 80GB or 1x H100 80GB |
| Qwen 2.5-Coder-32B | 32.5B Dense | INT4 / GGUF | ~20 GB | ~24 GB | 1x RTX 4090 24GB |
| GLM-4.5 / GLM-5.1 | 130B MoE | FP8 | ~130 GB | ~155 GB | 2x A100 80GB or 2x H100 80GB |
| Llama 3.3 / DeepSeek R1-Distill | 70B Dense | FP16 | ~140 GB | ~160 GB | 2x A100 80GB or 1x H200 141GB |
| Llama 3.3 / DeepSeek R1-Distill | 70B Dense | INT4 / GGUF | ~38 GB | ~45 GB | 2x RTX 4090 24GB or 1x L40S 48GB |
| Gemma 2 / GLM-4-9B | 9B-14B Dense | FP16 | ~18-28 GB | ~22-32 GB | 1x RTX 4090 24GB or 1x A10G 24GB |
Mixture of Experts (MoE) VRAM specifics: DeepSeek V4 & R1
A common point of confusion when sizing Mixture of Experts (MoE) models is the difference between total parameter count and active parameter count.[source]
For example, DeepSeek V3 and R1 contain 671 billion total parameters, but route only 37 billion active parameters per token through specialized expert sub-networks.[source]
MoE Memory Golden Rule
Active parameters dictate compute latency, but total parameters dictate VRAM footprint.[source] Because any expert can be selected for any token, all 671 billion weights must remain loaded in VRAM. Fetching un-allocated weights over PCIe buses during token generation slows inference significantly.
To serve DeepSeek V3/R1 in native FP8 precision, hosting environments require at least 700GB of VRAM.[source] Teams targeting enterprise inference workloads can read our roundup of the best GPU cloud for inference or evaluate multi-GPU setups on the best H200 GPU cloud.
- 8x NVIDIA H100 80GB: Provides 640GB of raw VRAM. Running in FP8 requires light 4-bit quantization or tensor parallelism across 8 cards to fit extended context windows.[source]
- 4x NVIDIA H200 141GB: Provides 564GB of HBM3e VRAM, allowing FP8 deployment across 4 GPUs with reduced inter-node communication.[source]
- AMD MI300X Clusters: Featuring 192GB HBM3 memory per card, a 4x MI300X node provides 768GB of VRAM, hosting the full FP8 model on a single node.[source]
Quantization trade-offs: FP16 vs FP8 vs INT4
Quantization compresses model weight precision from 16-bit floating point down to 8-bit or 4-bit representations.[source]
FP16 / BF16 (Full precision)
- Memory Footprint: 2.0GB per 1B parameters.[source]
- Accuracy Retention: 100 percent baseline accuracy.
- Best For: Enterprise production deployments, model fine-tuning, and task-critical reasoning where zero perplexity degradation is allowed.[source] Teams building training clusters can consult our guide on the best GPU cloud for ML training.
FP8 (8-bit floating point)
- Memory Footprint: 1.0GB per 1B parameters.[source]
- Accuracy Retention: Greater than 99.5 percent baseline accuracy. Supported natively by NVIDIA Hopper (H100/H200) and Ada Lovelace architectures.[source]
- Best For: Native precision serving for large MoE models like DeepSeek V3/R1.[source]
INT4 / AWQ / GGUF (4-bit quantization)
- Memory Footprint: Approximately 0.55GB per 1B parameters.[source]
- Accuracy Retention: 95 to 98 percent baseline accuracy depending on quantization scheme (AWQ, GPTQ, or GGUF K-quants).[source]
- Best For: Fitting 32B and 70B parameter models onto single or dual consumer GPUs on the best RTX 4090 cloud.[source]
Matching workloads to cloud GPU instances
Selecting the appropriate GPU cloud instance depends on model size, quantization precision, and expected concurrency.
Entry Tier (24GB to 48GB VRAM)
- Suitable Hardware: NVIDIA RTX 4090 (24GB), RTX 6000 Ada (48GB), L40S (48GB).[source]
- Compatible Models: Qwen 2.5 32B (INT4), DeepSeek-R1-Distill-32B (INT4), Gemma 4 27B (INT4), GLM-4-9B (FP16).
- Target Use Cases: Single-user development, local RAG pipelines, internal developer testing.
Mid-Range Tier (80GB to 141GB VRAM)
- Suitable Hardware: NVIDIA A100 80GB, H100 80GB, H200 141GB.[source][source]
- Compatible Models: Llama 3.3 70B (FP16), Qwen 2.5 72B (FP16), DeepSeek-R1-Distill-70B (FP16).
- Target Use Cases: Multi-user production APIs, high-throughput batch inference, long-context document processing.
Enterprise Cluster Tier (320GB to 1,500GB+ VRAM)
- Suitable Hardware: 8x H100 80GB, 4x H200 141GB, 4x AMD MI300X 192GB.[source]
- Compatible Models: DeepSeek V4 / R1 671B MoE, Llama 3.1 405B, GLM-5.2 130B MoE.
- Target Use Cases: Enterprise AI infrastructure, multi-tenant API serving, foundation model fine-tuning.
To compare current cloud provider hourly rates, region availability, and spot discounts across these GPU instances, explore our interactive GPU Cloud Lookup tool.
Frequently asked questions
How much VRAM is needed to run DeepSeek R1 locally?
Running the full 671B parameter DeepSeek R1 model in FP8 precision requires at least 700GB of VRAM, which mandates an enterprise multi-GPU server (such as 8x H100 80GB or 4x H200 141GB).[source][source] However, distilled versions of R1 (such as DeepSeek-R1-Distill-32B in 4-bit GGUF format) run locally on a single 24GB GPU like the RTX 4090.[source]
What happens when an LLM runs out of VRAM?
When VRAM capacity is exceeded, the execution engine either crashes with a CUDA Out-Of-Memory (OOM) error or offloads model layers into system RAM (CPU offloading).[source] CPU offloading avoids crashes but reduces generation speed by 90 to 98 percent due to PCIe bandwidth bottlenecks.
Does context length affect VRAM requirements?
Yes. Every token added to the prompt or generated response stores key-value attention pairs in the KV cache.[source] At context lengths of 64K to 128K tokens, the KV cache can consume more VRAM than the model weights themselves.
Is an RTX 4090 sufficient for 70B models?
To compare live hourly pricing and specs for GPUs capable of running these models, visit the GPU Cloud Lookup tool or read our guide on the Best GPU Cloud for Inference.