How Cloud GPU & VRAM Sizing Math Works

Deploying machine learning models in the cloud requires calculating memory across three distinct components: Model Weights, Dynamic KV Cache, and Runtime CUDA & Activation Overhead.

Total VRAM (GB)=Vweights+VKV+Voverhead\text{Total VRAM (GB)} = V_{\text{weights}} + V_{\text{KV}} + V_{\text{overhead}}

Where VoverheadV_{\text{overhead}} accounts for PyTorch CUDA context initialization (1.2 GB baseline) plus intermediate layer activations (a 15% to 20% safety margin to prevent Out-Of-Memory / OOM crashes).

1. Parameter Footprint & Model Weights

Static Memory

Model weights scale directly with precision bits per parameter. Once loaded into GPU memory, weight tensors occupy a constant baseline memory footprint:

FP16 / BF16 (16-bit) 2.0 bytes/param (2.0 GB / B params)
FP8 / INT8 (8-bit) 1.0 byte/param (1.0 GB / B params)
INT4 (AWQ/GPTQ) 0.5 bytes/param (0.5 GB / B params)
Vweights=Params (Billions)×109×Bytes per Parameter10243V_{\text{weights}} = \frac{\text{Params (Billions)} \times 10^9 \times \text{Bytes per Parameter}}{1024^3}

Example: A 70B parameter model in standard 16-bit FP16 precision consumes (70.6 × 109 × 2) / (10243) = 131.5 GB in raw weight tensors.

2. Key-Value (KV) Cache Scaling

Dynamic Memory

During autoregressive generation, Key and Value attention tensors accumulate across all transformer layers for every active token in context. Long context lengths (32k to 128k) and concurrent batch streams often demand 20 GB to 60 GB of additional VRAM beyond raw model weights:

KV Cache (GB)=2×nlayers×nkv_heads×dhead×Lctx×B×bkv10243\text{KV Cache (GB)} = \frac{2 \times n_{\text{layers}} \times n_{\text{kv\_heads}} \times d_{\text{head}} \times L_{\text{ctx}} \times B \times b_{\text{kv}}}{1024^3}

Where nlayersn_{\text{layers}} is the number of transformer layers, nkv_headsn_{\text{kv\_heads}} is the number of KV attention heads (Grouped-Query Attention), dheadd_{\text{head}} is the head dimension (typically 128), LctxL_{\text{ctx}} is the context length in tokens, BB is concurrent batch streams, and bkvb_{\text{kv}} is bytes per element (2 bytes for FP16, 1 byte for FP8 KV cache).

3. Activation & CUDA Overhead

Runtime Buffer

PyTorch memory pooling, CUDA kernel workspace, and attention activation matrices require a 15% to 20% safety buffer over static weights and KV cache to prevent Out-Of-Memory (OOM) termination under peak concurrency:

Voverhead=(Base CUDA Context 1.2 GB)+0.15×(Vweights+VKV)V_{\text{overhead}} = (\text{Base CUDA Context } 1.2\text{ GB}) + 0.15 \times (V_{\text{weights}} + V_{\text{KV}})

Serving engines (such as vLLM, SGLang, and TensorRT-LLM) manage this buffer via PagedAttention to eliminate memory fragmentation and maintain maximum GPU throughput.

4. Fine-Tuning & Training Memory (Full vs LoRA vs QLoRA)

Training Workloads

Training requires substantially more memory than inference due to optimizer states, gradients, and forward-pass activation checkpoints:

MemoryFull≈Weights (2B)+Gradients (2B)+Optimizer States (12B)+Activations\text{Memory}_{\text{Full}} \approx \text{Weights (2B)} + \text{Gradients (2B)} + \text{Optimizer States (12B)} + \text{Activations}

Full fine-tuning requires 16 to 18 GB of VRAM per billion parameters because 32-bit AdamW maintains FP32 master weights, first momentum, and second variance tensors. Parameter-efficient fine-tuning (PEFT) like LoRA freezes base weights in 16-bit, while QLoRA compresses base weights to 4-bit NormalFloat (0.5 GB/B) with paged optimizers, allowing 70B fine-tuning on a single 80 GB GPU.

Frequently Asked Questions

How accurate are the cloud GPU pricing estimates?
All prices are synchronized directly from verified public cloud provider rate cards (RunPod, Lambda Labs, Vast.ai, FluidStack, Hyperstack, Nebius, Crusoe Cloud) and re-verified on a strict 30-day cadence. Prices reflect standard compute without custom negotiated enterprise discounts.
Should I use Spot or On-Demand instances for model serving?
For critical production serving with strict latency SLAs, choose On-Demand instances to prevent unexpected node terminations. For batch training, offline evaluation, or dev experimentation, Spot instances offer 40% to 70% cost savings.
Why does fine-tuning require so much more VRAM than inference?
Inference only stores model weights and the active KV cache. Full fine-tuning requires holding model weights (2 bytes), gradients (2 bytes), and AdamW 32-bit optimizer states (12 bytes), reaching 16 to 20 GB of VRAM per billion parameters. QLoRA avoids this by freezing base weights in 4-bit precision and only training low-rank adapter matrices.
Can I run a 70B model on a single GPU?
In standard FP16 precision, a 70B model requires approximately 160 GB VRAM, necessitating at least 2x A100/H100 80GB GPUs. However, using 4-bit quantization (AWQ or GPTQ), a 70B model shrinks to approximately 40 GB VRAM and can comfortably run on a single 1x A100 80GB or 2x RTX 4090 24GB GPUs.
How does Key-Value (KV) cache scaling affect VRAM during long-context generation?
During autoregressive token generation, Key and Value tensors accumulate across all transformer layers for every active token. At 128k context lengths or high batch concurrency, the KV cache pool can exceed 30 GB to 60 GB, often matching or surpassing the memory footprint of the model weights.
What is the difference between VRAM sizing for Dense models vs Mixture of Experts (MoE)?
While MoE architectures (such as DeepSeek V3/R1 with 37B active parameters out of 671B total) only activate a subset of weights per token for compute, the entire 671B parameter weight set must remain resident in VRAM to achieve real-time throughput, requiring multi-node clusters with at least 8x H100 80GB or 4x H200 141GB GPUs.

Need in-depth architectural comparisons?

Check out our specific hardware guides: LLM VRAM Sizing Guide, Fine-Tuning Hardware Requirements, Diffusion & Image Gen Sizing, Qwen Hardware Sizing, DeepSeek Hardware Guide, Cost Optimization Guide, or browse the interactive GPU Cloud Lookup Index.