K2-Horizon-MoVA-36B GPU Requirements: VRAM & Cluster Sizing

Understanding the k2-horizon-mova-36b gpu requirements is essential for machine learning teams deploying the Institute of Foundation Models (IFM) flagship open-weights architecture.^[source] Released under the Apache 2.0 license, K2-Horizon-MoVA-36B-A4B introduces a compound sparse design that breaks conventional hardware sizing rules: the model stores 36 billion total parameters in high-bandwidth memory, yet activates only 4 billion parameters per token during forward passes.^[source]

This decoupled architecture produces a distinct operational trade-off. Token generation latency is as fast as a sub-7B parameter model, but static weights demand 72 GB of VRAM in unquantized 16-bit precision.^[source] Furthermore, because the model features a native 524,288-token (512K) context window, memory demands at long context lengths are governed by Key-Value (KV) cache allocation rather than parameter weights.^[source]

This guide provides concrete mathematical sizing formulas, verified VRAM requirements across precision tiers, multi-GPU topology recommendations, and validated deployment manifests for production serving.

The engineering dilemma: compound sparsity vs memory residency

Most sparse Mixture-of-Experts (MoE) architectures, including Mixtral and DeepSeek, confine sparsity strictly to the Feed-Forward Network (FFN) layers, keeping the self-attention projections dense. K2-Horizon-MoVA-36B changes this paradigm by introducing Mixture-of-Values Attention (MoVA) directly into the multi-head attention blocks.^[source]

The architectural layout comprises:

1. Feed-forward sparsity: 100 fine-grained FFN experts, routing 8 experts per token with 1 shared expert always active (num_experts: 100, num_experts_per_tok: 8, num_shared_experts: 1).^[source]
2. Attention value sparsity (MoVA): 64 value projection experts, routing 4 experts per token via a softplus gating mechanism (mova_num_experts: 64, mova_num_experts_per_tok: 4).^[source]
3. Layer depth and geometry: 48 transformer layers (with layers 0, 1, and 2 designated as dense MLP-only), a hidden dimension of 2,560, head dimension of 128, 32 query attention heads, and 8 key-value heads under Grouped Query Attention (GQA 4:1).^[source]

┌────────────────────────────────────────────────────────────────────────┐
│               K2-Horizon-MoVA-36B Forward Pass Topology                │
├────────────────────────────────────────────────────────────────────────┤
│                                                                        │
│   Input Token Vector (Dim: 2,560)                                      │
│         │                                                              │
│         ├──> Attention Layer (Layers 3-47)                             │
│         │     ├── Query & Key Projections (Dense GQA, 32/8 heads)      │
│         │     └── MoVA Value Projection: 64 Experts ──> Top-4 Routed  │
│         │                                                              │
│         └──> FFN Layer (Layers 0-47)                                   │
│               ├── 1 Shared Expert (Always Active)                      │
│               └── 100 Routed Experts ─────────────────> Top-8 Routed  │
│                                                                        │
│   Total Resident Parameters: 36.0B  │  Active Parameters / Token: 4.0B  │
└────────────────────────────────────────────────────────────────────────┘

Because all 36 billion parameters must remain resident in VRAM to service arbitrary token routings, hosting the model requires high memory capacity.^[source] However, because each token only executes matrix multiplications across 4 billion parameters, inference engines do not need the massive compute capability required by standard 70B dense models.^[source]

Calculating K2-Horizon-MoVA-36B GPU requirements: memory model

To calculate the exact GPU memory required for running K2-Horizon-MoVA-36B, engineers must separate static model weights, dynamic KV cache allocations, activation scratchpads, and CUDA runtime overhead:

1. Static model weight calculation

The memory required to hold the static weights in VRAM depends on the storage precision:

Where $P_{\text{total}} = 36.0 \times 10^9$ and $B_{\text{weight}}$ represents bytes per parameter:

• 16-bit (BF16 / FP16): $B_{\text{weight}} = 2.0 \implies M_{\text{weights}} = 72.0 \text{ GB}$^[source]
• 8-bit (FP8 / INT8): $B_{\text{weight}} = 1.0 \implies M_{\text{weights}} = 36.0 \text{ GB}$^[source]
• 4-bit (AWQ / GPTQ / GGUF Q4_K_M): $B_{\text{weight}} \approx 0.55 \implies M_{\text{weights}} = 19.8 \text{ GB}$ (including scale and zero-point metadata)^[source]

2. GQA KV cache calculation across context lengths

K2-Horizon uses Grouped Query Attention with 48 layers (45 attention layers) and 8 key-value heads with a head dimension of 128.^[source] The formula for KV cache memory as a function of sequence length $C$ and KV precision bytes $B_{\text{kv}}$ is:

Evaluating the constants across the 48 transformer layers yielding 98,304 elements per sequence position:^[source]

• In FP16 KV precision ($B_{\text{kv}} = 2$), the cache consumes approximately 196.6 KB per token ($0.1966 \text{ GB}$ per 1,000 tokens).^[source]
• In FP8 KV precision ($B_{\text{kv}} = 1$), the cache consumes approximately 98.3 KB per token ($0.0983 \text{ GB}$ per 1,000 tokens).^[source]

Notice that at 512K context, the FP16 KV cache alone consumes 103.1 GB, which is substantially larger than the base model weights of 72.0 GB.^[source] For long-context workloads, memory capacity planning is dominated by KV cache management rather than model size.

Hardware sizing matrix by GPU class

The following matrix compares common accelerator platforms for running K2-Horizon-MoVA-36B across different precision levels and context limits:

To check current hourly pricing across these cloud GPU classes on verified providers, consult our interactive GPU cloud price lookup tool.

Deep dive: choosing between single-GPU and multi-GPU setups

Selecting the optimal hardware configuration depends on whether your workload prioritizes local desktop experimentation, high-throughput microservices, or long-context reasoning.

1. The desktop developer tier: RTX 4090 (24GB) and RTX 5090 (32GB)

Developers looking to run K2-Horizon-MoVA-36B locally can leverage 4-bit quantization (AWQ or GGUF Q4_K_M).^[source]

• On a 24GB RTX 4090, 4-bit weights consume roughly 19.8 GB. After allocating 1.5 GB for CUDA context and activation memory, approximately 2.7 GB remains for the KV cache. This accommodates sequence lengths up to 16K tokens with FP8 KV cache compression.
• On a 32GB RTX 5090, the additional 8 GB of VRAM provides substantial breathing room. INT4 weights leave roughly 10.7 GB for KV caching, allowing the RTX 5090 to sustain 65K tokens in FP8 KV cache without offloading to host system RAM. Combined with 1,792 GB/s of GDDR7 bandwidth, the RTX 5090 delivers responsive interactive token speeds for local coding and agentic tasks.

For teams comparing consumer hardware against low-cost instances, our evaluation of the best GPU cloud for inference breaks down preemption and storage trade-offs.

2. The enterprise single-GPU tier: NVIDIA H100 (80GB) and H200 (141GB)

For production API endpoints serving short to medium context lengths (up to 128K tokens), a single 80GB enterprise accelerator provides an efficient deployment footprint.^[source]

Running the official FP8 checkpoint on an H100 80GB SXM consumes 36.0 GB for weights. An FP8 KV cache of 128K tokens requires 12.9 GB, leaving over 25 GB of unallocated VRAM for batch concurrency and CUDA graphs.^[source] Because the H100 provides native 8-bit floating-point Tensor Cores and 3,350 GB/s of HBM3 bandwidth, decoding 4B active parameters achieves exceptional tokens-per-second throughput.^[source]

When workloads demand unquantized BF16 weights or longer sequences, the NVIDIA H200 141GB offers enough headroom to hold 72.0 GB of BF16 weights alongside 50 GB of KV cache on a single device, eliminating the need for inter-GPU communication across sequence lengths up to 256K tokens.

3. The hyperscale single-node alternative: AMD Instinct MI325X (256GB)

The primary bottleneck of serving K2-Horizon at its full 512K context window is that dual-GPU 80GB clusters run out of headroom when using unquantized FP16 KV cache.

The AMD Instinct MI325X features 256 GB of HBM3e memory and 6.0 TB/s of peak bandwidth.^[source] This allows a single MI325X accelerator to host:

• 72.0 GB of base BF16 weights
• 103.1 GB of unquantized FP16 KV cache (full 512K tokens)
• 15.0 GB of activation and runtime buffers

Total VRAM consumption is approximately 190.1 GB, which fits within the 256 GB boundary of a single accelerator with room to spare.^[source] Deploying on a single high-capacity accelerator avoids the cross-GPU communication overhead that typically plagues sparse MoE models. For a detailed total cost analysis of large memory nodes, see our guide comparing AMD MI325X vs NVIDIA H200 cloud pricing.

Interconnect requirements and expert parallelism

Because K2-Horizon features sparsity in both attention (MoVA) and feed-forward (MoE) layers, multi-GPU partitioning requires careful orchestration.^[source]

When splitting K2-Horizon across multiple GPUs, inference engines such as vLLM and SGLang apply a combination of Tensor Parallelism (TP) and Expert Parallelism (EP):^[source]

• Tensor Parallelism (TP=2): Splits the dense query and key projections across 2 GPUs.
• Expert Parallelism (EP=2): Distributes the 100 FFN experts and 64 MoVA value experts across the devices, routing token vectors dynamically during inference.^[source]

┌────────────────────────────────────────────────────────────────────────┐
│             Interconnect Bandwidth vs MoVA Pipeline Latency            │
├────────────────────────────────────────────────────────────────────────┤
│                                                                        │
│   PCIe Gen4 x16 (31.5 GB/s bidirectional):                             │
│   [Severe Bottleneck] High all-to-all expert dispatch bubbles.         │
│   Token generation latency degrades by up to 3.2x.                     │
│                                                                        │
│   PCIe Gen5 x16 (63.0 GB/s bidirectional):                             │
│   [Marginal] Acceptable for small batch sizes (BS <= 4).                │
│   Exhibits queue contention under concurrent requests.                 │
│                                                                        │
│   NVLink 4 / NVLink 5 (900-1,800 GB/s bidirectional):                  │
│   [Optimal] Near-zero all-to-all dispatch overhead.                    │
│   Allows overlapping communication with active 4B FFN compute.         │
│                                                                        │
└────────────────────────────────────────────────────────────────────────┘

Deploying multi-GPU K2-Horizon instances on cloud servers without NVLink (such as commodity PCIe-only multi-GPU instances) introduces severe pipeline stalls during expert routing. For multi-GPU topologies, always prioritize SXM5 or HGX platforms featuring NVLink interconnects. A deeper comparison of engine communication strategies is available in our vLLM vs SGLang vs TensorRT-LLM analysis.

Production deployment manifests

Below are verified launch configurations for production deployment using official recipes from the vLLM and SGLang projects.

1. vLLM multi-GPU production recipe

The official vLLM recipe for K2-Horizon configures Tensor Parallelism, Expert Parallelism, and native tool-calling parsers:^[source]

vllm serve IFM/K2-Horizon-MoVA-36B-A4B \
  --revision main \
  --tensor-parallel-size 2 \
  --enable-expert-parallel \
  --trust-remote-code \
  --dtype bfloat16 \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.92 \
  --reasoning-parser k2_horizon \
  --tool-call-parser k2_horizon \
  --enable-auto-tool-choice

Key configuration flags:

• --tensor-parallel-size 2: Splits attention across 2 GPUs (such as 2x A100-80GB or 2x H100-80GB).^[source]
• --enable-expert-parallel: Activates distributed routing for MoVA value experts and FFN experts.^[source]
• --max-model-len 131072: Caps context window at 128K tokens to prevent OOM when serving high-concurrency requests.^[source]
• --reasoning-parser k2_horizon: Correctly parses internal reasoning thoughts into the OpenAI-compatible reasoning_content field.^[source]

2. SGLang recipe for high-throughput serving

For deployments prioritizing low prefill latency on 2x NVIDIA H100/H200 nodes, SGLang supports FlashAttention-3 and router GEMM partitioning:^[source]

python3 -m sglang.launch_server \
  --model-path IFM/K2-Horizon-MoVA-36B-A4B \
  --revision main \
  --tp 2 \
  --ep 2 \
  --dtype bfloat16 \
  --attention-backend fa3 \
  --json-model-override-args '{"xllm_source_router_gemm_partitions":2}' \
  --reasoning-parser k2_horizon \
  --tool-call-parser k2_horizon \
  --host 0.0.0.0 --port 30000

For broader context on sizing and baseline memory rules across model architectures, review our LLM GPU VRAM requirements guide.

Cloud hosting cost scenarios and deployment recommendations

When deploying K2-Horizon-MoVA-36B in the cloud, infrastructure costs vary based on uptime requirements, target sequence lengths, and precision tiers.

┌────────────────────────────────────────────────────────────────────────┐
│            K2-Horizon-MoVA-36B Workload Architecture Matrix            │
├────────────────────────────────────────────────────────────────────────┤
│                                                                        │
│   Workload 1: Local Developer / Small Team Testing                     │
│   • Hardware: 1x RTX 5090 32GB (INT4 AWQ)                              │
│   • Target Context: <= 32K tokens                                      │
│   • Strategy: Community Cloud or on-demand workstation instance.       │
│                                                                        │
│   Workload 2: Production Microservice API (High Concurrency)           │
│   • Hardware: 1x H100-80GB SXM (FP8 Weights, FP8 KV Cache)             │
│   • Target Context: <= 128K tokens                                     │
│   • Strategy: Dedicated secure cloud pod on CoreWeave, Lambda, RunPod. │
│                                                                        │
│   Workload 3: Full 512K Enterprise Context & Reasoning                 │
│   • Hardware: 1x AMD MI325X 256GB OR 2x NVIDIA A100-80GB SXM          │
│   • Target Context: 128K to 512K tokens                                │
│   • Strategy: Reserved cluster with high-speed unified memory.         │
│                                                                        │
└────────────────────────────────────────────────────────────────────────┘

Recommendation summary:

1. For prototyping: Deploy the INT4 quantized model on a single 32GB RTX 5090 or 24GB RTX 4090 on a community cloud provider. This minimizes cost while providing sufficient memory for up to 32K context windows.
2. For production web applications: Host the FP8 checkpoint on a single NVIDIA H100 80GB SXM instance. This setup offers high throughput (4B active parameter compute) and handles up to 128K context lengths with low latency.
3. For long-document analysis (512K context): Rent a single 256GB AMD Instinct MI325X or a 2x A100-80GB SXM pod connected via NVLink. Avoid PCIe-only clusters due to all-to-all expert routing overhead.

To inspect verified provider rates and cluster availability across all mentioned GPU architectures, use our real-time GPU cloud price lookup index. All research and benchmarking methodology adheres to our published editorial methodology.

Frequently asked questions