---
title: "How to Deploy LLM on GPU Cloud (2026): Step-by-Step Guide"
slug: "how-to-deploy-llm-on-gpu-cloud"
meta_description: "Learn how to deploy LLM on GPU cloud infrastructure using vLLM and TGI. Step-by-step tutorial covering instance selection, memory, and API setup."
schema_type: "howto"
author: "ahmad-nugraha"
primary_keyword: "how to deploy llm on gpu cloud"
secondary_keywords:
  - "deploy llm on gpu cloud"
  - "vllm cloud deployment"
  - "huggingface tgi setup"
  - "llm hosting gpu cloud"
search_intent: "informational"
commercial: true
published_at: "2026-08-01"
updated_at: "2026-08-01"
status: "published"
human_reviewed_by: "ahmad-nugraha"
human_reviewed_at: "2026-08-01"
sources:
  - id: vllm-docs
    title: "vLLM Production Documentation"
    publisher: "vLLM Project"
    url: "https://docs.vllm.ai/"
    accessed_at: "2026-08-01"
    source_type: technical
  - id: huggingface-tgi
    title: "Text Generation Inference (TGI) Documentation"
    publisher: "Hugging Face"
    url: "https://huggingface.co/docs/text-generation-inference/"
    accessed_at: "2026-08-01"
    source_type: technical
---

## How to deploy LLM on GPU cloud: step-by-step overview

Learning how to deploy LLM on GPU cloud infrastructure enables developers to serve high-throughput inference endpoints for open-weight Large Language Models such as Llama 3, Qwen 2.5, or Mistral.[cite id="vllm-docs"][cite id="huggingface-tgi"] Rather than relying on closed-source APIs, hosting open-weight models on cloud GPUs grants full control over data privacy, custom context windows, and inference latency.[cite id="vllm-docs"]

By selecting appropriate GPU hardware (such as an NVIDIA RTX 4090, A100, or H100) and utilizing high-performance inference servers like vLLM or Hugging Face Text Generation Inference (TGI), engineers can deploy OpenAI-compatible REST API endpoints in under fifteen minutes.[cite id="vllm-docs"][cite id="huggingface-tgi"]

[key-takeaways]
- Deploying open-weight LLMs requires matching model parameter size to GPU VRAM (e.g., 8B models fit 24GB VRAM, while 70B models require 80GB to 141GB VRAM).[cite id="vllm-docs"]
- vLLM utilizes PagedAttention to reduce KV cache memory fragmentation by up to 96 percent, boosting token generation throughput.[cite id="vllm-docs"]
- Text Generation Inference (TGI) provides built-in token streaming, tensor parallelism, and Prometheus metrics out of the box.[cite id="huggingface-tgi"]
- GPU Picks does not host user containers or perform live latency audits.
[/key-takeaways]

This guide provides a practical step-by-step framework for calculating VRAM requirements, selecting cloud instances, and launching vLLM inference containers. Learn more about our evaluation principles in our [editorial methodology](/methodology/).

## Hardware sizing: calculating VRAM requirements

Before launching a cloud GPU instance, calculate the estimated VRAM footprint required to host your target Large Language Model.[cite id="vllm-docs"]

### The VRAM calculation formula

Model memory consumption consists of three primary components:

$$\text{Total VRAM} = \text{Model Weights} + \text{KV Cache Memory} + \text{Activation Overhead}$$

1. **Model Weights**: Calculated based on parameter count and numeric precision:
   - **FP16 / BF16 (16-bit)**: $\text{Parameters (in billions)} \times 2\text{ GB}$ (e.g., Llama 8B requires ~16GB VRAM for weights alone).[cite id="vllm-docs"]
   - **INT8 (8-bit quantization)**: $\text{Parameters (in billions)} \times 1\text{ GB}$ (e.g., Llama 70B requires ~70GB VRAM).[cite id="vllm-docs"]
   - **INT4 (4-bit quantization)**: $\text{Parameters (in billions)} \times 0.5\text{ GB}$ (e.g., Llama 70B requires ~35GB VRAM).[cite id="vllm-docs"]
2. **KV Cache Memory**: Stores key-value attention tensors across active user context windows. Allocating 20 to 30 percent additional VRAM headroom above model weight sizing ensures high concurrent request handling without out-of-memory (OOM) crashes.[cite id="vllm-docs"]
3. **Activation Overhead**: Requires approximately 1GB to 2GB VRAM for temporary tensor buffers during forward passes.

### GPU hardware selection matrix

Use the matrix below to match target open-weight models with appropriate cloud GPU instances:

| Model Scale | Quantization | Required VRAM | Recommended GPU Cloud Instance |
|---|---|---|---|
| **7B / 8B Model** | FP16 (Unquantized) | ~18GB VRAM | 1x NVIDIA RTX 4090 (24GB) or 1x L4 (24GB)[cite id="vllm-docs"] |
| **8B Model** | INT4 / INT8 | ~10GB VRAM | 1x NVIDIA RTX 3090 (24GB) or 1x A10 (24GB)[cite id="vllm-docs"] |
| **70B Model** | INT4 Quantized | ~45GB VRAM | 1x NVIDIA L40S (48GB) or 1x A100 (80GB)[cite id="vllm-docs"] |
| **70B Model** | FP16 (Unquantized) | ~150GB VRAM | 2x NVIDIA A100 (80GB) or 1x NVIDIA H200 (141GB)[cite id="vllm-docs"] |

To compare live hourly pricing across these GPUs, use our interactive [GPU lookup tool](/lookup/).

## Step-by-step guide: how to deploy LLM on GPU cloud with vLLM

Follow these four steps to deploy an OpenAI-compatible vLLM server on a Linux cloud instance.

### Step 1: Launch a GPU instance with Docker and CUDA

Provision a cloud GPU instance (such as a RunPod Pod, Lambda VM, or TensorDock host) running Ubuntu with NVIDIA Container Toolkit pre-installed.[cite id="vllm-docs"] You can evaluate hosting platforms on our [best GPU cloud for inference](/best-gpu-cloud-for-inference/) guide.

Verify CUDA drivers using SSH terminal access:

```bash
nvidia-smi
```

### Step 2: Launch the vLLM Docker container

Run the official vLLM Docker container, passing your Hugging Face API token (if downloading gated models like Llama 3) and mapping host port 8000 to the container:[cite id="vllm-docs"]

```bash
docker run --gpus all \
    -e HUGGING_FACE_HUB_TOKEN="your_hf_token_here" \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model meta-llama/Meta-Llama-3-8B-Instruct \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.90
```

### Step 3: Test the OpenAI-compatible API endpoint

Once vLLM finishes loading model weights into GPU VRAM, send an HTTP POST request to test text generation:[cite id="vllm-docs"]

```bash
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "meta-llama/Meta-Llama-3-8B-Instruct",
        "messages": [
            {"role": "system", "content": "You are a helpful AI assistant."},
            {"role": "user", "content": "Explain GPU memory bandwidth in two sentences."}
        ],
        "temperature": 0.7
    }'
```

### Step 4: Expose and secure the API endpoint

For production use, secure your inference server:
- **Configure Reverse Proxy**: Set up NGINX or Caddy with SSL certificates (Let's Encrypt) to encrypt traffic over HTTPS.
- **Implement API Authentication**: Add an API gateway or NGINX authorization header rule to prevent unauthorized request execution.
- **Set Up Auto-Scaling**: On container platforms like RunPod Serverless or Salad, configure auto-scaling worker rules to spin down GPUs during idle periods. Explore setup options in our [RunPod review](/runpod-review/) and learn [how to reduce RunPod Serverless cold starts](/how-to-reduce-runpod-serverless-cold-starts/) for on-demand worker pools.

## Inference engine comparison: vLLM vs TGI vs TensorRT-LLM

Selecting the right inference engine impacts serving throughput and deployment complexity.

| Inference Engine | Key Strength | Best Use Case | License Model |
|---|---|---|---|
| **vLLM** | PagedAttention memory optimization & OpenAI API compatibility[cite id="vllm-docs"] | General LLM serving & dynamic batching | Apache 2.0 (Open Source) |
| **Hugging Face TGI** | Production-ready metrics, streaming, & Hugging Face Hub integration[cite id="huggingface-tgi"] | Enterprise pipelines & Hugging Face models | Open RAIL-M |
| **TensorRT-LLM** | Maximum kernel-level optimization for NVIDIA hardware | Ultra-low latency enterprise inference | NVIDIA License |

## Troubleshooting common deployment bottlenecks

- **Out-of-Memory (OOM) Errors**: Lower `--gpu-memory-utilization` (e.g., from 0.90 to 0.80) or reduce max context length (`--max-model-len`).[cite id="vllm-docs"]
- **Slow First Token Latency (TTFT)**: Ensure your cloud host uses fast local NVMe storage so model weights load into GPU VRAM quickly during instance startup.
- **Tensor Parallel Mismatch**: When running multi-GPU configurations, ensure the `--tensor-parallel-size` flag matches the exact number of physical GPUs attached to the instance.[cite id="vllm-docs"]

Learn more about decision criteria on our [how to choose GPU cloud](/how-to-choose-gpu-cloud/) guide.

## Frequently asked questions

[faq]
## What is the best GPU for deploying an 8B parameter LLM?
For an unquantized 8B model (FP16/BF16), a single 24GB GPU such as an NVIDIA RTX 4090, RTX 3090, or L4 provides sufficient VRAM for model weights and KV cache memory.[cite id="vllm-docs"] Explore options on our [best RTX 4090 cloud](/best-rtx-4090-cloud/) guide.

## Why is vLLM faster than standard PyTorch for LLM inference?
vLLM uses PagedAttention, an algorithm that manages KV cache memory like virtual memory pages in operating systems.[cite id="vllm-docs"] This eliminates memory fragmentation and enables dynamic batching, resulting in higher request throughput.

## Can I run quantized LLMs on vLLM?
Yes. vLLM natively supports AWQ, GPTQ, and SqueezeLLM quantized formats, allowing large models to run on GPUs with smaller VRAM allocations.[cite id="vllm-docs"]

## What is the difference between single-GPU and tensor-parallel deployment?
Single-GPU deployment runs the entire model on one accelerator. Tensor-parallel deployment splits individual matrix operations across multiple GPUs (e.g., 2x or 4x A100s) connected via high-speed NVLink interconnects.[cite id="vllm-docs"]

## Is vLLM compatible with OpenAI SDKs?
Yes. vLLM includes a built-in HTTP server that mimics the OpenAI REST API specification, allowing you to swap endpoint URLs in standard OpenAI Python or TypeScript SDKs.[cite id="vllm-docs"]
[/faq]

## Sourcing and editorial methodology

GPU Picks collects technical specifications, deployment instructions, and framework capabilities directly from official project documentation and open-source project repositories. We do not host user instances or perform paid latency benchmarks. Learn more on our [editorial methodology page](/methodology/).

For further comparisons, search live pricing across all providers using our interactive [GPU lookup tool](/lookup/).