How to Deploy LLM on GPU Cloud: Step-by-Step Guide

How to deploy LLM on GPU cloud: step-by-step overview

Learning how to deploy LLM on GPU cloud infrastructure enables developers to serve high-throughput inference endpoints for open-weight Large Language Models such as Llama 3, Qwen 2.5, or Mistral.^[source]^[source] Rather than relying on closed-source APIs, hosting open-weight models on cloud GPUs grants full control over data privacy, custom context windows, and inference latency.^[source]

By selecting appropriate GPU hardware (such as an NVIDIA RTX 4090, A100, or H100) and utilizing high-performance inference servers like vLLM or Hugging Face Text Generation Inference (TGI), engineers can deploy OpenAI-compatible REST API endpoints in under fifteen minutes.^[source]^[source]

Key takeaways

Deploying open-weight LLMs requires matching model parameter size to GPU VRAM (e.g., 8B models fit 24GB VRAM, while 70B models require 80GB to 141GB VRAM).^[source]
vLLM utilizes PagedAttention to reduce KV cache memory fragmentation by up to 96 percent, boosting token generation throughput.^[source]
Text Generation Inference (TGI) provides built-in token streaming, tensor parallelism, and Prometheus metrics out of the box.^[source]
GPU Picks does not host user containers or perform live latency audits.

This guide provides a practical step-by-step framework for calculating VRAM requirements, selecting cloud instances, and launching vLLM inference containers. Learn more about our evaluation principles in our editorial methodology.

Hardware sizing: calculating VRAM requirements

Before launching a cloud GPU instance, calculate the exact VRAM footprint required to host your target Large Language Model.^[source]

The VRAM calculation formula

Model memory consumption consists of three primary components:

Total VRAM = Model Weights + KV Cache Memory + Activation Overhead

Model Weights: Calculated based on parameter count and numeric precision:

FP16 / BF16 (16-bit): $ $text{Parameters (in billions)} \$ times 2 $text{ GB}$ (e.g., Llama 8B requires ~16GB VRAM for weights alone).<sup class="citation"><a href="#source-vllm-docs" aria-label="Citation: vLLM Production Documentation by vLLM Project">[source]</a></sup></li> <li><strong>INT8 (8-bit quantization)</strong>: $\$ text{Parameters (in billions)} $times 1\$ text{ GB}$ (e.g., Llama 70B requires ~70GB VRAM).^[source]

INT4 (4-bit quantization): $

text{Parameters (in billions)} \

times 0.5

text{ GB}$ (e.g., Llama 70B requires ~35GB VRAM).<sup class="citation"><a href="#source-vllm-docs" aria-label="Citation: vLLM Production Documentation by vLLM Project">[source]</a></sup></li> </ul> </li> <li><strong>KV Cache Memory</strong>: Stores key-value attention tensors across active user context windows. Allocating 20 to 30 percent additional VRAM headroom above model weight sizing ensures high concurrent request handling without out-of-memory (OOM) crashes.<sup class="citation"><a href="#source-vllm-docs" aria-label="Citation: vLLM Production Documentation by vLLM Project">[source]</a></sup></li> <li><strong>Activation Overhead</strong>: Requires approximately 1GB to 2GB VRAM for temporary tensor buffers during forward passes.</li> </ol> <h3>GPU hardware selection matrix</h3> <p>Use the matrix below to match target open-weight models with appropriate cloud GPU instances:</p> <table> <thead> <tr> <th>Model Scale</th> <th>Quantization</th> <th>Required VRAM</th> <th>Recommended GPU Cloud Instance</th> </tr> </thead> <tbody> <tr> <td><strong>7B / 8B Model</strong></td> <td>FP16 (Unquantized)</td> <td>~18GB VRAM</td> <td>1x NVIDIA RTX 4090 (24GB) or 1x L4 (24GB)<sup class="citation"><a href="#source-vllm-docs" aria-label="Citation: vLLM Production Documentation by vLLM Project">[source]</a></sup></td> </tr> <tr> <td><strong>8B Model</strong></td> <td>INT4 / INT8</td> <td>~10GB VRAM</td> <td>1x NVIDIA RTX 3090 (24GB) or 1x A10 (24GB)<sup class="citation"><a href="#source-vllm-docs" aria-label="Citation: vLLM Production Documentation by vLLM Project">[source]</a></sup></td> </tr> <tr> <td><strong>70B Model</strong></td> <td>INT4 Quantized</td> <td>~45GB VRAM</td> <td>1x NVIDIA L40S (48GB) or 1x A100 (80GB)<sup class="citation"><a href="#source-vllm-docs" aria-label="Citation: vLLM Production Documentation by vLLM Project">[source]</a></sup></td> </tr> <tr> <td><strong>70B Model</strong></td> <td>FP16 (Unquantized)</td> <td>~150GB VRAM</td> <td>2x NVIDIA A100 (80GB) or 1x NVIDIA H200 (141GB)<sup class="citation"><a href="#source-vllm-docs" aria-label="Citation: vLLM Production Documentation by vLLM Project">[source]</a></sup></td> </tr> </tbody> </table> <p>To compare live hourly pricing across these GPUs, use our interactive <a href="/lookup/">GPU lookup tool</a>.</p> <h2>Step-by-step guide: how to deploy LLM on GPU cloud with vLLM</h2> <p>Follow these four steps to deploy an OpenAI-compatible vLLM server on a Linux cloud instance.</p> <h3>Step 1: Launch a GPU instance with Docker and CUDA</h3> <p>Provision a cloud GPU instance (such as a RunPod Pod, Lambda VM, or TensorDock host) running Ubuntu with NVIDIA Container Toolkit pre-installed.<sup class="citation"><a href="#source-vllm-docs" aria-label="Citation: vLLM Production Documentation by vLLM Project">[source]</a></sup> You can evaluate hosting platforms on our <a href="/best-gpu-cloud-for-inference/">best GPU cloud for inference</a> guide.</p> <p>Verify CUDA drivers using SSH terminal access:</p> <pre><code class="language-bash">nvidia-smi </code></pre> <h3>Step 2: Launch the vLLM Docker container</h3> <p>Run the official vLLM Docker container, passing your Hugging Face API token (if downloading gated models like Llama 3) and mapping host port 8000 to the container:<sup class="citation"><a href="#source-vllm-docs" aria-label="Citation: vLLM Production Documentation by vLLM Project">[source]</a></sup></p> <pre><code class="language-bash">docker run --gpus all \

-e HUGGING_FACE_HUB_TOKEN="your_hf_token_here"

-p 8000:8000 \

--ipc=host

vllm/vllm-openai:latest \

--model meta-llama/Meta-Llama-3-8B-Instruct

--max-model-len 8192 \

--gpu-memory-utilization 0.90

Step 3: Test the OpenAI-compatible API endpoint

Once vLLM finishes loading model weights into GPU VRAM, send an HTTP POST request to test text generation:^[source]

curl http://localhost:8000/v1/chat/completions  $-H &quot;Content-Type: application/json&quot; \$ 
    -d '{
        "model": "meta-llama/Meta-Llama-3-8B-Instruct",
        "messages": [
            {"role": "system", "content": "You are a helpful AI assistant."},
            {"role": "user", "content": "Explain GPU memory bandwidth in two sentences."}
        ],
        "temperature": 0.7
    }'

Step 4: Expose and secure the API endpoint

For production use, secure your inference server:

Configure Reverse Proxy: Set up NGINX or Caddy with SSL certificates (Let's Encrypt) to encrypt traffic over HTTPS.
Implement API Authentication: Add an API gateway or NGINX authorization header rule to prevent unauthorized request execution.
Set Up Auto-Scaling: On container platforms like RunPod Serverless or Salad, configure auto-scaling worker rules to spin down GPUs during idle periods. Explore setup options in our RunPod review.

Inference engine comparison: vLLM vs TGI vs TensorRT-LLM

Selecting the right inference engine impacts serving throughput and deployment complexity.

Inference Engine	Key Strength	Best Use Case	License Model
vLLM	PagedAttention memory optimization & OpenAI API compatibility^[source]	General LLM serving & dynamic batching	Apache 2.0 (Open Source)
Hugging Face TGI	Production-ready metrics, streaming, & Hugging Face Hub integration^[source]	Enterprise pipelines & Hugging Face models	Open RAIL-M
TensorRT-LLM	Maximum kernel-level optimization for NVIDIA hardware	Ultra-low latency enterprise inference	NVIDIA License

Troubleshooting common deployment bottlenecks

Out-of-Memory (OOM) Errors: Lower --gpu-memory-utilization (e.g., from 0.90 to 0.80) or reduce max context length (--max-model-len).^[source]
Slow First Token Latency (TTFT): Ensure your cloud host uses fast local NVMe storage so model weights load into GPU VRAM quickly during instance startup.
Tensor Parallel Mismatch: When running multi-GPU configurations, ensure the --tensor-parallel-size flag matches the exact number of physical GPUs attached to the instance.^[source]

Learn more about decision criteria on our how to choose GPU cloud guide.

Frequently asked questions

What is the best GPU for deploying an 8B parameter LLM?

For an unquantized 8B model (FP16/BF16), a single 24GB GPU such as an NVIDIA RTX 4090, RTX 3090, or L4 provides sufficient VRAM for model weights and KV cache memory.^[source] Explore options on our best RTX 4090 cloud guide.

Why is vLLM faster than standard PyTorch for LLM inference?

vLLM uses PagedAttention, an algorithm that manages KV cache memory like virtual memory pages in operating systems.^[source] This eliminates memory fragmentation and enables dynamic batching, resulting in higher request throughput.

Can I run quantized LLMs on vLLM?

Yes. vLLM natively supports AWQ, GPTQ, and SqueezeLLM quantized formats, allowing large models to run on GPUs with smaller VRAM allocations.^[source]

What is the difference between single-GPU and tensor-parallel deployment?

Single-GPU deployment runs the entire model on one accelerator. Tensor-parallel deployment splits individual matrix operations across multiple GPUs (e.g., 2x or 4x A100s) connected via high-speed NVLink interconnects.^[source]

Is vLLM compatible with OpenAI SDKs?

Yes. vLLM includes a built-in HTTP server that mimics the OpenAI REST API specification, allowing you to swap endpoint URLs in standard OpenAI Python or TypeScript SDKs.^[source]

Sourcing and editorial methodology

GPU Picks collects technical specifications, deployment instructions, and framework capabilities directly from official project documentation and open-source project repositories. We do not host user instances or perform paid latency benchmarks. Learn more on our editorial methodology page.

For further comparisons, search live pricing across all providers using our interactive GPU lookup tool.