Deploying LLM Workloads: GPUs, Kubernetes, and Serverless

A team I know shipped a fine-tuned 13B model to production on a single A100 VM. It worked fine for two months. Then a product launch brought 20x their usual traffic. The VM maxed out, requests queued, TTFT climbed to 45 seconds, and users assumed the product was broken. They had a great serving engine config. They had zero autoscaling. The engine is only as good as the infrastructure running it.

This article is about that infrastructure layer — GPU clouds, Kubernetes patterns, and serverless GPU platforms — specifically for LLM workloads, which behave very differently from normal web services under load.

GPU cloud options, ranked by operational overhead

Three deployment tiers exist, and the right one depends entirely on your traffic volume and operational bandwidth.

Managed inference endpoints — AWS Bedrock, Google Vertex AI Model Garden, Azure AI Foundry, Hugging Face Inference Endpoints — handle everything below the API. You provide a model ID and target throughput; they provision hardware, run the serving engine, autoscale, and charge per token. The trade-off is cost: managed endpoints typically run 2-5x more expensive per token than self-hosted equivalents at the same hardware level, because you're paying for operational overhead and margin. For teams without GPU infrastructure experience, this premium is usually worth it, at least until monthly GPU spend crosses a level that justifies hiring someone who knows what nvidia-smi is telling them.

Self-managed on cloud instances — running vLLM or SGLang directly on GPU VMs from AWS (p3/p4/p5 families), GCP (A100/H100 VMs), Azure (NC/ND series), or GPU-focused clouds like Lambda Labs, CoreWeave, or Vultr — gives full control over the serving engine and pays the raw hardware rate. H100 SXM 80GB instances ran roughly $3-5/hr on-demand and $2-3/hr spot in mid-2026 (illustrative; prices shift). The catch is everything the managed tier handles for free: you provision the node, manage the Kubernetes cluster, handle node failures, track GPU utilization, implement autoscaling, and figure out where to store 140GB of model weights.

Serverless GPU platforms — Modal, RunPod Serverless, Replicate, Baseten — sit between these two. You write code that runs on a GPU and they manage node pools, cold start optimization, and per-second billing. You pay only for active inference time — at a per-hour premium over raw instances (an H100 that runs ~$3.50/hr as a dedicated VM costs roughly $6-8/hr serverless, illustrative mid-2026). At low to moderate traffic, the pay-for-active-seconds model beats that premium easily, making this the cheapest self-hosting option by a significant margin.

flowchart LR
    TRAFFIC["Traffic shape"] --> LOW["Low / bursty\n< 10 GPU-hrs/day"]
    TRAFFIC --> MED["Moderate\n10-50 GPU-hrs/day"]
    TRAFFIC --> HIGH["High sustained\n> 50 GPU-hrs/day"]
    LOW --> SL["Serverless GPU\nModal / RunPod"]
    MED --> COMP["Compare:\nserverless vs dedicated instances"]
    HIGH --> SELF["Self-managed on K8s\nor dedicated VMs"]
    style SL fill:#a855f7,color:#fff
    style SELF fill:#15803d,color:#fff
    style COMP fill:#ffaa00,color:#0a0a0f

The crossover point with the illustrative rates above is roughly 10-15 GPU-hours of active inference per day — the point where serverless's per-hour premium equals the cost of an always-on dedicated box ($84/day ÷ ~$7/hr ≈ 12 hours). Below that, the idle time on a dedicated instance makes serverless obviously cheaper. Above 50 GPU-hours per day (multi-replica territory), a dedicated instance or self-managed cluster wins with any realistic pricing. In between, the answer depends on your actual rates — run the numbers.

Kubernetes for LLM workloads

If you're running vLLM or SGLang on Kubernetes, most standard patterns work — but four LLM-specific problems will burn you if you don't account for them.

Autoscaling on the wrong signal

The Kubernetes Horizontal Pod Autoscaler defaults to CPU and memory utilization. Neither is a useful signal for LLM serving pods. A vLLM instance can have 10% CPU utilization while its GPU is saturated and new requests are queuing for 20 seconds. Scaling on CPU% in this scenario does nothing useful.

The correct signal is queue depth or pending requests — how many requests are waiting to be processed by the serving engine. vLLM exposes this via its /metrics Prometheus endpoint as vllm:num_requests_waiting. SGLang exposes similar metrics.

flowchart TD
    VLLM["vLLM pod\n/metrics endpoint"] -->|"vllm:num_requests_waiting"| PROM["Prometheus"]
    PROM --> ADAPTER["Prometheus Adapter\n(or KEDA)"]
    ADAPTER -->|custom metric| HPA["HPA\nscaleOn: pending_requests > 5"]
    HPA -->|"scale up"| NEWPOD["New vLLM pod"]
    style VLLM fill:#0e7490,color:#fff
    style HPA fill:#00e5ff,color:#0a0a0f
    style NEWPOD fill:#15803d,color:#fff

The practical implementation uses either the Prometheus Adapter (map the Prometheus metric to a Kubernetes custom metric that HPA understands) or KEDA (Kubernetes Event-Driven Autoscaling), which has a Prometheus scaler built in and is generally easier to configure. A KEDA ScaledObject targeting vllm:num_requests_waiting > 5 will scale up aggressively when requests queue and back down slowly when load drops.

Scale-up should be fast. Scale-down should be slow. Set a stabilization window of 15-30 minutes on scale-down — LLM traffic is bursty, and a pod that goes idle for 3 minutes during a lull will be killed, then you'll cold-start it 8 minutes later when traffic comes back. That cycle is painful for users and wastes the per-request GPU spin-up cost.

GPU resource requests and limits

Set nvidia.com/gpu: 1 (or however many GPUs your model needs) in the pod's resource requests. Do not set GPU limits differently from requests — Kubernetes GPU scheduling doesn't support overcommitment, and mismatched request/limit values cause confusing scheduling behavior.

For multi-GPU models (anything that doesn't fit on a single device), tensor parallelism requires all the GPUs to be on the same node and connected via NVLink. This means you need pods that request multiple GPUs and nodes large enough to hold them. A Llama 3.1 70B model at BF16 needs ~140GB — two H100 80GB cards minimum, or four A100 40GB cards. Request nvidia.com/gpu: 2 and use a node selector or node affinity to land on NVLink-equipped nodes (AWS: p4de or p5 instances; GCP: A100 or H100 VMs with NVLink topology).

For single-GPU models with quantization, a 7B AWQ model fits on an A10G (24GB) with room for the KV cache. A 13B FP8 model fits on an H100 80GB with generous KV cache headroom. The KV cache memory math determines how much of that 80GB the KV cache eats at your target concurrency — work that out before you spec your node type.

Model weight storage

Model weights are large. Llama 3.1 70B in BF16 is 140GB. Baking that into a container image is impractical — you'd push 140GB on every image update and your container registry would be expensive. The production pattern is:

1. Store weights in object storage (S3, GCS) or in a dedicated artifact store (Hugging Face Hub, a self-hosted registry).
2. On pod startup, download weights to a local path inside the container. This is the cold-start cost.
3. Use a hostPath volume or a Persistent Volume with a node-local SSD to cache weights across pod restarts on the same node.

The difference in startup time between "download 140GB from S3 from scratch" (4-6 minutes at ~400MB/s sustained) and "load from node-local NVMe SSD" (~25-35 seconds of raw disk read at 4-6 GB/s, plus deserialization, CUDA context init, and the HBM transfer — 90-120 seconds end to end) is significant. Keeping weights warm on the node is worth the disk cost.

An alternative for smaller models (under ~20GB): bake weights into the container image. A 7B FP8 model is ~7GB, which is manageable as a container layer. You get reproducible deployments and instant startup at the cost of a larger registry footprint.

Model weight load time estimates (illustrative, 2026 typical cloud speeds):

7B BF16 (14GB):
  From S3 (500 MB/s): ~28 seconds
  From node-local NVMe: ~5-10 seconds
  
70B BF16 (140GB):
  From S3 (500 MB/s): ~280 seconds (4.7 minutes)
  From node-local NVMe (4-6 GB/s): ~25-35 seconds
  
70B AWQ 4-bit (35GB):
  From S3: ~70 seconds
  From node-local NVMe: ~7-10 seconds

→ For large models, node-local weight caching cuts cold start by 5-10x.
  Quantization cuts it further by reducing the bytes to load.
  (Figures are raw read time. End-to-end pod startup adds deserialization,
  CUDA context init, and the disk→HBM transfer — roughly another 60-90s
  for a 70B model.)

Health probes and graceful shutdown

Those multi-minute weight loads have a direct consequence that default Kubernetes config gets wrong. A default liveness probe gives a pod roughly 30 seconds to respond before killing it — a vLLM pod that's two minutes into loading a 70B model gets killed mid-load, restarted, killed again, and lands in CrashLoopBackOff at exactly the moment autoscaling is trying to add capacity. Use a startupProbe with a failureThreshold × periodSeconds budget sized to your worst-case weight load (failureThreshold: 60, periodSeconds: 10 buys ten minutes), gate readiness on the serving engine's own health endpoint (/health on vLLM) so traffic only arrives once the engine can serve, and keep the liveness probe lenient after startup completes.

The mirror problem on the way down: LLM requests are long. A streaming generation can run for minutes, and the default 30-second termination grace period means every rolling deploy or scale-down kills in-flight generations mid-sentence. Set terminationGracePeriodSeconds to your longest expected generation plus a buffer, and add a preStop hook that stops accepting new requests and drains the active ones before the SIGTERM-to-SIGKILL clock runs out.

Serverless GPU: what you actually get

Modal is the most engineering-friendly of the serverless GPU platforms as of mid-2026. You define your serving function in Python, annotate it with @app.function(gpu="H100"), and Modal handles container builds, GPU pool management, concurrency, and per-second billing. Cold starts are Modal's main engineering investment — they pre-warm containers and cache snapshots, with startup times for a pre-built container of around 5-15 seconds for small models. The GPU itself spinning up (CUDA context init, weight loading) adds more.

RunPod offers both serverless (event-driven, per-second billing) and pod (persistent instance, hourly billing) modes. The serverless mode is similar to Modal; pods are persistent VM-like GPU instances useful for long-running jobs like fine-tuning or large inference workloads that need a stable endpoint. RunPod's per-hour GPU prices tend to be lower than Modal for equivalent hardware, but with less SDK polish and more manual configuration.

Replicate wraps models in a simple prediction API. It's the easiest path if you want to call an existing published model — they host many popular open-weight models out of the box. For custom models with custom serving configurations, Replicate's Cog framework adds complexity. Less flexible for tuning vLLM parameters.

All three platforms have the same fundamental limitation at high sustained load: they sit between you and the raw hardware, adding latency and markup. Once you're running more than 10-15 GPU-hours per day consistently, compare the serverless all-in price against a dedicated instance and do the math.

Cold starts: the actual numbers

Cold start latency on serverless GPU has three components, and vendors often quote only the first:

1. Container scheduling and GPU attachment: 2-15 seconds depending on pool availability and instance type.
2. Container startup (if not pre-warmed): 10-60 seconds to load your Python environment and dependencies.
3. Model weight loading: dominates for large models. A 70B BF16 model loading from object storage takes 4-6 minutes. With quantization (35GB AWQ), it drops to ~70 seconds. With node-local weight caching (platform-dependent), it drops further.

For most production applications, a 5-minute cold start is a user-facing failure. The mitigations:

• Minimum warm replicas: keep 1-2 instances always running. This eliminates cold starts for most traffic patterns but costs ~$2-5/hr in GPU idle time. Do the math against your traffic to decide if it's worth it.
• Container snapshots: Modal's memory snapshotting and RunPod's template system can capture the post-weight-load state and resume from there, cutting cold start dramatically for models where weights are the bottleneck.
• Quantized models: load 4-bit AWQ weights instead of BF16 — 4-5x smaller, 4-5x faster to load, near-equivalent quality for most tasks (see the quantization tradeoffs article for the cases where it isn't).

Multi-GPU and tensor parallelism

When your model doesn't fit on one GPU, you need tensor parallelism (TP) — splitting the model's weight matrices across multiple GPUs and doing synchronized matmul operations. vLLM and SGLang both support this natively via the --tensor-parallel-size flag.

The constraint: all TP ranks must communicate during each forward pass. On a single node, GPUs connected by NVLink exchange data at ~600-900 GB/s (NVLink 4.0 on H100), which adds a few milliseconds per token. Across nodes over InfiniBand or RoCE (network-attached interconnects), latency is higher and more variable, and the engineering complexity of multi-node tensor parallelism is significant. Stay single-node for TP if at all possible.

Pipeline parallelism (PP) — where different transformer layers run on different GPUs — is an alternative for very large models but adds pipeline bubble latency (stages wait for each other) and is harder to tune for serving workloads versus training. For inference, TP is almost always preferred over PP.

If you need to serve a 405B model (Llama 3.1 405B BF16 = ~810GB weights), a single 8x H100 node's 640GB of HBM can't even hold the weights — you're looking at 16x H100 80GB across two nodes for BF16, or a single 8-GPU node running the FP8 checkpoint (~405GB), which is exactly why Meta ships an FP8 reference build for 405B. At that scale, you're typically running on dedicated bare-metal or large multi-GPU cloud instances, not Kubernetes pods.

The practical deployment stack

For most teams self-hosting LLM inference, the stack looks like this:

flowchart TD
    CLIENT["Client / API caller"] --> GATEWAY["LLM Gateway\n(rate limiting, auth, routing)"]
    GATEWAY --> LB["K8s Service / Load Balancer"]
    LB --> POD1["vLLM pod (H100)\nReplica 1"]
    LB --> POD2["vLLM pod (H100)\nReplica 2"]
    LB --> POD3["vLLM pod (H100)\nReplica 3"]
    POD1 & POD2 & POD3 --> WEIGHTS[("Model weights\nNode-local NVMe\nor shared PVC")]
    KEDA["KEDA\nscales on queue depth"] -.-> POD1
    PROM["Prometheus"] -.->|"metrics scrape"| POD1
    PROM -.-> KEDA
    style GATEWAY fill:#00e5ff,color:#0a0a0f
    style POD1 fill:#0e7490,color:#fff
    style POD2 fill:#0e7490,color:#fff
    style POD3 fill:#0e7490,color:#fff
    style KEDA fill:#a855f7,color:#fff

The LLM gateway sits in front and handles rate limiting, API key auth, cost attribution, and routing across models or providers. Behind it, a standard Kubernetes Service load-balances across serving pods. KEDA watches Prometheus metrics from the pods and adjusts replica count based on queue depth. Weights live on node-local SSDs, pre-downloaded via an init container or a DaemonSet that pre-populates the cache on new nodes.

Each vLLM pod exposes the OpenAI-compatible API endpoint, which means your application code doesn't care whether it's talking to a self-hosted vLLM, a managed endpoint, or the actual OpenAI API — useful for swapping between environments.

What breaks

GPU node failures

GPU nodes fail more often than CPU nodes. HBM memory errors, CUDA driver faults, thermal events — these are not rare. Your serving deployment needs to handle pod eviction and rescheduling gracefully. In Kubernetes, a PodDisruptionBudget ensures that rolling node maintenance doesn't take down all replicas simultaneously. But an unexpected node failure will still take one pod down hard. With three replicas, you need the remaining two to handle the traffic spike that happens during the ~2-5 minutes it takes the new pod to come up and warm.

Design for N-1 capacity: your target steady-state should run at ~60-70% of maximum throughput per pod, leaving headroom for one pod to drop without breaching SLAs.

KV cache OOM at unexpected request lengths

The KV cache scales with context length — longer requests eat more VRAM. If you size your vLLM configuration for average request lengths but your 99th-percentile requests are 10x longer, you'll hit OOM errors at unexpected moments. vLLM handles this somewhat gracefully (it swaps sequences out to CPU memory and recomputes rather than crashing), but swapping is expensive and dramatically increases latency for those requests.

Set --max-model-len to a realistic upper bound for your workload and reject requests that exceed it, rather than letting the engine swap or crash. Monitor vllm:gpu_cache_usage_perc — if it's consistently above 85%, you're close to the edge.

Multi-LoRA adapter routing errors

If you're serving multiple LoRA adapters from a single base model (the multi-LoRA pattern for per-customer fine-tuning), routing errors where the wrong adapter is loaded for a request are subtle and hard to catch. The serving engine doesn't know which adapter should handle which request — that logic lives in your routing layer. Implement adapter ID verification: log the requested adapter and the loaded adapter for every request, and alert on mismatches. This sounds obvious; teams regularly skip it until a customer complains that their fine-tuned model is returning generic responses.

Cold start under autoscaling

The most insidious failure mode: traffic spikes, KEDA triggers a scale-out, new pods start — and take 3-5 minutes to load weights and start serving. During those minutes, the existing pods are handling 100% of traffic above their steady-state capacity. If they're already at 70% utilization and traffic doubles, you've got 3-5 minutes of degraded service while the new pod loads.

Mitigation: pre-warm at least one extra replica during known high-traffic windows (product launches, business hours spikes), or keep a slightly oversized fleet so that scaling events trigger before the existing pods are saturated. The goal is to trigger autoscaling at 60% utilization, not 95%.

Spot interruptions and GPU scarcity

Those $2-3/hr spot prices come with a contract clause people skip: the cloud can reclaim the node with roughly two minutes' warning, and replacing an LLM serving replica takes 3-5 minutes of weight loading. Do that math — a spot-only serving fleet has a built-in capacity hole on every interruption, and interruptions cluster exactly when regional GPU demand spikes. If you serve from spot, handle the interruption notice (drain the pod the moment it arrives) and keep enough on-demand capacity to absorb a lost replica without breaching SLAs. Or restrict spot to batch and offline inference, where a killed job just restarts.

The quieter version of the same problem is getting GPUs at all. H100 quota in your preferred region can take weeks to approve, some instance types simply aren't stocked in some regions, and serverless platforms' shared pools can be exhausted during demand spikes — that's the "pool availability" component of cold start from earlier. Treat quota and regional availability as deployment constraints you verify before committing your architecture to a specific GPU type, not paperwork you file afterward.

When to use what

The decision is mostly about traffic pattern and team capacity, not technical capability:

Use a managed endpoint if you're pre-PMF, running a pilot, or don't have someone who can be paged for GPU node failures at 2am. The 2-5x cost premium buys you that operational quiet. Managed endpoints are also the right answer for compliance-heavy environments where you want the vendor to own infrastructure SLAs.

Use serverless GPU if your traffic is bursty — several hours of activity per day, then silence. Paying for a GPU that's idle 80% of the time is expensive. Modal and RunPod let you pay only for the active window. Keep one warm replica if your latency SLA is tighter than your measured worst-case cold start — roughly 30 seconds for a small pre-warmed model, 4-6 minutes for a 70B loading from storage. Skip it only if your use case can absorb that worst case at whatever cold-start rate you actually observe.

Self-managed vLLM on Kubernetes is right when you have sustained high traffic, cost is a top concern, and you have the engineering bandwidth to own the infrastructure. The operational investment pays back at scale — at 1,000 GPU-hours per month, a 2x cost difference is real money. It's also the only option if you need specific serving engine configurations that managed platforms don't expose: custom chunked-prefill chunk sizes, multi-LoRA serving, disaggregated prefill/decode across a mixed H100+H200 fleet.

For the serving engine running inside that infrastructure, the serving engine comparison covers vLLM vs SGLang vs TensorRT-LLM in depth. The short version: start with vLLM unless you have a specific reason for SGLang (heavy prefix sharing) or TensorRT-LLM (need the last 20% of peak H100 throughput and are willing to maintain a compiled model build pipeline). The throughput vs latency tradeoffs article covers how to configure whichever engine you pick for your specific SLAs.

The underlying tension in LLM deployment is that the infrastructure decisions that save money (fewer replicas, no warm reserves, smaller instances) are exactly the decisions that hurt latency and reliability. There's no clever abstraction that escapes this — only an honest accounting of where your traffic sits on the cost-latency curve, and an infrastructure setup that matches it.

And if you land on the managed, per-token side of the decision, know what you're actually paying for: the widget below breaks a monthly API bill into input, output, and cached tokens. Output tokens carry a 3-5x premium over input, and the cache hit rate moves the total more than most teams expect — which is why the per-token option stays cheap far longer than intuition suggests.

{ "type": "token-cost", "title": "What a monthly API bill looks like: input vs output vs cached tokens" }