In modern AI‑driven products, serving large language models (LLMs) from on‑prem hardware is becoming a practical alternative to expensive cloud APIs. For DevOps engineers, the challenge is not just getting a model to run, but making it resilient, scalable, and cost‑effective inside a Kubernetes cluster. This guide walks through the end‑to‑end process of deploying a local LLM using the vLLM inference engine, wiring it with Helm, and applying best‑in‑class Kubernetes patterns.

Why Run LLMs Locally on Kubernetes?

Running LLMs in‑house gives you full control over data residency, latency, and compute budgeting. Kubernetes adds a layer of abstraction that lets you treat each model as a microservice—complete with health checks, rolling updates, and declarative configuration. When paired with GPU‑aware scheduling, you can pack multiple inference workloads onto a single node without over‑committing resources.

Prerequisites

Before you start, make sure the following components are in place:

1. GPU Operator – NVIDIA’s GPU Operator automates driver installation, the device plugin, and the container runtime needed for GPU access.
2. NVIDIA Container Toolkit – Enables Docker (or containerd) to expose GPU devices inside containers.
3. Kubernetes Cluster – A cluster with at least one GPU‑enabled node (e.g., a node with an A100 or H100).
4. Helm 3+ – The package manager we’ll use to install vLLM and related services.
5. kubectl – Configured to talk to your cluster.

If any of these pieces are missing, the deployment will fail at the resource‑allocation stage.

Choosing a Serving Layer

There are three common ways to expose an LLM inside Kubernetes:

1. Simple Deployment

A plain Deployment with a single container running vLLM. This is the quickest way to get started but offers limited observability and scaling flexibility.

2. KServe

KServe (formerly KFServing) provides a CRD‑based workflow for model serving, including model versioning, canary rollouts, and built‑in metrics. It integrates well with Istio or Ambassador for traffic routing.

3. Ray Serve

Ray Serve shines when you need complex request routing or model ensembles. It runs a Ray cluster inside Kubernetes and lets you define custom Python back‑ends.

For most teams that need a balance between simplicity and production‑grade features, KServe is the sweet spot. The example in this guide uses a plain Deployment for clarity, but the Helm chart can be swapped to a KServe InferenceService with minimal changes.

Resource Management: GPU Requests, Limits, and Bin‑Packing

Kubernetes treats GPUs as a special resource. You must declare both requests and limits for the nvidia.com/gpu resource in your pod spec. For example, a 7B model may need 1 GPU, while a 70B model could require 8 GPUs.

When multiple pods share a node, the scheduler attempts to bin‑pack them based on the declared GPU count. Over‑requesting leads to wasted capacity; under‑requesting can cause pod eviction. Use the GPU Operator’s node-feature-discovery to expose GPU memory and compute capability as node labels, then write a custom scheduler extender or use the open‑source gpu-scheduler to make bin‑packing smarter.

Autoscaling with Custom Metrics

Horizontal Pod Autoscaler (HPA) works out of the box with CPU and memory, but LLM workloads are often bottlenecked by request queue depth. Export a custom metric—such as vllm_queue_length—via Prometheus, then configure HPA to react to it.

When the queue length exceeds the threshold, the HPA will spin up additional pods, each pulling a GPU from the node pool.

Helm Chart Walkthrough

The Helm chart provided in this guide encapsulates all the pieces discussed above. Below is a high‑level overview of the key templates.

Chart Structure

my-vllm-chart/ ├─ Chart.yaml ├─ values.yaml ├─ templates/ │ ├─ deployment.yaml │ ├─ service.yaml │ ├─ hpa.yaml │ └─ configmap.yaml

values.yaml

deployment.yaml

The template injects the GPU requests, mounts the model volume, and sets the entrypoint to launch vLLM with appropriate flags (--model, --dtype, etc.). It also adds a side‑car container that runs a Prometheus exporter for queue metrics.

hpa.yaml

Conditional on hpa.enabled, this file creates the custom‑metric HPA shown earlier.

Deploying the Chart

After installation, verify the pods are running with kubectl get pods -l app=my-llm. Use kubectl port-forward or expose the service via an Ingress to start sending inference requests.

Observability and Debugging

1. Prometheus + Grafana – Scrape the exporter endpoint and visualize queue depth, GPU utilization, and request latency.
2. kubectl logs – Tail the vLLM container logs for error messages related to model loading or CUDA failures.
3. NVIDIA‑SMI – Run nvidia-smi inside the pod (via kubectl exec) to confirm the GPU is visible and not throttled.

Best Practices Checklist

• Pin GPU driver versions via the GPU Operator to avoid mismatched CUDA libraries.
• Store models on a fast, shared filesystem (e.g., NFS or Lustre) and mount it read‑only.
• Enable pod security policies that restrict privileged escalation but allow GPU device access.
• Set resource limits conservatively; monitor actual usage before scaling up.
• Automate Helm upgrades with CI/CD pipelines to roll out new model versions without downtime.

Key Takeaways

• Kubernetes provides a robust platform for serving local LLMs when combined with GPU‑aware scheduling.
• vLLM offers high‑throughput inference; Helm makes its deployment repeatable and version‑controlled.
• Custom‑metric HPA based on request queue depth yields better scaling than CPU‑based autoscaling.
• Proper GPU resource requests, bin‑packing, and observability are essential to keep costs under control.

Conclusion

Deploying a local LLM on Kubernetes is no longer a research‑only activity; it’s a production‑ready pattern that can be codified in a Helm chart and managed like any other microservice. By following the steps outlined—setting up the GPU Operator, choosing the right serving layer, managing GPU resources, and leveraging custom‑metric autoscaling—you can deliver low‑latency, high‑throughput AI capabilities while retaining full control over your infrastructure.

---

Source: Deploy Local LLMs on Kubernetes: Complete vLLM + Helm Guid