Last month I needed to stand up a Llama 3 inference endpoint for an internal tool. The requirements were simple: OpenAI-compatible API, auto-scaling, and it couldn't cost more than the team's coffee budget. AWS wanted $3.06/hr for a g5.xlarge. Azure quoted something similar.
Then I looked at OCI's GPU shapes. VM.GPU.A10.1 — a single NVIDIA A10 with 24GB VRAM — at $1.52/hr on-demand. Half the price. And on preemptible? $0.46/hr. That's a latte.
Here's how I got vLLM running on OKE in about 20 minutes.
The OKE Cluster Setup
If you already have an OKE cluster, skip ahead. If not, this is the fastest path:
# Create a VCN (or use an existing one)
oci network vcn create \
--compartment-id $COMPARTMENT_ID \
--cidr-blocks '["10.0.0.0/16"]' \
--display-name "ai-inference-vcn"
# Create the OKE cluster
oci ce cluster create \
--compartment-id $COMPARTMENT_ID \
--name "inference-cluster" \
--vcn-id $VCN_ID \
--kubernetes-version "v1.30.1" \
--service-lb-subnet-ids "[$PUBLIC_SUBNET_ID]"
The key part is the GPU node pool. OCI has several GPU shapes, but for inference the A10 is the sweet spot:
| Shape | GPU | VRAM | $/hr (on-demand) | $/hr (preemptible) |
|---|---|---|---|---|
| VM.GPU.A10.1 | 1x A10 | 24 GB | ~$1.52 | ~$0.46 |
| VM.GPU.A10.2 | 2x A10 | 48 GB | ~$3.04 | ~$0.91 |
| BM.GPU.A100-v2.8 | 8x A100 | 640 GB | ~$26.52 | N/A |
For a 7B parameter model, a single A10 is plenty. For 70B, you'd want 2xA10 or the A100 bare metal.
# Create the GPU node pool
oci ce node-pool create \
--cluster-id $CLUSTER_ID \
--compartment-id $COMPARTMENT_ID \
--name "gpu-a10-pool" \
--node-shape "VM.GPU.A10.1" \
--size 1 \
--node-config-details \
'{"size": 1, "placementConfigs": [{"availabilityDomain": "'"$AD"'", "subnetId": "'"$WORKER_SUBNET_ID"'"}]}' \
--node-source-details \
'{"sourceType": "IMAGE", "imageId": "'"$GPU_IMAGE_ID"'"}'
Make sure you use the OKE GPU image — it comes with NVIDIA drivers and nvidia-container-toolkit pre-installed. You don't want to deal with driver installation yourself. Trust me.
The NVIDIA Device Plugin
OKE's GPU images already include the drivers, but Kubernetes needs the device plugin to expose GPUs as a schedulable resource:
# nvidia-device-plugin.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: nvidia-device-plugin-daemonset
namespace: kube-system
spec:
selector:
matchLabels:
name: nvidia-device-plugin-ds
template:
metadata:
labels:
name: nvidia-device-plugin-ds
spec:
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
containers:
- image: nvcr.io/nvidia/k8s-device-plugin:v0.16.1
name: nvidia-device-plugin-ctr
env:
- name: FAIL_ON_INIT_ERROR
value: "false"
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
volumeMounts:
- name: device-plugin
mountPath: /var/lib/kubelet/device-plugins
volumes:
- name: device-plugin
hostPath:
path: /var/lib/kubelet/device-plugins
kubectl apply -f nvidia-device-plugin.yaml
Verify GPUs show up:
kubectl get nodes -o json | jq '.items[].status.capacity["nvidia.com/gpu"]'
# "1"
If that says "1", you're golden.
Deploying vLLM
vLLM's Docker image is the easiest way to run it. No pip installs, no dependency conflicts, no wondering why PyTorch can't find CUDA.
# vllm-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-llama3
labels:
app: vllm-inference
spec:
replicas: 1
selector:
matchLabels:
app: vllm-inference
template:
metadata:
labels:
app: vllm-inference
spec:
containers:
- name: vllm
image: vllm/vllm-openai:v0.6.4
args:
- "--model"
- "meta-llama/Llama-3.1-8B-Instruct"
- "--max-model-len"
- "4096"
- "--gpu-memory-utilization"
- "0.90"
- "--dtype"
- "auto"
ports:
- containerPort: 8000
name: http
resources:
limits:
nvidia.com/gpu: 1
requests:
nvidia.com/gpu: 1
memory: "24Gi"
cpu: "4"
env:
- name: HUGGING_FACE_HUB_TOKEN
valueFrom:
secretKeyRef:
name: hf-token
key: token
readinessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 120
periodSeconds: 10
livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 180
periodSeconds: 30
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
---
apiVersion: v1
kind: Service
metadata:
name: vllm-service
spec:
selector:
app: vllm-inference
ports:
- port: 8000
targetPort: 8000
type: ClusterIP
Create the HuggingFace token secret first:
kubectl create secret generic hf-token \
--from-literal=token=$HF_TOKEN
Then deploy:
kubectl apply -f vllm-deployment.yaml
The model download takes a few minutes depending on the model size. Watch the logs:
kubectl logs -f deployment/vllm-llama3
You'll see it load the model weights, compile the CUDA kernels, and eventually:
INFO: Uvicorn running on http://0.0.0.0:8000
Testing It
Port-forward and hit it with curl:
kubectl port-forward svc/vllm-service 8000:8000
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"messages": [{"role": "user", "content": "Explain Kubernetes in one sentence"}],
"max_tokens": 100
}'
The API is OpenAI-compatible. Your existing code that talks to gpt-4 just needs a base URL change.
What I Learned
A few things that bit me:
Model download speed — OKE nodes have good bandwidth to the internet, but the first pull of a 16GB model takes time. I ended up baking the model into a custom Docker image so pod restarts don't re-download. That's a separate blog post.
Memory headroom — gpu-memory-utilization: 0.90 leaves 10% for KV cache overhead. Don't set this to 0.99 thinking you're being efficient. vLLM will OOM during burst traffic.
Readiness probe timing — initialDelaySeconds: 120 seems high, but model loading legitimately takes 60-90 seconds on an A10. If your probe fires too early, Kubernetes will restart the pod in a loop.
Preemptible instances — At $0.46/hr they're incredible for dev/staging. For production, use on-demand and set up a second preemptible pool as overflow. I'll cover that in a future post about cost optimization.
Cost Comparison
Running Llama 3.1 8B on different clouds (single GPU, on-demand):
| Cloud | Shape | $/hr | $/month (24/7) |
|---|---|---|---|
| OCI | VM.GPU.A10.1 | $1.52 | ~$1,094 |
| AWS | g5.xlarge | $3.06 | ~$2,203 |
| Azure | NC24ads_A100_v4 | $3.67 | ~$2,642 |
| GCP | g2-standard-8 | $2.86 | ~$2,059 |
OCI is roughly half the price for equivalent hardware. And the preemptible pricing makes it even more dramatic for non-production workloads.
What's Next
This is the simplest possible setup — one model, one GPU, one replica. In the next posts I'll cover:
- Cost optimization with preemptible GPU pools and scale-to-zero
- Multi-model serving with vLLM's LoRA adapter support
- Monitoring GPU utilization with OpenTelemetry on OKE
The full YAML files are on my GitHub. If you're running inference on OCI, I'd love to hear what shapes you're using.
Pavan Madduri — CNCF Golden Kubestronaut, building GPU/AI infrastructure tools. GitHub | LinkedIn | Website | Google Scholar | ResearchGate










