A single A10 GPU on OCI costs $1.52/hr. Running 24/7, that's $1,094/month. For a production inference service with steady traffic, that's fine. But I had a staging environment and a couple of internal tools that got maybe 20 requests per day. I was paying over $2,000/month for GPUs that sat idle 95% of the time.
The obvious solution: scale to zero when there's no traffic, spin up when a request comes in. KEDA does this on Kubernetes, but getting it to work properly with GPU pods took some figuring out.
Why Scaling GPUs Is Harder Than Scaling CPU Pods
With normal HTTP services, KEDA watches a metric (HTTP requests, queue depth, whatever), and Kubernetes can spin up a new pod in seconds. The user barely notices.
GPU pods are different:
- Cold start is slow - Model loading takes 60-120 seconds
- Node scaling is slow - If there's no GPU node in the pool, OKE needs to provision a new VM (3-5 minutes)
- Images are huge - 5-15GB pull times if not cached
- GPU resources are binary - You can't give a pod "half a GPU" (without MIG)
So you can't just scale-to-zero and expect sub-second response times when traffic returns. The trade-off is cost savings vs. cold start latency. For my use case (internal tools, staging), a 2-3 minute cold start was acceptable.
The Setup
1. Install KEDA on OKE
helm repo add kedacore https://kedacore.github.io/charts
helm install keda kedacore/keda \
--namespace keda-system \
--create-namespace
2. Prometheus for Request Metrics
I'm using the nginx ingress controller's Prometheus metrics to track request rate. If you're using OCI's native load balancer, you'd use OCI Monitoring metrics instead.
# prometheus-scaledobject.yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: vllm-scaler
namespace: inference
spec:
scaleTargetRef:
name: vllm-inference
minReplicaCount: 0 # scale to zero
maxReplicaCount: 3
cooldownPeriod: 300 # wait 5 min of no traffic before scaling down
pollingInterval: 15
triggers:
- type: prometheus
metadata:
serverAddress: http://prometheus.monitoring:9090
metricName: http_requests_total
query: |
sum(rate(nginx_ingress_controller_requests{
namespace="inference",
service="vllm-inference"
}[2m]))
threshold: "1" # scale up if >1 req/sec averaged over 2 min
activationThreshold: "0.1" # activate from zero if any traffic
The key settings:
-
minReplicaCount: 0— this is what enables scale-to-zero -
cooldownPeriod: 300— 5 minutes of no traffic before scaling down (prevents flapping) -
activationThreshold: "0.1"— even a trickle of traffic triggers scale-up from zero
3. Handling the Cold Start
When the pod scales from zero, there's a gap. The request that triggered the scale-up needs to wait for the pod to be ready. I handle this with a simple queue pattern:
# queue-proxy.yaml — lightweight proxy that holds requests during cold start
apiVersion: apps/v1
kind: Deployment
metadata:
name: inference-proxy
namespace: inference
spec:
replicas: 1 # always running, tiny resource footprint
template:
spec:
containers:
- name: proxy
image: iad.ocir.io/mytenancy/inference-proxy:v1
ports:
- containerPort: 8080
env:
- name: BACKEND_URL
value: "http://vllm-inference:8000"
- name: TIMEOUT_SECONDS
value: "180" # wait up to 3 min for backend
resources:
requests:
cpu: 50m
memory: 64Mi
The proxy is a tiny Go service (always running, costs almost nothing) that:
- Accepts incoming requests immediately
- Forwards them to the vLLM backend
- If the backend isn't ready, retries with exponential backoff up to 3 minutes
- Returns a 503 with "model loading, please retry" if it times out
func proxyHandler(w http.ResponseWriter, r *http.Request) {
backendURL := os.Getenv("BACKEND_URL")
timeout, _ := strconv.Atoi(os.Getenv("TIMEOUT_SECONDS"))
deadline := time.Now().Add(time.Duration(timeout) * time.Second)
backoff := 2 * time.Second
for time.Now().Before(deadline) {
resp, err := http.DefaultClient.Do(cloneRequest(r, backendURL))
if err == nil {
copyResponse(w, resp)
return
}
time.Sleep(backoff)
backoff = min(backoff*2, 15*time.Second)
}
http.Error(w, "inference backend unavailable, try again shortly", 503)
}
4. Keeping a Warm GPU Node
The slowest part of cold start isn't model loading — it's waiting for OKE to provision a GPU node when none exist. This takes 3-5 minutes.
My workaround: keep one GPU node always available, but let the inference pods on it scale to zero. The node costs money even when idle, but it's a single node vs. multiple. And when traffic comes in, the pod starts in ~90 seconds (model loading) instead of 5+ minutes (node provisioning + model loading).
# GPU node pool with min 1 node (always warm)
oci ce node-pool update \
--node-pool-id $GPU_NODE_POOL_ID \
--node-config-details '{
"size": 1,
"placementConfigs": [...]
}'
For staging environments where the 5-minute cold start is acceptable, I set the node pool to autoscale from 0 to 2 nodes and let OKE handle it.
The Savings
My three GPU workloads (staging vLLM, internal summarizer, internal code review tool) were running 24/7 on three A10 instances:
| Before | After |
|---|---|
| 3x A10 always-on | 1x A10 warm node + scale-to-zero pods |
| $3,282/month | ~$1,094/month (warm node) + ~$50 (burst usage) |
| $3,282/month | ~$1,144/month |
65% savings. The internal tools scale up when someone uses them (a few times a day) and scale back down after 5 minutes of idle. The warm node means cold starts are 90 seconds, which is fine for internal users.
When Not to Do This
- Customer-facing APIs - 90-second cold starts are unacceptable. Keep replicas warm.
- Steady traffic - If your GPU is busy >60% of the time, scale-to-zero doesn't save enough to justify the complexity.
- Real-time applications - Anything latency-sensitive should stay always-on.
This works for internal tools, batch endpoints, staging environments, and anything where "please wait a moment" is an okay response.
Pavan Madduri — Oracle ACE Associate, CNCF Golden Kubestronaut. I'm also building keda-gpu-scaler for GPU-aware autoscaling. GitHub | LinkedIn | Website | Google Scholar | ResearchGate













