How I Practise Istio Upgrades Locally Before Touching Production EKS.

Part 5 of 7 — The Mac Kubernetes Lab: A Production-Mirror Setup from Scratch.

Previously in Part 4: kubeadm 1.34 is bootstrapped, the M1 vs M4 CNI problem is solved (Calico on M4, Cilium on M1), and the three-node cluster is running with certificates signed by our Vault PKI. Now we install Istio and give the cluster real LoadBalancer IPs.

I’ve done an in-place Istio upgrade in production exactly once. A misconfigured istiod update broke sidecar injection across every namespace simultaneously. That was the last time.

The revision-based approach is the alternative. Run multiple versions of the Istio control plane in parallel. Install the new version alongside the old. Migrate namespaces one at a time. Verify the injection is working at each step. Then remove the old version when you’re confident. If anything misbehaves, you relabel the namespace back to the old revision. Full rollback at any point, no scrambling.
This is how I handle Istio upgrades on the production EKS clusters at work. Building it locally first means the production upgrade procedure is something I’ve already practised on a cluster where the stakes are zero.

Installing Istio via Helm (revision-based)

# 💻 Mac
kubectx lab-cluster

helm repo add istio https://istio-release.storage.googleapis.com/charts
helm repo update

# Step 1 - Base CRDs with default revision set to 1-26
helm install istio-base istio/base \
  --namespace istio-system --create-namespace \
  --set defaultRevision=1-26

# Step 2 - istiod with revision tag
# The revision label (1-26) is what namespaces reference for sidecar injection.
helm install istiod-1-26 istio/istiod \
  --namespace istio-system \
  --set revision=1-26 \
  --set global.proxy.resources.requests.cpu=50m \
  --set global.proxy.resources.requests.memory=128Mi \
  --set pilot.resources.requests.cpu=100m \
  --set pilot.resources.requests.memory=256Mi \
  --wait

# Step 3 - Ingress gateway tied to revision 1-26
helm install istio-ingress istio/gateway \
  --namespace istio-system \
  --set revision=1-26 \
  --set service.type=LoadBalancer

# Label the namespace for injection using the revision label.
# Note: use istio.io/rev=1-26, NOT istio-injection=enabled.
# The revision label tells the webhook which control plane to use.
kubectl label namespace default istio.io/rev=1-26

Verify istiod is running:

# 💻 Mac
kubectl get pods -n istio-system
# NAME                           READY   STATUS    AGE
# istiod-1-26-xxx                1/1     Running   60s
# istio-ingress-xxx              1/1     Running   45s

The MetalLB problem.

On the OrbStack-native cluster from Part 2, LoadBalancer services get real IPs automatically. On the VM cluster, there’s no cloud provider doing that work, services stay in pending state indefinitely.

# 💻 Mac
kubectl get svc istio-ingress -n istio-system
# NAME            TYPE           CLUSTER-IP   EXTERNAL-IP   PORT(S)
# istio-ingress   LoadBalancer   10.x.x.x     <pending>     80:xxx

MetalLB solves this. It implements the Kubernetes LoadBalancer spec for bare-metal and VM environments. It watches for LoadBalancer services and assigns IPs from a pool you define.

Installing MetalLB

# 💻 Mac
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.5/config/manifests/metallb-native.yaml
kubectl get pods -n metallb-system -w
# Wait for the controller and speaker pods to be Running

Configuring the IP pool.

The IP pool must be in the same subnet as your OrbStack VMs. Get the subnet first:

# 💻 Mac
orb run -m cp01 hostname -I
# M4: 192.168.139.200
# M1 with Cilium: 192.168.139.159 10.0.0.105 fd07:...
#                 ^^^^^ use this — the first 192.168.x.x IP

M1 quirk: With Cilium running, hostname -I returns multiple IPs: the OrbStack internal IP, a Cilium pod-network IP, and an IPv6 address. Always use the first 192.168.x.x IP. The others are internal to Cilium's networking and don't help you here.
Configure the pool using a range that doesn’t overlap with the VM IPs:

# 💻 Mac
kubectl apply -f - <<EOF
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: lab-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.139.200-192.168.139.250
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: lab-l2
  namespace: metallb-system
EOF

Watch the ingress gateway get its IP:

# 💻 Mac
kubectl get svc istio-ingress -n istio-system -w
# NAME            TYPE           EXTERNAL-IP       PORT(S)
# istio-ingress   LoadBalancer   192.168.139.200   80:xxx,443:xxx

Setting up Mac /etc/hosts.

Unlike the native cluster, the VM cluster doesn’t have wildcard DNS. We add entries to /etc/hosts manually:

# 💻 Mac
export INGRESS_IP=$(kubectl get svc istio-ingress -n istio-system \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo $INGRESS_IP

# Remove any stale entry from a previous attempt
sudo sed -i '' '/lab.local/d' /etc/hosts
echo "$INGRESS_IP  httpbin.lab.local bookinfo.lab.local api.lab.local" \
  | sudo tee -a /etc/hosts

Gateway + VirtualService on the VM cluster.

With MetalLB handing out real IPs, we can create Gateways and VirtualServices using *.lab.local hostnames.

Bookinfo the Istio reference app.

Bookinfo is Istio’s reference demo. It’s perfect for testing traffic management because it has multiple services with multiple versions of one of them.

# 💻 Mac
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.26/samples/bookinfo/platform/kube/bookinfo.yaml

kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: bookinfo-gateway
spec:
  selector:
    istio: ingress
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "bookinfo.lab.local"
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: bookinfo
spec:
  hosts:
  - "bookinfo.lab.local"
  gateways:
  - bookinfo-gateway
  http:
  - match:
    - uri:
        prefix: /productpage
    route:
    - destination:
        host: productpage
        port:
          number: 9080
EOF

Open http://bookinfo.lab.local/productpage in your Mac browser.

Traffic splitting.

Now we can practise the same canary deployment patterns used in production:

# 💻 Mac
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: reviews-dr
spec:
  host: reviews
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
  - name: v3
    labels:
      version: v3
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: reviews-canary
spec:
  hosts:
  - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v1
      weight: 80
    - destination:
        host: reviews
        subset: v2
      weight: 20
EOF

Refresh the bookinfo page a few times. 80% of requests go to reviews-v1 (no stars), 20% to reviews-v2 (black stars).

Header-based routing.

Route specific users to a different version based on a request header:

# 💻 Mac
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: reviews-header
spec:
  hosts:
  - reviews
  http:
  - match:
    - headers:
        end-user:
          exact: user
    route:
    - destination:
        host: reviews
        subset: v3
  - route:
    - destination:
        host: reviews
        subset: v1
EOF

Fault injection.

Introduce artificial latency and errors to test resilience:

# 💻 Mac
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: ratings-fault
spec:
  hosts:
  - ratings
  http:
  - fault:
      delay:
        percentage:
          value: 50.0
        fixedDelay: 3s
      abort:
        percentage:
          value: 10.0
        httpStatus: 500
    route:
    - destination:
        host: ratings
        port:
          number: 9080
EOF

50% of requests to ratings will get a 3-second delay. 10% will return 500. Useful for testing timeout and retry configurations on calling services.

Strict mTLS.

Enforce mutual TLS across the mesh:

# 💻 Mac
kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: default
spec:
  mtls:
    mode: STRICT
EOF

Practising an Istio upgrade.

This is where the revision-based setup actually pays off. Let’s simulate upgrading from 1.26 to 1.28:

# 💻 Mac
# Step 1 - Install the new control plane alongside the old
helm install istiod-1-28 istio/istiod \
  --namespace istio-system \
  --set revision=1-28 \
  --wait

# Step 2 - Migrate a namespace to the new revision
kubectl label namespace default istio.io/rev=1-28 --overwrite

# Step 3 - Restart pods to pick up the new sidecar version
kubectl rollout restart deployment -n default

# Step 4 - Verify new sidecars
kubectl get pods -n default -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.sidecar\.istio\.io/status}{"\n"}{end}'

# Step 5 - Once you're satisfied, remove the old control plane
helm uninstall istiod-1-26 -n istio-system

During Steps 1–3, both `istiod-1-26` and `istiod-1-28` are running simultaneously. If anything goes wrong after migrating a namespace, you can roll back by re-labelling the namespace back to `1-26`. This is exactly the procedure I use for production EKS Istio upgrades, and having practised it locally means there's no improvising on the day.

Where we are
The VM cluster now has:
✅ Istio 1.26 installed with revision label (istiod-1-26)
✅ Ingress gateway with a real LoadBalancer IP from MetalLB
✅ /etc/hosts on the Mac routing *.lab.local to the ingress IP
✅ Bookinfo deployed with traffic splitting, header routing, and fault injection
✅ Strict mTLS enforced across the default namespace
✅ Experience practising a revision-based Istio upgrade

In Part 6, we complete the EKS mirror: Vault Kubernetes auth, Crossplane with the AWS provider, and a LimitRange configuration that mirrors a production gap we hit at work, the one that caused a real disk-pressure incident.

← Part 4: Same Cluster, Different Mac: A Debugging Story About Unprivileged LXC Containers, iptables, and Why Cilium Replaces kube-proxy | Part 6: The Disk-Pressure Incident That Taught Me to Always Set LimitRanges — and Other Lessons from Mirroring EKS Locally →

I’m Noah Makau, a DevSecOps engineer based in Nairobi. I run a small DevOps consultancy and hold CKA, CKAD, and the AWS Solutions Architect Professional certifications, currently preparing for CKS. I write about Kubernetes, Vault, Crossplane, and the day-to-day of running platforms that actually have to stay up.

originally published at blog.arkilasystems.com