Introduction
One of the most common mistakes during incident response is assuming Kubernetes is the problem simply because the application runs on Kubernetes.
A customer reports an outage.
Requests start timing out.
Applications return errors.
Transactions begin failing.
The first reaction is usually:
AKS is down.
In many cases, AKS is completely healthy.
The nodes are healthy.
The pods are healthy.
The ingress controller is healthy.
The cluster is operating exactly as designed.
The actual problem exists somewhere outside Kubernetes.
After working through enough production incidents, I have learned that many application outages blamed on Kubernetes are eventually traced back to dependencies such as:
- Databases
- Redis
- DNS
- Key Vault
- Private Endpoints
- Firewalls
- Certificates
- API Gateways
- Third-party services
Understanding how to identify these situations quickly can save hours of troubleshooting and reduce unnecessary pressure on platform teams.
Start With Evidence, Not Assumptions
The first step during any incident is determining whether Kubernetes itself is actually unhealthy.
Start with:
kubectl get nodes
Healthy output:
STATUS
Ready
Ready
Ready
Next:
kubectl get pods -A
Look for:
Running
Completed
If the cluster is healthy and workloads are running, immediately expand the investigation.
Do not spend hours troubleshooting Kubernetes simply because the application is deployed there.
The Most Misleading Incident
A common scenario looks like this:
Customers Reporting Errors
↓
Pods Running
↓
CPU Normal
↓
Memory Normal
↓
No Recent Deployment
At this point many engineers become stuck.
Everything inside Kubernetes appears healthy.
The next question should be:
What does the application depend on?
That question often reveals the actual root cause.
Azure SQL: The Silent Failure
Databases are one of the most common causes of application incidents.
Symptoms often include:
500 Errors
Timeouts
Slow Responses
Connection Failures
Meanwhile:
Pods Healthy
Example application logs:
SqlException:
Execution Timeout Expired
or
Connection Timeout Expired
Many teams immediately begin investigating AKS.
The actual issue may be:
- Database maintenance
- Query blocking
- Connection exhaustion
- Network restrictions
- Failover events
Before touching Kubernetes, verify database connectivity.
Test directly:
kubectl exec -it pod-name -- sh
Then:
nc -zv database-server 1433
or
telnet database-server 1433
The objective is determining whether the pod can reach the database.
Redis Is Running but Applications Are Failing
Redis failures are particularly deceptive.
Applications may appear healthy while functionality quietly degrades.
Examples:
Authentication Delays
Session Failures
Cache Misses
Performance Issues
Application logs may show:
ECONNRESET
or
Connection Lost
Meanwhile:
kubectl get pods
shows:
Running
This is not a Kubernetes problem.
The application is unable to communicate with Redis.
Common causes include:
- Firewall restrictions
- DNS failures
- Connection limits
- Redis maintenance
- TLS misconfiguration
DNS: The Root Cause Nobody Expects
DNS issues are among the most frustrating incidents because they often affect healthy applications.
Consider:
Application
↓
api.internal.local
If DNS resolution fails:
Connection Failed
even though:
Application Healthy
and
AKS Healthy
Inside a pod:
nslookup api.internal.local
or
dig api.internal.local
can immediately expose the problem.
Common DNS-related causes include:
- Incorrect private zones
- Broken forwarding rules
- Missing records
- CoreDNS issues
- Private Endpoint resolution failures
DNS should always be part of the investigation process.
Private Endpoints and Private DNS
Private Endpoints introduce another layer of complexity.
Example architecture:
AKS
↓
Private Endpoint
↓
Database
The database exists.
The endpoint exists.
The application still cannot connect.
Why?
Because successful Private Endpoint connectivity requires:
Private Endpoint
+
Private DNS
+
Network Routing
Missing any one component can break connectivity.
Common symptoms:
Connection Timeout
Host Not Found
Name Resolution Failure
Testing from within a pod quickly reveals whether DNS or routing is involved.
Firewalls and Network Security Rules
Network controls frequently create incidents that appear to be application failures.
Example:
Application
↓
Database
A new firewall rule is introduced.
Traffic becomes blocked.
Application logs begin showing:
Timeout
Connection Refused
The pods remain healthy.
Kubernetes remains healthy.
The network path is not.
Whenever connectivity issues occur, verify:
Source
Destination
Port
Route
Firewall
before investigating Kubernetes internals.
Key Vault Dependency Failures
Modern applications often retrieve secrets from Key Vault during startup.
If Key Vault becomes inaccessible:
Application Startup Fails
Pod status:
CrashLoopBackOff
Many engineers immediately investigate:
Container Image
Deployment YAML
Node Health
The actual issue may be:
Key Vault Access Failure
Common causes:
- Expired permissions
- Managed Identity issues
- Workload Identity issues
- Firewall restrictions
- DNS resolution failures
Always verify:
kubectl logs
before assuming Kubernetes is responsible.
Certificate Chain Problems
Certificate issues frequently appear as networking problems.
Example:
HTTPS Endpoint
returns:
TLS Handshake Failed
Application logs:
unable to verify first certificate
or
unable to get local issuer certificate
The cluster is healthy.
The application is healthy.
The certificate chain is incomplete.
Testing:
openssl s_client \
-connect endpoint:443 \
-servername endpoint
can expose these problems immediately.
API Gateways and Reverse Proxies
Applications often sit behind:
- API Gateways
- Application Gateways
- Reverse Proxies
A backend service may be healthy while traffic fails before reaching it.
Example:
Client
↓
Gateway
↓
Application
Application logs:
No Requests Received
Backend health:
Healthy
The issue may exist entirely within:
Gateway Routing
Health Probes
TLS Configuration
Investigating only the application delays resolution.
Third-Party Service Dependencies
Some incidents originate completely outside your environment.
Examples:
Payment Gateway
SMS Provider
Identity Provider
Email Provider
Application logs:
503
429
Timeout
Kubernetes cannot fix an unavailable third-party service.
Understanding external dependencies is critical during incident response.
A Practical Investigation Workflow
Whenever an application outage occurs, I follow the same sequence.
Step 1
Verify cluster health.
kubectl get nodes
Step 2
Verify pod health.
kubectl get pods
Step 3
Verify ingress.
kubectl get ingress
Step 4
Review logs.
kubectl logs
Step 5
Identify dependencies.
Examples:
SQL
Redis
Key Vault
DNS
External APIs
Step 6
Test connectivity from inside the pod.
kubectl exec
Step 7
Verify DNS resolution.
nslookup
Step 8
Verify TLS connectivity.
openssl s_client
Step 9
Validate network controls.
Firewall
NSG
Route Table
Private Endpoint
Step 10
Only investigate Kubernetes internals if evidence points there.
Common Mistakes During Incident Response
Mistake 1
Assuming Kubernetes is guilty because the application runs on Kubernetes.
Mistake 2
Ignoring application logs.
Mistake 3
Skipping connectivity tests.
Mistake 4
Never checking DNS.
Mistake 5
Ignoring certificate validation errors.
Mistake 6
Treating timeouts as Kubernetes issues.
Most timeouts originate elsewhere.
The Dependency Mindset
One useful mental model is:
Application
↓
Dependencies
The application is only as healthy as the systems it depends on.
A healthy AKS cluster cannot compensate for:
- Unreachable databases
- Broken DNS
- Invalid certificates
- Blocked firewalls
- Unavailable third-party APIs
When troubleshooting production incidents, dependencies deserve as much attention as Kubernetes itself.
Conclusion
Some of the longest outages I have investigated had nothing to do with Kubernetes.
The cluster was healthy.
The nodes were healthy.
The pods were healthy.
The actual issue was hidden behind a dependency that nobody initially considered.
Successful troubleshooting starts by eliminating assumptions.
Verify the cluster.
Verify the application.
Then systematically validate every dependency the application relies on.
Kubernetes is often the first component blamed during an outage because it is highly visible.
In many cases, it is also completely innocent.
The faster teams learn to distinguish between platform failures and dependency failures, the faster they resolve incidents and the more effective their troubleshooting process becomes.
