An alert that fires too often stops being an alert. When on-call engineers are conditioned to dismiss notifications because most of them resolve on their own, the one real incident that demands immediate attention gets lost in the noise. This is alert fatigue, one of the most common and costly problems in production engineering.
The antidote is not fewer alerts. It's better alerts, thresholds designed around the behavior of your actual system, routed to the right people, at the right time, with enough context to act immediately.
Grafana Alerting provides a flexible, powerful platform for building exactly this. But the tooling is only as good as the threshold design behind it. This guide covers how to think about alert thresholds, how to implement them correctly in Grafana, and how to build an alerting strategy that your on-call team will trust, not mute.
Why Most Alerting Systems Fail
Before designing better thresholds, it helps to understand the failure modes of typical alerting setups:
Static thresholds on dynamic systems. Setting CPU > 80% as a hard alert ignores the fact that your system's normal baseline changes across time of day, day of week, and traffic patterns. An 80% CPU reading at 3am during off-peak hours is a crisis. The same reading at noon on Black Friday is expected.
Alerting on symptoms instead of impact. High memory usage, elevated CPU, and increased error rates are symptoms. The question your alert should answer is: are users being impacted right now? An alert on HTTP 500 rate > 1% is more actionable than an alert on heap memory > 70%.
Missing evaluation windows. A single spike that recovers in 10 seconds should not page anyone. Alerts that fire on instantaneous values rather than sustained conditions generate constant noise for transient anomalies.
No severity differentiation. Treating every alert with the same urgency trains engineers to treat all alerts as low urgency. A P1 production outage and a P4 warning about disk space should trigger completely different responses.
The Four Properties of a Meaningful Alert
Every alert in your system should satisfy all four of these properties before it pages anyone:
Actionable - the person receiving the alert knows what to do. If the response to an alert is "check the dashboard and see if it's a real problem," the alert is not actionable. Every alert should link directly to a runbook.
Accurate - the alert fires when something is genuinely wrong, not when the system is behaving within acceptable variance. False positives are as damaging as missing real incidents.
Timely - the alert fires early enough to allow intervention before users are significantly impacted, but not so sensitive that it fires on noise.
Contextual - the alert carries enough information to begin diagnosis without opening five dashboards. Service name, environment, current value, threshold, and a runbook link should be included in every notification.
Step 1 - Establish Baselines Before Setting Thresholds
The most common mistake is setting thresholds without data. Before writing a single alert rule, spend time understanding your system's normal behavior:
- What is the p50, p95, and p99 latency for your critical endpoints on a normal day?
- What does error rate look like across different times of day and days of week?
- What is the normal range for CPU, memory, and connection pool utilization under typical load?
In Grafana, use the Explore view with a 30-day time range to visualize historical metric distributions. Look for:
# p95 latency over the last 30 days, understand your normal range
histogram_quantile(0.95,
sum by (le, service) (
rate(http_request_duration_seconds_bucket[5m])
)
)
If your p95 latency normally fluctuates between 120ms and 280ms, an alert threshold of > 500ms sustained for 5 minutes is meaningful. A threshold of > 150ms will fire constantly during normal operation.
Step 2 - Use Rate-Based Thresholds, Not Absolute Values
Absolute value thresholds (error count > 50) are fragile, they don't account for traffic volume. A system handling 10,000 requests per minute with 50 errors (0.05% error rate) is healthy. A system handling 100 requests per minute with 50 errors (50% error rate) is on fire.
Always alert on rates and ratios, not raw counts:
# Error rate as a percentage of total requests, traffic-aware
(
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
) * 100
This expression returns the percentage of requests resulting in 5xx errors over the last 5 minutes, meaningful regardless of whether your system is handling 100 or 100,000 requests per minute.
Similarly for latency, alert on percentiles rather than averages:
# p99 latency, catches tail latency issues that averages hide
histogram_quantile(0.99,
sum by (le) (
rate(http_request_duration_seconds_bucket{service="orders-service"}[5m])
)
)
Averages mask outliers. A p99 of 8 seconds with an average of 200ms means 1% of your users are waiting 8 seconds, and the average would never alert on it.
Step 3 - Configure Evaluation Windows and Pending Periods
Grafana Alerting evaluates rules on a configurable interval and supports a pending period, the duration a condition must be continuously true before the alert fires. This single setting eliminates the majority of false positives from transient spikes:
# Grafana Alert Rule configuration
evaluateEvery: 1m # evaluate the query every minute
for: 5m # condition must hold for 5 consecutive minutes before firing
Guidelines for pending periods by alert severity:
| Severity | Pending Period | Rationale |
|----------|---------------|-----------|
| P1 - Critical | 1–2 minutes | Production down, act fast, accept occasional false positives |
| P2 - High | 3–5 minutes | Significant degradation, needs confirmation before paging |
| P3 - Medium | 10–15 minutes | Trend-based concern, sustained issue, not a spike |
| P4 - Low | 30–60 minutes | Capacity planning signal, no urgency |
A 5-minute pending period on a P2 alert means the condition must be true for five consecutive 1-minute evaluations. A 30-second CPU spike at minute 3 resets the clock, preventing the alert from firing on transient load.
Step 4 - Implement Multi-Condition Alerts
Some failure modes only become meaningful when multiple conditions are true simultaneously. Grafana supports multi-condition alert rules that combine signals for higher-precision alerting:
# Condition A - error rate elevated
(
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
) > 0.02
# Condition B - latency degraded simultaneously
histogram_quantile(0.95,
sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
) > 1.0
Requiring both conditions to be true simultaneously, elevated error rate AND elevated latency, dramatically reduces false positives compared to alerting on either signal alone. A brief error rate spike during a deployment might not coincide with latency degradation; genuine service degradation almost always shows both.
Step 5 - Define Alert Severity and Routing
Not all alerts should wake someone up at 3am. Define a clear severity taxonomy and route accordingly:
# Grafana Alerting - labels for routing
labels:
severity: critical # critical | high | medium | low
team: backend # backend | frontend | platform | data
service: orders-service
environment: production
Map severities to notification channels in your Contact Points and Notification Policies:
| Severity | Notification Channel | Response Expectation |
|---|---|---|
| Critical | PagerDuty (immediate page) | Respond within 5 minutes |
| High | PagerDuty (page after 10 min) | Respond within 30 minutes |
| Medium | Slack #alerts-medium
|
Review within business hours |
| Low | Slack #alerts-low
|
Review in weekly ops meeting |
Grafana's Notification Policy tree allows routing based on label matchers, so severity=critical AND environment=production pages on-call, while severity=low posts silently to a Slack channel that gets reviewed on Monday.
Step 6 - Write Alerts That Explain Themselves
An alert notification that says "High Error Rate" forces the on-call engineer to open dashboards before they can even begin to understand the situation. Grafana's alert annotations and templated messages fix this:
{{ define "alert-summary" }}
{{ .Labels.severity | toUpper }} — {{ .Labels.service }}
Environment: {{ .Labels.environment }}
Condition: Error rate has exceeded 2% for 5 consecutive minutes
Current Value: {{ $value | printf "%.2f" }}%
Threshold: 2.00%
Service: {{ .Labels.service }}
Started: {{ .StartsAt | date "2006-01-02 15:04:05 UTC" }}
Runbook: https://runbooks.internal/{{ .Labels.service }}/high-error-rate
Dashboard: https://grafana.internal/d/service-overview?var-service={{ .Labels.service }}
{{ end }}
Every alert notification should answer:
- What is wrong (metric name and current value)
- How wrong it is (comparison to threshold)
- How long it has been wrong
- Where to look next (dashboard link)
- What to do (runbook link)
Step 7 - Maintain Alerts as Code with Grafana Provisioning
Alerts defined manually in the Grafana UI are fragile, they live only in the database, can't be code-reviewed, and can't be rolled back. Use Grafana's provisioning system to manage alerts as version-controlled YAML:
# provisioning/alerting/orders-service.yaml
apiVersion: 1
groups:
- name: orders-service
folder: Backend Services
interval: 1m
rules:
- uid: orders-error-rate-p2
title: "orders-service — High Error Rate"
condition: C
for: 5m
labels:
severity: high
team: backend
service: orders-service
environment: production
annotations:
summary: "Error rate above 2% for 5 minutes"
runbook_url: "https://runbooks.internal/orders-service/high-error-rate"
data:
- refId: A
datasourceUid: prometheus
model:
expr: |
(
sum(rate(http_requests_total{service="orders-service",status=~"5.."}[5m]))
/
sum(rate(http_requests_total{service="orders-service"}[5m]))
) * 100
- refId: C
datasourceUid: "__expr__"
model:
type: threshold
conditions:
- evaluator:
type: gt
params: [2.0]
query:
params: [A]
Store alert provisioning files in your infrastructure repository, apply them via Helm or Terraform, and treat every alert change as a pull request requiring review.
Step 8 - Audit and Prune Regularly
An alerting system accumulates debt over time. Schedule a monthly or quarterly alert audit:
- Identify never-firing alerts. An alert that hasn't fired in 90 days is either covering a scenario that never happens or has a threshold set so high it would never catch real issues.
- Identify always-firing alerts. Any alert with a sustained firing rate above 20% is background noise, raise the threshold or add a pending period.
- Review resolved-without-action alerts. If alerts are regularly acknowledged and resolved without any remediation, the threshold is too sensitive.
- Retire obsolete alerts. Services get deprecated, features get removed. Orphaned alert rules are noise that erodes trust in the system.
Track alert quality metrics, false positive rate, mean time to acknowledgement, and alert-to-incident conversion rate, and treat them as engineering KPIs alongside uptime and latency.
Conclusion
Alerting is not a set-and-forget configuration, it's an ongoing engineering discipline. The difference between an alerting system your team trusts and one they mute is threshold design: understanding your system's baseline, alerting on rates rather than absolutes, requiring sustained conditions before paging, differentiating severity, and ensuring every notification carries enough context to act immediately.
Start with your three most critical services. Establish baselines from 30 days of historical data, set rate-based thresholds with appropriate pending periods, define severity routing, and write runbooks before the alerts go live. An alert without a runbook is a fire alarm without an evacuation plan.
Build alert quality into your engineering culture, review and prune regularly, track false positive rates, and treat every unnecessary 3am page as a bug worth fixing.
