I ran into something dumb last week. Caddy's certificate renewal kept failing silently, and it took longer than I'd like to admit to figure out the culprit was systemd-resolved.
What happened
Caddy uses ACME challenges to renew certificates. The process involves a DNS query from your server to Let's Encrypt — nothing unusual. Except mine was returning SERVFAIL for the specific TXT record Caddy needed, while every other query worked fine.
The catch: systemd-resolved has a stub resolver behavior where it selectively returns errors for certain record types or domains depending on how your /etc/resolv.conf is configured. In my case, it was filtering outbound queries for _acme-challenge.example.com silently.
How I found it
Running resolvectl query _acme-challenge.example.com showed SERVFAIL, while dig @8.8.8.8 _acme-challenge.example.com TXT returned the correct record immediately. The stub resolver was the problem, not the network or Caddy.
The fix
Temporarily bypass the stub resolver for renewals. Edit /etc/resolv.conf and replace 127.0.0.53 with 8.8.8.8, or point Caddy at an upstream resolver directly:
{
email "your@example.com"
acme_ca "https://acme-v02.api.letsencrypt.org/directory"
resolver "8.8.8.8"
}
The lesson
systemd-resolved is fine until it isn't. When something works manually but fails in automation, the local resolver is worth checking. The kind of thing that only surfaces as a renewal failure when nobody's watching.
