K3S, split-horizon DNS, DNSSEC, and PowerDNS

On my home K3S cluster, I was running into a string of weird DNS issues. Here’s how I ended up fixing it.

Architecture Overview


I was mostly having issues resolving hosts in example.com. For example, if a pod were making a query to look for foo.example.com, I would see in the PDNS logs queries for:

The first two would obviously return NXDOMAIN, which is as expected. However, the last one returned an empty result but with 1 additional.

Red Herrings

Here’s a handful of things I tried that ultimately didn’t solve the problem (but may be useful avenues to explore if you happen to see similar issues).

Other things you might find useful include running an ad-hoc pod in-cluster on which you can install tools like dig, and using the +trace option to dig.


The problem ended up being because example.com was DNSSEC signed externally. For reasons I don’t fully understand, CoreDNS (and seemingly only CoreDNS) cared.

To fix it, I added the following to the PDNS Recursor Lua config located at /etc/powerdns/recursor.lua on my machine:

addNTA("example.com", "Some comment, doesn't matter")

and restarted the pdns-recursor service.

This added a Negative Trust Anchor for example.com such that clients would expect unsigned (or bogus-signed) responses for records in example.com. Make sure you add it on all of your DNS servers!