K3S, split-horizon DNS, DNSSEC, and PowerDNS


On my home K3S cluster, I was running into a string of weird DNS issues. Here’s how I ended up fixing it.

Architecture Overview

  • 6-node K3S cluster (3x worker, 3x control plane)
  • In-cluster DNS using standard off-the-shelf CoreDNS
  • Intranet DNS provided by PowerDNS
    • 2 DNS servers
    • Each runs PDNS Authoritative (for internal zone example.com and reverse)
    • Each runs PDNS Recursor (for all other queries; forwards to auth for example.com)
  • Internet DNS for example.com provided by Cloudflare and DNSSEC-signed.
  • Note the split-horizon DNS using the same domain.


I was mostly having issues resolving hosts in example.com. For example, if a pod were making a query to look for foo.example.com, I would see in the PDNS logs queries for:

  • foo.example.com.svc.cluster.local
  • foo.example.com.cluster.local
  • foo.example.com.

The first two would obviously return NXDOMAIN, which is as expected. However, the last one returned an empty result but with 1 additional.

Red Herrings

Here’s a handful of things I tried that ultimately didn’t solve the problem (but may be useful avenues to explore if you happen to see similar issues).

  • DNS search suffixes and ndots. This didn’t matter; K3S does the right thing here.
  • systemd-resolved not playing nicely with K3S
    • K3S will take the host’s /etc/resolv.conf for use by pods (especially CoreDNS).
    • To work around this, the most complete way would be to do something like:
    rm /etc/resolv.conf
    cat <<EOF >/etc/resolv.conf
    search example.com
    and restart.
  • CoreDNS doing something weird
    • You can temporarily change its behavior by editing the coredns configmap in kube-system.
    • Things you might want include log to log all queries and replacing the forward line with forward .
    • Config reload is supposed to happen automatically but doesn’t; restart the pod(s) with kubectl rollout restart deployment coredns

Other things you might find useful include running an ad-hoc pod in-cluster on which you can install tools like dig, and using the +trace option to dig.


The problem ended up being because example.com was DNSSEC signed externally. For reasons I don’t fully understand, CoreDNS (and seemingly only CoreDNS) cared.

To fix it, I added the following to the PDNS Recursor Lua config located at /etc/powerdns/recursor.lua on my machine:

addNTA("example.com", "Some comment, doesn't matter")

and restarted the pdns-recursor service.

This added a Negative Trust Anchor for example.com such that clients would expect unsigned (or bogus-signed) responses for records in example.com. Make sure you add it on all of your DNS servers!

Enter your instance's address