I was looking at your public stats and noticed the cache hit ratio sits around ~19.8% (2.32K hits vs 9.40K misses). To be fair, for a public resolver serving a huge diversity of clients and domains, a lower hit rate than a typical ISP/home resolver isn’t unusual — long-tail unique queries and low-TTL CDN records naturally pull the average down. So this isn’t necessarily something „broken,“ more like a potential area for marginal gains if you’re interested.
As your infrastructure is exclusively in Germany, and as you have stated that you do not utilise ECS a couple of the usual caveats around aggressive caching (breaking geo-routing accuracy) matter less in your case — your resolver already returns CDN answers optimized for the resolver’s location rather than the end user’s, so bumping TTLs a bit shouldn’t make that any worse than it already is. The main thing to watch would be domains that rely on very fast DNS-based failover/load balancing, where a longer minimum TTL could delay picking up a change.
With that context, a few things that might be worth experimenting with, depending on your resolver software:
Cache-min-ttl — even a modest floor (e.g. 60–120s) instead of the CDN-provided 15-30s could meaningfully cut repeat upstream queries, with fairly low risk given the no-ECS setup.
Serve stale (RFC 8767) — if supported, this won’t change the hit-rate number itself, but it can mask the latency impact of a miss by serving a slightly expired record while refreshing in the background.
Prefetch — refreshing hot records just before TTL expiry, low-risk and usually a straightforward win if the software supports it.
Cache size — just curious whether the allocated cache memory might be causing early eviction of popular records, but that’s more a question than a suggestion — you’d know better than me based on your actual load.
No pressure at all here — just sharing some ideas in case they’re useful. Really appreciate the work you put into the service either way!
thanks a lot for this, really thoughtful stuff and it landed exactly as you meant it. We went through all four points and you were pretty much on the money.
The minimum-TTL angle we already had covered, since we floor those short CDN TTLs at the filtering layer. Serve-stale and prefetch weren’t on though, so we’ve enabled both now: a miss on an expiring record gets answered straight away from a slightly stale copy while it refreshes in the background, and popular records get refreshed just before they expire.
The cache size was the big one, and you nailed it. Our record cache was sitting completely full with a hit ratio around a third, which is textbook early eviction. We’ve bumped it up significantly (scaled to each server’s memory) and enlarged the filtering-layer cache too, which was smaller than it should have been. Nice side effect is that better hit ratios mean fewer upstream lookups, so it takes some load off the resolvers as well.
We’ll keep an eye on the numbers over the next few days. Thanks again, genuinely useful input!
Thanks for actually taking the time to dig into it, not every project would bother. Appreciate the transparency with the public stats too, makes this kind of feedback loop possible in the first place.
One follow-up thought though, since I noticed something in the newer stats that could undo part of that gain over time if left alone.
The NXDOMAIN spikes look pretty persistent rather than one-off, and if that traffic is mostly random-subdomain style queries (a different garbage name every time), it’s not something the bigger cache alone fixes long-term — each of those is a guaranteed miss and creates its own one-off entry in the cache. So even with more headroom, a sustained flood like that could gradually refill the cache with junk and slowly push the hit ratio back down to where it was before, just on a longer timescale. Figured it’s worth flagging before it becomes a „why did the ratio drop again“.
Two things that might help cut this off at the source rather than just giving it more room to live in:
IPv6 ban scope — noticed the dynamic block list entries show a /128 suffix, meaning bans are per single address. Since most ISPs delegate a /64 per subscriber, a source can just rotate within their own allocation and dodge that pretty easily, which would explain why it never really dies down. Banning at /64 instead should be safe collateral-wise (that’s normally one subscriber anyway) and should actually stick.
NXDOMAIN flood mitigation at the query level, not just the IP level — for IPv4, a lot of the offending sources are big consumer ISPs (Telekom, Vodafone, etc.) that commonly use CGNAT, so IP-based bans there risk catching innocent users sharing the same address. Since this traffic is almost certainly random-subdomain (water-torture style) queries, it might be worth attacking the pattern directly instead: if the target zones are DNSSEC-signed, enabling aggressive NSEC/NSEC3 caching (RFC 8198) — aggressive-nsec: yes in Unbound, or the PowerDNS Recursor equivalent — lets the resolver synthesize NXDOMAIN answers for any number of random garbage subdomains from a single cached NSEC range, with no upstream query and no new cache entry per name. That kills both the upstream load and the cache-pollution problem at once, and doesn’t care who’s asking, so the CGNAT issue becomes irrelevant for this specific case.
For zones that aren’t signed and this doesn’t apply, capping or shortening TTL specifically for the negative/NXDOMAIN portion of the cache (separate from positive records) could stop one-off garbage names from slowly eating back into the space you just freed up.
spot on again, and thanks for keeping an eye on the stats with us. You’ve actually put your finger on exactly what we’re seeing on our end: the larger record cache fills right back up within a few hours and the hit ratio plateaus, because a big chunk of that traffic is random-subdomain junk that’s only ever queried once. Classic water-torture pattern, and it lines up perfectly with what you noticed.
On the IPv6 point you’re right that per-/128 blocking is mostly cosmetic when a subscriber has a whole /64 to rotate through. We can widen the dynamic blocking to aggregate on /64, which is safe here since a residential /64 maps to a single line and we’re not risking the CGNAT collateral you’d get on IPv4.
The aggressive NSEC/NSEC3 caching (RFC 8198) is a good call too. On our resolver version that’s typically on by default already, so we’ll confirm it’s actually active and sized sensibly, and it’ll absorb a lot of the NXDOMAIN floods for signed zones without touching upstreams. For the unsigned targets, where 8198 can’t help, we can tighten the negative-cache TTL so that one-shot junk expires out faster instead of squatting in the cache.
Really appreciate you thinking this through with us. We’ll try a couple of these and see how the curve behaves.
That’s great to hear! It’s good to know that it wasn’t just a hunch, but actually matches what you’re seeing internally.
I’ll keep half an eye on the dashboard, and if anything interesting shows up over the next while, I’ll drop a note. Thanks for taking the time to dig through all this. I appreciate it.
Did a bit more digging on those recurring /64s and it lines up nicely with the hosting-provider theory:
2a0a:4cc0:c1:35bf::/64
2a0a:4cc0:c0:a29f::/64
2a03:4000:6:3aaf::/64
All three resolve back to netcup GmbH — a German VPS/dedicated server hosting provider, not a residential ISP. That confirms why banning individual /64s keeps whack-a-moling: spinning up a new VPS (and getting a fresh /64) at a hosting provider costs the attacker almost nothing, so the source just keeps rotating within netcup’s larger allocation.
Since this is a hosting/datacenter network rather than residential, the CGNAT caution from before doesn’t really apply here — genuine dnsforge users most likely never sit inside a VPS provider’s own address space as their „home connection.“ So it might be worth treating datacenter/hosting ASNs differently from residential ones:
Ban at the provider’s actual announced block level (e.g. their /29 or /32) instead of per-/64, once a pattern like this is confirmed
Apply a lower rate-limit threshold by default for known datacenter/hosting ASNs, since legitimate traffic volume from there should be minimal anyway
Maybe keep a small running list of ASNs (via Team Cymru/MaxMind ASN lookup) tagged as „hosting“ vs „residential“ so the existing dynamic system can apply the right policy automatically instead of treating every /64 the same way
Two smaller things I noticed while looking at this, in case they’re useful:
One row in the block list shows the „Server“ column as 10.10.0.0 instead of a public IP like the other entries (176.9.93.198, 176.9.1.117). Since that looks like a private/internal address, wondering if that’s just a display quirk, or if it’s worth double-checking that this resolver instance isn’t unexpectedly reachable from outside where it shouldn’t be.
On the IPv4 side, 161.97.0.0 and 75.119.0.0 seem to sustain ~500 hits/sec pretty much around the clock, 24/7, rather than bursting and dying down. That steady, non-stop pattern usually points to dedicated infrastructure rather than a residential connection, so might be worth an ASN check on those too — if they also turn out to be hosting/datacenter ranges, the same „block the real allocation, not just one slice of it“ approach would probably apply there as well.
two things. On the 10.10.0.0 you spotted: that was just a labeling gap in our monitoring, not an actual blocked client. A couple of frontends were showing under an internal IP instead of their public address, so we fixed the dashboard, it now shows the real public IPs instead of the 10.x ones.
On the ASN / provider-level blocking idea: really solid thinking, and it’s the right way to deal with the datacenter stuff long-term. For now though we’re going to leave the blocking as it is, the current dynamic blocks are holding up fine and a full ASN-aware setup is a bigger project than we want to take on right now. We’re keeping it on the list in case the abuse picture changes.
Thanks a lot for all the input, genuinely appreciated.