Subject: [Doh] DoT and DoH at Cambridge

From: Tony Finch <[email-masked]>
Date: Thu, 6 Sep 2018 10:35:35 +0100

Yesterday I deployed doh101 (https://github.com/fanf2/doh101/) on
Cambridge University's central recursive DNS servers. These are currently
doing about 2000 Do53 qps total (out of term - they will be busier in
October).

A few operational notes that may be of interest:

## DNS-over-TLS

I expected this to have a small amount of traffic initially, ramping up as
Android 9 "pie" devices (which do opportunistic DoT) become more common.
But I was immediately getting a few DoT qps.

On further investigation the vast majority of these were queries like
RRRRRR-dnsotls-ds.metric.gstatic.com (where RRRRRR is random) from devices
on eduroam. So to me it looks like Google are measuring availability of
DoT support using some small proportion of Android users. (1% of recent
versions?)

Real DoT queries for useful qnames were negligible (but not zero) as
expected.

## DNS-over-HTTPS

I'm using Firefox 62 (I upgraded from 61 while I was fiddling around with
it yesterday afternoon) and I have adjusted its network.trr settings to
use my DoH servers.

I accept application/dns-udpwireformat to support Firefox's older DoH
implementation, as well as application/dns-message.

With trr.mode = 3 (strictly DoH only) it seems a bit brittle: sometimes
Firefox will decide to stop talking to the DoH servers for some unknown
reason. trr.mode = 2 (DoH first with fallback to system resolver) works
more reliably.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Malin: Northwest 5 or 6. Moderate. Showers. Good.

---

From: Erik Kline <[email-masked]>
Date: Thu, 6 Sep 2018 18:59:40 +0900

> DoT support using some small proportion of Android users. (1% of recent

Ben Schwartz can comment further on whether there's an experiment
ongoing or not.  However, these kind of queries are also used by the
DoT code in Pie to help validate whether the DoT answering thing
actually speaks DNS (as opposed to someone's random webserver they
left running or whatnot).  I wouldn't have necessarily expected the
disparity between dnsotls queries and actual subsequent DoT traffic.
Hmm...

---

From: Tony Finch <[email-masked]>
Date: Thu, 6 Sep 2018 11:32:51 +0100

> disparity between dnsotls queries and actual subsequent DoT traffic.

Yes, I thought at first that they were "tap tap tap is this thing on?"
queries but the lack of followup real queries made me think otherwise.

I'm looking again now and there's a lot more real traffic, so it must have
been just the time of day (early evening) when I was examining the traffic
yesterday so there was a misleadingly low volume of traffic.

I should not be so quick to make inferences from too little data :-)

At the moment there's a roughy 1:10 ratio of probe queries to real
queries, typical TLS session is 30-40 milliseconds.

Thanks for prompting me to look again!

(X-proxied-for support would make this a bit easier...)

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
democracy, participation, and the co-operative principle

---

From: Tony Finch <[email-masked]>
Date: Thu, 6 Sep 2018 11:50:28 +0100

> queries, typical TLS session is 30-40 milliseconds.

Actually, that's 30 seconds (I misunderstood the nginx docs!)

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Sole, Lundy, Fastnet, Irish Sea: West or northwest 4 or 5, increasing 6 at
times, occasionally 7 later in Irish Sea. Slight or moderate, occasionally
smooth at first in Irish Sea. Rain or showers. Good, occasionally moderate.

---

From: bert hubert <[email-masked]>
Date: Thu, 6 Sep 2018 13:41:52 +0200

Hi,

Thanks for this experiment, I'd been hoping someone would run it! Of
specific interest in terms of CPU sizing is how long the TLS connection
stays open, if and how it resumes the TLS connection later on and perhaps
what algorithms you see in use.

We are attempting to convince millions of user scale deployments to turn
this on but we can't yet give any estimate on CPU load. Knowing how
efficient the actual connections are would be very good.

Can you see?

Fwiw, Firefox appears to eke out only a few DNS connections per TCP/IP
connection before reconnecting (at least on doh.powerdns.org).

	Bert

On Thu, Sep 06, 2018 at 11:32:51AM +0100, Tony Finch wrote:
> _______________________________________________
> Doh mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/doh

---

From: Tony Finch <[email-masked]>
Date: Thu, 6 Sep 2018 13:58:11 +0100

> what algorithms you see in use.

I have not done ANY tuning of this setup - defaults across the board on
nginx, and on BIND I have only increased the max concurrent `tcp-clients`
limit (or rather, I did that 4 years ago). I'm kind of winging it and
waiting to see how it grows :-)

With my setup (nginx as a TLS proxy) BIND controls the TCP session times
which in turn controls TLS session times; by default its idle timeout is
30s which is probably why that is a common session lifetime. I'm seeing a
good amount of session reuse, often lots of queries on the same
connection. Nicely done, Android :-)

Everyone is using TLS/1.2 ECDHE-RSA-AES256-GCM-SHA384.
I have configured nginx to use the "modern" TLS profile from
https://mozilla.github.io/server-side-tls/ssl-config-generator/

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Faeroes, Southeast Iceland: Easterly or northeasterly 4 or 5, increasing 6 at
times. Slight or moderate. Rain or showers. Good, occasionally poor.

---

From: Daniel Stenberg <[email-masked]>
Date: Thu, 6 Sep 2018 16:55:36 +0200 (CEST)

> connection before reconnecting (at least on doh.powerdns.org).

Firefox sets "NS_HTTP_LARGE_KEEPALIVE" for the connection used to the DoH 
server (meaning that it will use a longer idle timeout than for regular 
connections), so as long as you actually use the browser that connection is 
expected to be kept alive and reused. As long as the server side doesn't end 
it.

It can be noted that if you for example run nginx for this, the default 
(crazy) 'http2_max_requests' config value of 1000 is likely to get hit by a 
normal long-lasting DoH client such as Firefox.

-- 

  / daniel.haxx.se

---

From: Tony Finch <[email-masked]>
Date: Fri, 7 Sep 2018 15:25:05 +0100

I have been looking at a few more details of how DNS-over-TLS is being
used on my servers.

I have configured BIND with a 5 second idle connection timeout, because I
don't want to stress it with too many concurrent sessions. (I don't know
if I am being too pessimistic here.)

I'm getting on average 2.5 queries per connection; my WiFi clients are
IPv4 only, and Android appears to be clever enough to mostly avoid wasted
queries for AAAA when it lacks IPv6, so I guess a dual-stack network would
see more queries per connection.

I've set a 1 day TLS session cache lifetime (the Mozilla recommendation).
I'm seeing about 44% of connections are resumed sessions, which suggests
that it would be beneficial to hold connections open longer, if I was less
worried about that.

I've disabled session tickets, since I don't have a good way to manage the
ticket keys. (This is also what Mozilla recommends.)

One of my colleagues (David McBride) configured Unbound to use DoT;
it does one query per connection and does not resume TLS sessions, so it
looks quite different in the logs!

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Rockall, Malin, Hebrides, Bailey: Mainly northwest at first in Malin and
Hebrides, otherwise cyclonic, 4 or 5, occasionally 6 in Rockall. Slight or
moderate in east Malin and east Hebrides, moderate throughout elsewhere. Rain
or showers. Moderate or good.