Subject: [tsvwg] CE-marking observations

From: "Holland, Jake" <[email-masked]>
Date: Thu, 30 Apr 2020 17:01:45 +0000

Hi everyone,

I have a few numbers to report about some CE-marking queue
deployment observations over the last few months.

I've urged that we not break normal ECN, and I'll repeat that
urging here.  To me this looks like the early stages of rollout
of a technology that's finally getting a little bit of traction,
and breaking it over the next couple of years seems like a
mistake, and worth avoiding if possible.

(I still think regular AQM deployed in the right places solves
most of the problem, and that ECN helps plenty at the application
level for TCP, so starting a rollout of something incompatible
right when it finally starts happening would be harmful to the
goal of improving latency, regardless of the potential benefits.)

This is all based on observations from production traffic to end
users, since we've turned on ECN support by default since
mid-February.

There were 3 regional consumer ISPs with strong evidence of
ubiquitous marking queue deployment (above 65% of their IPs). One
of these 3 may have deployed something within the last month, as
we saw no such evidence from late February, but it was clearly
present in late March (it also could have been higher congestion
that wasn't ever hit in Feb, since there was substantially more
traffic as well).

There were 7 such ISPs that had marking queues on at least about 15%
of their IPs.

Those numbers seemed to me probably ISP-deployed, and could plausibly
be either shared queue devices or a managed deployment of CPEs with
something like the cake/codel implementations.  I have no visibility
at this point into which they used, only that there were notably more
CE-marking IPs within the AS than what's expected from individuals buying
home gateways with ECN support.

There were 70 or so ASs with observed marking capability prevalence
above 5% across their client IPs, and 30 or so ASs with observed
marking capability over 10%, but many of these were quite small.  I
again have no visibility into whether they were shared queues or no,
and at the lower end it's much harder to make a good guess at whether
it's a managed deployment or individual actions that happen to have
some coherence.  There might be a mix, especially across the lower
end of the 70, of AS-owned vs. not.

Globally I saw a lower bound on client IPs that exhibit marking at
about 0.1%.  The vast majority seemed isolated individuals most
likely using one of the various available home router installs for
their gateway, so to me the more interesting part was the early
edge of ISPs that seem to be doing something that might end up
scaling.

All these are lower bounds, in that I can only tell it's marking if
we saw CE or ECE, and in some cases we plausibly never had congestion
during the observation periods I used.  Also, it of course only has a
chance to get marked when the clients have it enabled, which happens
at quite a small rate in some areas.

These numbers also have a chance at being falsely low for the general
case because it was filtered to nearby connections with enough
connections to usefully examine their latency variation.  But it's at
least something of a lower bound.

HTH.

-Jake

---

From: Dave Taht <[email-masked]>
Date: Thu, 30 Apr 2020 10:11:38 -0700

Very good information, thank you. I'd love to see a breakdown of
results from free.fr's AS number, as they are
normally on the bleeding edge here.

With a big enough capture, and some insight from an expert, it is
possible to see the drop or mark patterns generated from pie or
fq_codel in any given flow. (pie is random, so codel's relatively fix
ed drop schedule can be picked out relevant to that)

I've often thought about trying to train up an AI to be able to see it
better than I can.

Another potential cause of seeing an uptick in these stats is from
overburdened servers (running in containers or elsewhere),
which are self congesting for some reason or another.

One thing that google does is slam a piece of epf in front of sch_fq
that drops or marks packets once the delays get intolerable.
It doesn't matter what form of ecn is used, anything that keeps the
server from self congesting is a goodness from their perspective, and
from the added stats that pop out, it's easier to migrate a congested
service to more servers.

I have no idea to what extent akamai's cloud has fq_codel in it or uses sch_fq.

-- 
Make Music, Not War

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-435-0729

---

From: Dave Taht <[email-masked]>
Date: Thu, 30 Apr 2020 10:22:18 -0700

> ed drop schedule can be picked out relevant to that)

Another way to break down the data might be that since wifi fq_codel
deployment in commercial products
has been ongoing for quite some time, is that you should see increased
marking probability the faster
the observed network behavior is, where the bottleneck shifts from the
isp to the wifi.

you should see, on short transactions, a fairly low percentage of
ecn-echo coming back from apple receivers,
relative to the larger ones, and the transaction length, at minimum,
needs to be over 100ms at a fuller rate, where
the slope of the observed rtt passes 20ms additional from the beginning.

The flaw in the dataset I had from apnic is they only tested for
capability in a 10 packet flow, rather than stressing
out the network more. It too centered on only a few ISPs. It would be
interesting if your ASes lined up with mine.

-- 
Make Music, Not War

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-435-0729