Subject: [tsvwg] SCE-L4S ECT(1) test results

From: Pete Heist <[email-masked]>
Date: Fri, 24 Apr 2020 11:29:37 +0200

All,

We have completed a series of new tests which may be referred to in the upcoming discussion around ECT(1):

https://github.com/heistp/sce-l4s-ect1

For SCE, these include tests of the new low-latency PHB. For L4S, we test the recently posted classic bottleneck detection code, and frame those results around the ECT(1) discussion. We feel these results are highly relevant to the discussion, and hope they contribute to the advancement of high-fidelity congestion control on the Internet.

Regards,
Pete Heist & Jonathan Morton

---

From: "Tilmans, Olivier (Nokia - BE/Antwerp)" <[email-masked]>
Date: Sat, 25 Apr 2020 13:05:55 +0000

Pete, Jonathan,

 > https://github.com/heistp/sce-l4s-ect1

Thanks for sharing this. 

You uncovered a bug where TCP Prague performs racy ssthresh updates
when receiving loss signals, causing it to stick to INF, 
i.e., stuck in slow-start hence the devastating effect this has on
latency/fairness. This is of course unacceptable, and was fixed in
https://github.com/L4STeam/linux/commit/0e7cf8acb318873c3f61084453f8da15b2e398be

This unfortunately never showed up in our tests as we've been relying on
sch_fq due to bugs with the internal tcp pacing [1], and this somehow
shielded us from that.

I've re-run your test suite (only for the L4S flows) on a machine using the
Kernel built by github from that commit (and 2 others adding additional
qdiscs/...), i.e., the artifacts of this job:
https://github.com/L4STeam/linux/actions/runs/87218044

The results are available there:
https://l4steam.github.io/ect1-tests/
Note that I did not edit the text actually commenting the graphs, so
those will be outdated in some places. They will be updated at a later
time. As visible on the plots, the permanent slow-start issue is
addressed, hence the unwanted insensitivity to drops resolved.

As the pcaps (seems to) have been stripped out by Github, I also uploaded
the traces/pcaps/logs/graphs there:
https://drive.google.com/drive/folders/1sj9Ox6nHJN-v3xrK4ginlfO3AewAsy85?usp=sharing

Thanks for helping to quickly cross-check the differences between our
respective setups!

Best,
Olivier

[1] https://lore.kernel.org/netdev/[email-masked]/

---

From: Pete Heist <[email-masked]>
Date: Mon, 27 Apr 2020 11:37:03 +0200

Thanks, we have re-run the test batch and can confirm that the drop issue reported as #2 in our Key Findings appears to be fixed. I have posted the updated raw l4s results here: http://sce.dnsmgr.net/results/ect1-2020-04-26-l4s-dropfix/

The results also show that the other Key Findings regarding L4S remain (https://github.com/heistp/sce-l4s-ect1#key-findings). While we feel those results should be evaluated when making an engineering decision on ECT(1), at this stage it’s probably best to leave their interpretation either for the meeting's discussion or later posting.

Regards,
Pete

> On Apr 25, 2020, at 3:05 PM, Tilmans, Olivier (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> Pete, Jonathan,
>

---

From: Pete Heist <[email-masked]>
Date: Tue, 28 Apr 2020 22:18:38 +0200

As promised, I’d like to comment on the L4S results from these tests further to interpret the consequences of what they actually mean. Here are the remaining three overarching issues in the Key Findings list (https://github.com/heistp/sce-l4s-ect1#key-findings), using the same numbering scheme as in the report:

1a. Here we describe cases when bottleneck detection does not detect classic queues when there is one, which leads to starvation. The following plot shows where a competing CUBIC flow is starved to a cwnd of about 5 packets:

http://sce.dnsmgr.net/results/ect1-2020-04-26-l4s-dropfix/l4s-s2-twoflow/l4s-s2-twoflow-ns-cubic-vs-prague-codel1q_40ms_-80ms_tcp_delivery_with_rtt.svg

What it means: If AQMs are configured with lower targets, lower buffer size limits, or are misconfigured, starvation can occur. Here is one plausible scenario for that:

An administrator has set up fq_codel to manage the queue for a backhaul link, and has further decided that since much of the traffic is regional, reducing the target and interval could help improve peak latency, with an acceptable drop in utilization. She looks at the Linux man page for fq_codel, finds the second example which uses a target of 3ms and an interval of 40ms, and uses those settings. Initial testing looks good, but for some reason, users sometimes complain of intermittent network issues that are difficult to troubleshoot.

What’s happening here is that there are occasional hash collisions within fq_codel’s default of 1024 queues, sometimes placing L4S flows and classic flows in the same queue. With the tuned AQM parameters, there are false negatives in bottleneck detection for endpoints running TCP Prague, and flows from those endpoints starve out competing traffic in the same hash bucket.

Since we have agreed in prior meetings that “Starvation is not a acceptable outcome”, something will have to change about the way bottleneck detection is currently done to avoid that.

1b. Here we show false positives, where classic queues are misidentified as low latency queues, leading to loss of efficacy.

What it means: When classic bottleneck detection is enabled, the benefits of HFCC are lost with traffic that has about mean 2ms of more of IPDV (jitter), as in this plot:

http://sce.dnsmgr.net/results/ect1-2020-04-23-final/l4s-s6-jitter/l4s-s6-jitter-ns-prague-dualpi2-50Mbit-80ms-netem_delay_4ms_2ms_10__tcp_delivery_with_rtt.svg

A large portion of traffic on the open Internet has at least 2ms of jitter, which means that enabling bottleneck detection effectively disables L4S mode for a large portion of Internet flows. The concern here is that this would be a strong incentive to disable bottleneck detection, which could lead to more cases of starvation.

Classic bottleneck detection does not appear to be listed in the draft with an explicit MUST. It is in Appendix A.1.4 of draft-ietf-tsvwg-ecn-l4s-id-10 as a requirement, but without an indication on what specifically is required of it and how it should work, so it seems likely that implementors will add a toggle to turn it on and off, especially given the effect of false positives on performance, and how commonly they can occur.

3. Here we show how a typical bursty traffic traffic scenario leads to close to a 90% reduction in utilization on a 50Mbit link with dualpi2 using default parameters. Note that the scale is to 50Mbit for the link’s capacity, but throughput hovers around 5Mbit, so it’s a bit lost in the RTT data:

http://sce.dnsmgr.net/results/ect1-2020-04-23-final/l4s-s5-burstiness/l4s-s5-burstiness-ns-prague-dualpi2-50Mbit-80ms_tcp_delivery_with_rtt.svg

What it means: We should temper the expectations we have for what minimum delay is possible from HFCC for traffic on the open Internet, and decouple “ultra-low delay” from the codepoint decision.

Burstiness is an equal challenge for either proposed usage of ECT(1). We all agree that “ultra-low delay” is a desirable goal, but in the presence of sufficiently bursty traffic, which the open Internet has in abundance, ultra-low delay and ultra-high utilization is not possible at the same time, subject of course to the definitions of what each of those terms means. Currently, the default dualpi2 marking scheme is configured for very little burst tolerance, which is why low queueing delays can be demonstrated in closed environments.

> On Apr 27, 2020, at 11:37 AM, Pete Heist <[email-masked]> wrote:
>