Subject: [tsvwg] AQM target delay and link utilization (was: Re: RTT independent Prague results and code)

From: "Tilmans, Olivier (Nokia - BE/Antwerp)" <[email-masked]>
Date: Sun, 26 Apr 2020 10:53:47 +0000

Hi Sebastian,

Piggybacking on this particular point:

 > 	[SM] Pardon me? Codel has a default target of 5ms (which I consider
 > considerable lower than 15ms), as I have been repeating ad nauseam. I also
 > proposed multiple times to test a target of 5ms for your classic queue, but
 > so far all I got was "crickets".

Without entering any debate on particular numbers, I'll just point out the
differences between a 5ms and a 15ms target delay/excursion in queuing on link
utilization for "median Internet RTT of 80ms": [*]
5ms: https://www.heistp.net/downloads/sce-l4s-bakeoff/bakeoff-2019-11-11T090559-r2/sce-s1-1/batch-sce-s1-1-cubic-50Mbit-80ms_var.png
80ms: https://www.heistp.net/downloads/sce-l4s-bakeoff/bakeoff-2019-11-11T090559-r2/l4s-s1-1/batch-l4s-s1-1-cubic-50Mbit-80ms_var.png

Whether that under-utilization in the 5ms case is (un)acceptable is always debatable,
but it is there nonetheless. And it will worsen as the RTT increases.

Given that the main benchmark marketing[**] metric is single-flow throughput--I
am not arguing whether that's correct or not, it's just there and thus part of
the main KPIs for an ISP--, I hope this helps to see why you'll find higher default
target values in the ISP world than what you're using in the SQM world as your 
goals tend to differ.

*: While those tests put CoDel vs PI2, the results won't change much if you swap
   the target/AQMs around.
**: It is obviously more than a marketing metric too, as a non-negligible part of
    a customer base expect to get what they're paying for (either as best-effort
    or guaranteed service), i.e., also runs throughput tests. And I do not think
    you can have customer support reply "you can only get your full throughput
    with multiple connections/mptcp/home-made flow aggregation".

Best,
Olivier

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 26 Apr 2020 13:48:01 +0200

Dear Tillman,

> On Apr 26, 2020, at 12:53, Tilmans, Olivier (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> Hi Sebastian,
> 

	[SM] You might look it up in the archive, but I already cited the utilization numbers from the PIE paper way back (e.g February 11th):

"Looking at the PIE paper it looks like PIE at a target of 5ms ref_delay has slightly higher utilization cost than Codel but we are talking PIE is 96.6% vs. CoDel is 98.4%, IMHO both well acceptable."

And it is not that a) there is 100% utilization at 15ms and b) you have ripped out the burst tolerance out of your PIE for the classic queue, which also carries a utilization cost, so this is already a grey area to begin with. IMHO you are making the wrong trade-offs.

> target values in the ISP world than what you're using in the SQM world as your 

	[SM] I o understand that, but unlike the L4S team I have already openly accepted, that low latency will have a (small) throughput cost, and am willing to pay that cost. If you are unhappy with the classic PIE's utilization at 5ms, you can always switch to codel on the classic queue. From the PIE paper:
@100ms RTT:
"For the 20ms target delay case, the throughput for PIE is 9.87Mbps vs. CoDel is 9.94Mbps. For the 5ms target delay case, the throughput for PIE is 9.66Mbps vs. CoDel is 9.84Mbps. CoDel’s throughput is slightly better than PIE." If utilizatiopn is of such importance, jettison PIE might be a better approach than pretending that 15ms is perfect magic number, no?

>    with multiple connections/mptcp/home-made flow aggregation".

	[SM] Sorry, that ship has mostly sailed, end-customers use mostly ookla's speedtest.net and they default to multi-stream testing, o does Germanieis official BEREC/EU condoned speedtest breitbandmessung.de. For inter ISP traffic things might be different BUT I do not for one minute believe the hypothesis that the L4S AQM will see major roll-out away from the customer edge, so that seems pretty irrelevant.

Best Regards
	Sebastian

---

From: "Tilmans, Olivier (Nokia - BE/Antwerp)" <[email-masked]>
Date: Sun, 26 Apr 2020 14:12:52 +0000

> > 5ms: https://www.heistp.net/downloads/sce-l4s-bakeoff/bakeoff-2019-11-
 > 11T090559-r2/sce-s1-1/batch-sce-s1-1-cubic-50Mbit-80ms_var.png
 > > 80ms: https://www.heistp.net/downloads/sce-l4s-bakeoff/bakeoff-2019-11-
 > 11T090559-r2/l4s-s1-1/batch-l4s-s1-1-cubic-50Mbit-80ms_var.png
 > >

I realize I made a typo earlier, which might have introduced confusion:
The above links show the goodput of a cubic flow with a base RTT of 80ms
when controlled by:
- Codel/cobalt/cake(5ms)--about 8-10% under-utilization
- PI2(15ms)--about 1-2% under-utilization
Despite the plot legends, the actual throughput are about 4% higher due to 
ethernet/IP/TCP headers (i.e., they show goodput).

 > 
 > 	[SM] You might look it up in the archive, but I already cited the
 > utilization numbers from the PIE paper way back (e.g February 11th):
 > 
 > "Looking at the PIE paper it looks like PIE at a target of 5ms ref_delay
 > has slightly higher utilization cost than Codel but we are talking PIE is
 > 96.6% vs. CoDel is 98.4%, IMHO both well acceptable."

I assume you refer to those in HPSR'13 (with its 9.66Mbps and 9.84Mbps over
10Mbps). These results were running 20 flows in parallel. 

 > And it is not that a) there is 100% utilization at 15ms and b) you have
 > ripped out the burst tolerance out of your PIE for the classic queue,
 > which also carries a utilization cost, so this is already a grey area to
 > begin with. IMHO you are making the wrong trade-offs.

Wrt. a) Yes, if we wanted to optimize for a single-flow@80ms we'd need a
higher target. See at end of the mail.
As to b), the 5ms case is cake and not PIE/PI2 (if I understood your comment
properly, and if so, sorry for the confusing links)--if you were referring
to the PIE burst heuristic instead, there's a series of emails (in aqm@ietf
IIRC) from Bob analysing the PIE heuristics and how PI2 side-steps most of
them with an alternate control equation.

 > "For the 20ms target delay case, the throughput for PIE is 9.87Mbps vs.
 > CoDel is 9.94Mbps. For the 5ms target delay case, the throughput for PIE
 > is 9.66Mbps vs. CoDel is 9.84Mbps. CoDel’s throughput is slightly better
 > than PIE." If utilizatiopn is of such importance, jettison PIE might be
 > a better approach than pretending that 15ms is perfect magic number, no?

I am not debating the choice of AQM, their respective (in)ability to properly
control bursts of (un)responsive traffic to the target delay, to  or their
influence on flow startup performance. All of these are in the literature.

I am just pointing out that these target delay do have real impacts,
irrespectively of the AQM flavor, as visible in the above graphs.

Picking 15ms or 5ms as your sweet spot just translates to a different BW-delay
tradeoff in the worst-case, and a different level of statistical multiplexing
as your "normal" load. The 15ms comes from the DOCSIS recommendation
for cable modems, which suits the usage pattern their members (i.e.,
65 operators). Greg can certainly elaborate on that if it is such a concern.

If this does not match the (future) reality (esp. in lights of the increased
homeworking in recent weeks), it can always be changed.

Best,
Olivier

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 26 Apr 2020 18:23:59 +0200

Hi Olivier,

please excuse my name mix-up in the last email.

> On Apr 26, 2020, at 16:12, Tilmans, Olivier (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 

	All of these tests were performed at fixed packet-sizes, so the difference between goodput/thoughput at different layers becomes orthogonal to the quesation at hand, thanks for pointing that out.
	But, this Cake @5ms versus PIE2@15ms is not the comparison I asked for.

> 10Mbps). These results were running 20 flows in parallel. 

	[SM] Yes, I was referring to that data, Thanks for clearing this up. I assume you must have done the equivalent for the one flow per class case to deduce that 15ms is the correct value. I would really really like to see that data that made you decide 15ms is perfect. Either you have this data, then please present it, or you did not do this analysis, then please confirm that 15ms simply is your first guess that was never empirically optimized. There is nothing inherently bad in the latter, but it needs to be spelled out.

> higher target. See at end of the mail.

	[SM] So explain to me, in what way is 15ms acceptable to the L4S approach, but 5ms is not, especially given the nasty side-effect this has on the dual queue coupled AQM's sharing behavior? I realize that this is all about trade-offs, I want to understand why you are not willing to negotiate one millisecond here and even propose to fudge-up your transport protocols congestion response fidelity instead?

> them with an alternate control equation.

	[SM] I am sorry to be blunt, but after Bob's descriptions of how codel works, I am unsure whether I trust his analysis of AQMs that far.

> irrespectively of the AQM flavor, as visible in the above graphs.

	[SM] Yes, they do. I have been asking for a demonstration with data  why 15ms was chose, so far none was forthcoming. And again you offer generalities instead of had cold data. 

> as your "normal" load.

	[SM] Fair enough, no show me that the trade-off @5ms is unacceptable, while the one @15ms is acceptable, please.

> The 15ms comes from the DOCSIS recommendation

	[SM] Please explain, why the docsis PIE RFC "Active Queue Management (AQM) Based on Proportional Integral Controller Enhanced (PIE) for Data-Over-Cable Service Interface Specifications (DOCSIS) Cable Modems" (https://tools.ietf.org/html/rfc8034#page-5) does recommend 10ms then?

"4.1.  Latency Target

   The latency target (a.k.a. delay reference) is a key parameter that
   affects, among other things, the trade-off in performance between
   latency-sensitive applications and bulk TCP applications.  Via
   simulation studies, a value of 10 ms was identified as providing a
   good balance of performance.  However, it is recognized that there
   may be service offerings for which this value doesn't provide the
   best performance balance.  As a result, this is provided as a
   configuration parameter that the operator can set independently on
   each upstream Service Flow.  If not explicitly set by the operator,
   the modem will use 10 ms as the default value."

I have also brought this up in the past, only to hear nothing in the way of a rationale.

This is getting old fast, you guys need to figure out your story and data and stick to it, please.

> homeworking in recent weeks), it can always be changed.

	[SM] Again, is it too much to ask for you to actually go optimize your free parameters before proposing roll-out into the wider internet?

Regards
	Sebastian