Subject: [tsvwg] question regarding L4S claims

From: Sebastian Moeller <[email-masked]>
Date: Thu, 4 Jun 2020 11:27:21 +0200

Dear list,

in a recent presentation, the L4S team attempted to show why L4S is a major improvement over the status quo.

Here I want to focus on the implicit claims in slide 4 of http://bobbriscoe.net/presents/2004ietf/202005L4S-brief.pdf

The claim here is that within a budget of 50ms* L4S will allow a distance between endpoints of 3200 Km, while pie or fq_codel would only allow a distance of 500Km.

I believe this analysis to be simplistic to a point of being deliberately misleading, the data presented here are:

AQM:							fq_codel	L4S
Delay budget for 'responsive fee l[ms]:	50		50
min non-network delay [ms]:			-13		-13
99.9th %ile queuing delay [ms]:		-32		-4.5

Left for propagation round trip [ms];	5		32 		[Note: 50-13-4.5 actually equals 32.5, which traditionally rounds to 33ms...]
Equivalent reach in fibre [km]:			500		3200
Equivalent reach in fibre [miles]:		310		2000

This has multiple issues that I want to enumerate.

1) The budget value itself, is debatable, but since it will affect all AQMs equally I am willing to ignore that (besides raising questions about where the 50ms were derived from initially).

2) min non-network delay of 13 ms is a projected value in https://tools.ietf.org/html/draft-han-iccrg-arvr-transport-problem-01, which has not been demonstrated to be actually achievable, the reported current values were 130 to 1118 ms. This immediately makes it obvious that the comparison seems extremely optimistic.

3) 99.9th %ile queuing delay: the assumption here seems to be that only the single AQM hop actually controls all variable RTT jitter. And that 99.9 is a relevant number here, which obviously is debatable.

4) Left for propagation round trip & equivalent reaches: the 1 ms per 100Km RTT rule of thumb really only works for complete fiber runs, any conversion between photons and electrons on the way, any router and other network equipment will introduce additional variable delay, as will retransmits due to data corruption.

I would really like to see real world tests, in which RTT and RTT variance are measured and shown for a 3200Km path across the continental U.S. with an L4S AQM at the bottleneck, and I would like to see the full path RTT and jitter over time as well as as CDF, as for the supposed application it does not really matter where along the path the jitter is acquired.
More hard data, less overly optimistic back of the envelope estimates, please.

Best Regards
	Sebastian

*) I like the latency budget approach, we can haggle about the exact number, but the idea generally seems sane and sound, and both numbers used often 100 and here 50ms are too round to be realistic results of empiric experiments, but rather indicate some aggregation of empiric results with a modest amount of rounding ot result in a nice and clean number.
I also note that the cited https://tools.ietf.org/html/draft-han-iccrg-arvr-transport-problem-01#page-12 actually claims:
"AR/VR developers generally agree that MTP latency becomes imperceptible below about 20 ms [Carmack13].  However, some research
   has concluded that MTP latency MUST be less than 17ms for sensitive users [MTP-Latency-NASA]." The original Carmack13 website is gone and even the wayback machine comes up empty, but given the URL:
[Carmack13]
              Carmack, J., "Latency Mitigation Strategies", February
              2013, <https://www.twentymilliseconds.com/post/latency-mitigation-strategies/>

www.twentymilliseconds.com, I really wonder where the 50ms budget was derived from.

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 4 Jun 2020 11:34:22 +0200

Addendum:

http://www.altdevblogaday.com/2013/02/22/latency-mitigation-strategies/

Has a copy of Carmack's assessment:
"“A total system latency of 50 milliseconds will feel responsive, but still subtly lagging. One of the easiest ways to see the effects of latency in a head mounted display is to roll your head side to side along the view vector while looking at a clear vertical edge. Latency will show up as an apparent tilting of the vertical line with the head motion; the view feels “dragged along” with the head motion. When the latency is low enough, the virtual world convincingly feels like you are simply rotating your view of a stable world."

So according to Carmack the goal for immersive VR/AR should be 20ms:
“It’s twenty milliseconds, or go home to play with your space rockets, which are probably an easier task than a genuine virtual reality headmounted display."

"Human sensory systems can detect very small relative delays in parts of the visual or, especially, audio fields, but when absolute delays are below approximately 20 milliseconds they are generally imperceptible."

So I respectfully argue that ig Carmack is dragged into this discussion, then please use his preferred reference value of 20ms, or refer to actual psychophysical research.

Best Regards
	Sebastian

> On Jun 4, 2020, at 11:27, Sebastian Moeller <[email-masked]> wrote:
> 
> Dear list,
>

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 4 Jun 2020 14:32:06 +0300

> On 4 Jun, 2020, at 12:27 pm, Sebastian Moeller <[email-masked]> wrote:
> 

I've been wanting to bring up something related to this point for a while, but other matters have previously taken precedence.  My question is: when and under what circumstances does the observed 99.9th-percentile delay occur?

In the case of fq_codel, tested with a conventional TCP stream, it will obviously be at the termination of slow-start, when the exponential growth (doubling cwnd per RTT) cannot be stopped instantaneously, but continues for one more RTT after the first congestion signal is given (which for CUBIC may be a delay signal interpreted by HyStart).  So the peak queue depth at slow-start termination will tend to be at least the baseline path RTT, and will then quickly settle down to congestion-avoidance mode in which the peak delay is only slightly greater than the Codel target (ie. 5ms).  And I remind readers that fq_codel protects other flows from experiencing the peak delay of one flow's startup.

The situation with L4S is less obvious, noting that the data presented is very old, and comes from DCTCP rather than any recent version of TCP Prague.  I consider it likely that DCTCP's slow-start is terminated prematurely by L4S' very aggressive AQM interacting with the natural burstiness of an unpaced slow-start, so the early phase is characterised primarily by linear rather than exponential growth.  We might therefore reasonably guess (in the absence of a corresponding time-series plot which could confirm or deny this hypothesis) that the 99.9th-percentile delay quoted for DCTCP corresponds to the congestion-avoidance phase rather than slow-start.

If true, the quoted 4.5ms figure would represent only a couple of milliseconds' improvement over the congestion-avoidance latency performance of fq_codel with CUBIC, and an actual deficit of perhaps a millisecond compared to CUBIC-SCE performance with an SCE-enabled AQM, both at default settings.  And of course it is entirely possible to delete the slow-start phase entirely and rely only on the linear or polynomial growth factors, as long as you're willing to accept the greater delay before the path capacity is actually found.  If minimising peak latency is paramount, that might just be a sacrifice you have to accept.

But it's pretty clear that remote-rendering VR headset displays is not practical over a network.  It's barely practical even within a single, up-to-date gaming-spec PC, without involving a network link at all.  I can entirely believe a 20ms latency budget for that application, as it's consistent with a 90fps minimum frame rate for VR headsets that I've seen quoted elsewhere.  That equates to 11ms between frames, and the information needed to start rendering that frame has to be prepared in advance.  Just getting data over a USB cable reliably within 9ms is challenging enough!

 - Jonathan Morton

---

From: Pete Heist <[email-masked]>
Date: Fri, 5 Jun 2020 13:33:14 +0200

> On Jun 4, 2020, at 1:32 PM, Jonathan Morton <[email-masked]> wrote:
> 

I’m interested in knowing the answer to this as well.

> The situation with L4S is less obvious, noting that the data presented is very old, and comes from DCTCP rather than any recent version of TCP Prague.  I consider it likely that DCTCP's slow-start…

According to the presentation, there are 300 "web-like flows” per second with ECN, and 300 w/o ECN, using an exponential arrival process, with file sizes having a Pareto distribution from 9.1K to 1M, on a 120Mbit link with 10ms path RTT. I also expect at those high arrival rates that flows drop out of slow start early, but this would need to be reproduced to verify it. FCT would be interesting to see here for the web-like flows, as well as how performance changes with a few different realistic levels of burstiness and jitter.

> If true, the quoted 4.5ms figure would represent only a couple of milliseconds' improvement over the congestion-avoidance latency performance of fq_codel with CUBIC, and an actual deficit of perhap…

The concern I have is that minimizing peak latency of capacity seeking flows in CA isn’t paramount, but that is what has been used as a KPI in some discussions. A difference of 2-3ms in CA will often be lost in comparison to delays from other sources than queueing. More important for capacity seeking flows is that they achieve high utilization with reasonable delays. Since we’ve seen that burstiness can dramatically affect the utilization of HFCC flows depending on the marking scheme used, and since high capacity is also identified as important in draft-han-iccrg-arvr-transport-problem-01, burst tolerance and utilization need to be balanced for this application.

Delays for both isochronous and VBR UDP flows are also important to consider, because those generally will be latency sensitive, e.g. for voip, videoconferencing and interactive applications. Afaik those haven’t been tested, so we need to do that, and also look at what impacts they’ll have on HFCC flows in a shared low latency queue, and again, how the marking strategy affects utilization in that case.

Hopefully some of Sebastian’s concerns can be addressed with more realistic traffic patterns and path characteristics, which admittedly is work needed in our testing also. Reproducing a test similar to what is in that presentation but with some key variations mentioned above should be worthwhile.