Subject: [tsvwg] RTT independent Prague results and code

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Fri, 24 Apr 2020 08:55:10 +0000

Hi all,

Just to point you to the presentation update with more results than on the previous interim, and code: https://l4steam.github.io/rtt-independence/. There is also a link to the video demonstrated at the interim.

Here you can clearly see the RTT-independence issues are solved now.

The available source code of TCP Prague has now the means to incorporate an RTT conversion functions f(rtt)=... This function converts the measured rtt to the f(rtt) that Prague uses to converge its rate to. So setting f(rtt)=15ms, will cause Prague always to converge to a rate of a flow that would have an rtt of 15ms, independent of the rtt it has. It will behave like this when competing to other flows. We made sure in our implementation that having a smaller rtt will not influence the "dynamic" advantages, like faster up to speed, etc...

The results show 2 examples we implemented:

  *   the proposed f(rtt)=max(rtt, 15ms), effectively reducing Pragues rtt based aggressiveness to a flow of 15ms
  *   the gimmick f(rtt)=rtt+15ms, to show that if you wanted, you could undo the 'rate advantage' of Prague facing a lower queue size in the dualQ

As you can see in the first 3 experiments there is not much difference between both equations. In these experiments we vary the base_rtt for both Prague and Cubic from 0 to 50ms and show how the rate evolves. In the first experiment we use standard Prague (say f(rtt)=rtt), in the second and third the above equations.

The fourth experiment is different as we compare Prague with Prague_rtt_independent and only vary the base_rtt of Prague_rtt_independent, while we keep the base rtt of Prague at 15ms! We then show that f(rtt)=max(rtt, 15ms) effectively works also on Prague to Prague competition on a STEP AQM.

All code (including linux kernel with DualPI2 and Prague at https://github.com/L4STeam/linux including also the Classic_ECN_Fallback code in the meantime) and scripts are available to fully reproduce these experiments.

Regards,
Koen.

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 24 Apr 2020 11:56:23 +0200

Dear Koen,

what I see in that data drop is what I feared all along, you poke some seemingly random umber (by default 10000µs) close to what you put into the dual queue coupled AQM into TCP Pragues response function and declare the issue to be solved. 

I note that according to the website, this is OFF by default, the default value of 10ms does not really rhyme with the dual queue coupled AQM's 15ms.

In short I am not convinced that:

A) this method deserves the name decreased RTT-dependence, it really is just a work-around for the dual queue coupled AQM's questionable design choices. I respectfully ask you to test one 500ms cubic flow against a 50ms TCP-Prague ECT(1) flow and show them coming up roughly equally before declaring "the RTT-independence issues are solved".

B) I have trouble understanding the data in the slides (I will ignore the video, as that is a terrible format to base a discussion on)

Looking at the data: 

Test results—100Mbit/s bottleneck, increasing base RTTs:

I see that the add-15ms trick ameliorates TCP Pragues more agressive bandwidth hogging at RTTs below 15ms, but it is also clear that at longer RTTs TCP Prague always has a consistent throughput advantage over cubic. This seems true independent on the f() used in TCP Pragues congestion response. I do not find this to be in line with the RFC's promised "Scalable and Classic flows competing under similar conditions run at roughly the same rate." Here we see a _systematic_ bias in favour of TCP Prague, the RFC should IMHO either reflect that, if that is the desired outcome, or the implementation needs to be fixed to actually deliver on that promise.

Test results—100Mbit/s bottleneck, mixed base RTTs:

At low RTTs there is a noticeable throughput difference between TCP Prague (fixed15ms) and TCP Prague max(rtt,15ms), but none at a real RTT of 15ms. According to how this works, I wonder why both do not share the link more equitably, as I assume they should. Also pitting TCP Prague max(rtt, 15) at increasing RTTs against TCP Prague (10) once more drives the message home that RTT dependence has NOT been solved in any meaningful way.

Again, I am not convinced that your reference L4S-compliant transport protocol is near to a shape where experimental deployment into the wider internet seems like the prudent thing to do. 
In fact I question whether you convincingly demonstrated yet, that the requirements that L4S places upon prospective qualifying protocols can actually be achieved with sufficient robustness and fidelity in the real world. 
What am I missing?

Best Regards
	Sebastian

> On Apr 24, 2020, at 10:55, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> Hi all,
>  
> The available source code of TCP Prague has now the means to incorporate an RTT conversion functions f(rtt)=… This function converts the measured rtt to the f(rtt) that Prague uses to converge its rate to. So setting f(rtt)=15ms, will cause Prague always to converge to a rate of a flow that would have an rtt of 15ms, independent of the rtt it has. It will behave like this when competing to other flows. We made sure in our implementation that having a smaller rtt will not influence the “dynamic” advantages, like faster up to speed, etc…

	Why do these advantages actually result in an apparent throughput disadvantage of TCP Prague max(rtt, 15) against TCP Prague (15) then, as demonstrated in the last two slides?

> 	• the proposed f(rtt)=max(rtt, 15ms), effectively reducing Pragues rtt based aggressiveness to a flow of 15ms
> 	• the gimmick f(rtt)=rtt+15ms, to show that if you wanted, you could undo the ‘rate advantage’ of Prague facing a lower queue size in the dualQ
>

---

From: Wesley Eddy <[email-masked]>
Date: Fri, 24 Apr 2020 10:53:08 -0400

> A) this method deserves the name decreased RTT-dependence, it really is just a work-around for the dual queue coupled AQM's questionable design choices. I respectfully ask you to test one 500ms cub…

Since CUBIC is RTT unfair to itself, I think we should only expect 
Prague (in a dual queue) to be no worse.  Rather than expecting equal 
performance for Prague vs CUBIC at that 10x RTT imbalance, I think it 
makes more sense to compare how the CUBIC flow does in relation to CUBIC 
vs CUBIC at the same RTT imbalance.  In essence, I believe we just want 
a lower-RTT Prague (+ dual queue) flow to hurt it less or equal to how 
much a lower-RTT CUBIC flow hurts it.

Unless I misunderstood, I think this is similar to what Jana said at one 
of the prior meetings.

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Fri, 24 Apr 2020 15:11:43 +0000

Hi Sebastian,

I agree with Wes. You cannot expect Prague to guess (mind-read) the RTT other flows have it competes with and be RTT fair with all of them at the same time. It is really impossible, as these CUBIC flows might have different RTTs by themselves and have different rates already. What do you expect other than Prague to Prague agreeing on a reference RTT that maps to the minimum typical delay that a Classic flow experiences on a bottleneck under congestion. The "seemingly random" value is exactly that value (arguable 10ms or 15ms). We didn't fix a default yet. It would be good to find consensus on the list for this value and the function for the default Prague. By the way, also Classic Congestion control can become more RTT independent in the same way. Our method also works for Classic Congestion Controls.

What you expect is not a feature of SCE and definitely not a reason to promote SCE, but a feature of FQ that gives all flows an equal rate enforced by the network. So if you really expect all flows to have equal rate, you can use FQ, OR/AND make all flows (Classic and L4S) RTT-independent to a "seemingly random" reference RTT. Good thing of L4S is that we can do both at the same time: support FQ AND support the end-system RTT-independent fairness protocol on shared and coupled queues.

Hope this helps setting realistic (instead of paranormal ;-) ) expectations.

Thanks,
Koen.

-----Original Message-----
From: Wesley Eddy <[email-masked]> 
Sent: Friday, April 24, 2020 4:53 PM
To: Sebastian Moeller <[email-masked]>; De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>
Cc: tsvwg IETF list <[email-masked]>
Subject: Re: [tsvwg] RTT independent Prague results and code

> A) this method deserves the name decreased RTT-dependence, it really is just a work-around for the dual queue coupled AQM's questionable design choices. I respectfully ask you to test one 500ms cub…

Since CUBIC is RTT unfair to itself, I think we should only expect Prague (in a dual queue) to be no worse.  Rather than expecting equal performance for Prague vs CUBIC at that 10x RTT imbalance, I think it makes more sense to compare how the CUBIC flow does in relation to CUBIC vs CUBIC at the same RTT imbalance.  In essence, I believe we just want a lower-RTT Prague (+ dual queue) flow to hurt it less or equal to how much a lower-RTT CUBIC flow hurts it.

Unless I misunderstood, I think this is similar to what Jana said at one of the prior meetings.

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 24 Apr 2020 17:12:02 +0200

HI Wesley,

> On Apr 24, 2020, at 16:53, Wesley Eddy <[email-masked]> wrote:
> 
> Since CUBIC is RTT unfair to itself, I think we should only expect Prague (in a dual queue) to be no worse.

	Sorry, the L4S ecn RFC:= (https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-10) states:

" o  A scalable congestion control MUST reduce or eliminate RTT bias
      over as wide a range of RTTs as possible, or at least over the
      typical range of RTTs that will interact in the intended
      deployment scenario (see Appendix A.1.5 for rationale)."

Not doing worse than CUBIC is IMHO falling well short of that requirement. It could be that better is not achievable, which I would grudgingly accept, but the RFC text IMHO needs to be realistic about this. A MUST that nobody can fulfill and that nobody is going to monitor/police seems not very productive.
But, given that the dual queue coupled AQM's purpose is to isolate the two traffic classes, I wonder whether it is proper to give L4S that much leeway here?

>  Rather than expecting equal performance for Prague vs CUBIC at that 10x RTT imbalance, I think it makes more sense to compare how the CUBIC flow does in relation to CUBIC vs CUBIC at the same RTT imbalance.

	That certainly is a way of addressing the issue, but that needs to be reflected in the actual FRC requirement IMHO.

>  In essence, I believe we just want a lower-RTT Prague (+ dual queue) flow to hurt it less or equal to how much a lower-RTT CUBIC flow hurts it.
> 

Again then let's rephrase the requirement in the RFC that it becomes clear that all that is mandated is to not be worse than the reference TCP.
I am again only trying to measure how well L4S's chosen reference transport implementation fulfills the L4S requirements; and I am not convinced that RFC and implementation are as well aligned as I would consider desirable. As far as I understand, the reference implementation's purpose is partly to demonstrate that the requirements are 

Best Regards
	Sebastian

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 24 Apr 2020 17:18:04 +0100

I think reducing RTT bias is a reasonable goal, but I would not expect 
RTT-bias to stop work in the IETF. At least it seeme to me such bias is 
an artefact of many IETF control loops.

I'll respond though to this usage of RFC2119 language: My own view would 
be that the "MUST" is not accaptable in a a sentence that includes  "as 
low as possible", and this is at most "RECOMMENDED" practice.

Gorry Fairhurst

(as an inidividual)

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 24 Apr 2020 19:22:22 +0200

Hi Koen,

thanks for the reply. More below in-line.

> On Apr 24, 2020, at 17:11, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> Hi Sebastian,
> 
> I agree with Wes. You cannot expect Prague to guess (mind-read) the RTT other flows have it competes with and be RTT fair with all of them at the same time.

	[SM] Well, "A scalable congestion control MUST reduce or eliminate RTT bias over as wide a range of RTTs as possible" was/is your requirement, not mine. I just noticed that under the moniker "RTT independent Prague results" you demonstrated something clearly RTT-dependent. At the very least the RFC should de-escalte the MUST to a SHOULD..., no?

> It is really impossible, as these CUBIC flows might have different RTTs by themselves and have different rates already.

	[SM] In the example under discussion there was a single cubic flow (and single TCP Prague flow), maybe postpone discussing multi-flow scenarios, until after we convinced ourselves that we have a solution for the one-flow-per class scenario?

> What do you expect other than Prague to Prague agreeing on a reference RTT that maps to the minimum typical delay that a Classic flow experiences on a bottleneck under congestion.

	[SM] First I expect calling a spade a spade. You are not really decreasing RTT-dependence in any meaningfully way (except in the RTT range from 0-15ms), you are really just fixing a clear mis-design in the dual queue AQM somewhere where it really does not effect your RFC drafts much. 
	I have been mentioning that IMHO RTT-dependence is a natural consequence of shorter control loops being tighter, the only fix from the end-points view is to make short control loops artificially longer, but that comes with its own side-effects (sluggishness, there is a reason why nobody proposed this obvious idea to "fix" TCP in general). However, as your FQ example from below makes it clear, at the bottleneck a competent AQM can avoid letting these RTT biases in control loops actually translate into throughput imbalances. So it seems pretty clear to me that the AQM is a much superior place to fix apparent RTT-dependence issues. And that holds doubly if we are talking about RTT differences that the AQM introduced artificially. 

> The "seemingly random" value is exactly that value (arguable 10ms or 15ms).

	[SM] Yes, I realize that, but none of the L4S RFC drafts actually and TCP Prague has no real RFC compare against. But IIRC the DualQ RFC specifies 15ms, where do the default 10ms come from?

> We didn't fix a default yet.

	[SM] Then the text on the web-page you posted might be misleading? It says there that prague_rtt_scaling <mode> defaults to 0 (off), and prague_rtt_target <target_us>, defaults to 10000 or 10ms. I read this as defaults, where am I going wrong? But the fact that there is a natural default, namely exactly those 15ms this mechanism is intended to counter-act, so why was that not enshrined as active default, as a proposal?

> It would be good to find consensus on the list for this value and the function for the default Prague.

	[SM] Honestly, go fix the dual queue coupled AQM already so you do not need this fudge, instead? 

> By the way, also Classic Congestion control can become more RTT independent in the same way. Our method also works for Classic Congestion Controls.

	[SM] I do not think that this artificial reduction of temporal fidelity of congestion control is a good idea, not in general but also not for L4S.

> 
> What you expect is not a feature of SCE and definitely not a reason to promote SCE,

	[SM] Note, I am not promoting SCE here, I did not even mention it.

> but a feature of FQ that gives all flows an equal rate enforced by the network. So if you really expect all flows to have equal rate, you can use FQ, OR/AND make all flows (Classic and L4S) RTT-independent to a "seemingly random" reference RTT.

	[SM] As long as nobody along the path is afflicted with L4S' dual queue coupled AQM that is a valid argument, but as far as I can tell you envision rolling this out quickly, e.g. to down- and upstreams of DOCSIS subscribers, which make up a considerable chunk of internet end-users. The making all "all flows (Classic and L4S) RTT-independent" is obviously a pie in the sky idea, which I am going to ignore.

> Good thing of L4S is that we can do both at the same time: support FQ

	[SM] Except, you do not do this, all you do is basically not forbid to use L4S with FQ (IMHO that is distinct from supporting something), most the L4S work is aimed at avoiding FQ at all costs. IMHO, if you compete with FQ it is fair game ot point out where your solution fall short. 

> AND support the end-system RTT-independent fairness protocol on shared and coupled queues.

	[SM] You are again claiming RTT-independence after above arguing that that is really not generally possible, what am I missing?

> Hope this helps setting realistic (instead of paranormal ;-) ) expectations.

	[SM] Well, then please adjust the text in the RFC drafts to only raise realistic expectations. 

Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 24 Apr 2020 19:41:11 +0200

Hi Gory,

> On Apr 24, 2020, at 18:18, Gorry Fairhurst <[email-masked]> wrote:
> 
> I think reducing RTT bias is a reasonable goal,

	[SM] That is well true, but the proposed modification to TCP Prague does not even try to do this. This is plain and simple a band-aid for the fact that the dual queue coupled AQM as currently designed/implemented is unsuitable to its intended purpose, without massive changes into completely different layers. Let's call a spade a spade, please.

> but I would not expect RTT-bias to stop work in the IETF.

	[SM] Am I unreasonable to expect that if a RFC requires something that it should have first demonstrated the feasibility? Is it enough that something is theoretically desirable?

> At least it seeme to me such bias is an artefact of many IETF control loops.

	[SM] This is not an artifact, but a consequence of the speed of light, control loops of different temporal length have different temporal fidelity, I see no real way around that short of not playing that game at all. As Koen observed, FQ-AQMs can equally share bandwidth between flows of wildly different RTTs, so we have a situation that is both not really fixable from the endpoints, but demonstrably can be solved by the bottleneck AQM. So why are we even contemplating to fix an isolation failure in a different AQM by requiring the end-points to attempt the impossible?

> 
> I'll respond though to this usage of RFC2119 language: My own view would be that the "MUST" is not accaptable in a a sentence that includes  "as low as possible", and this is at most "RECOMMENDED" practice.

	[SM] My sentiment as well. Maybe a re-scrutinization of the MUST/SHOULD classification in the L4S drafts is in order?

Regards
	Sebastian

---

From: Gorry Fairhurst <[email-masked]>
Date: Sat, 25 Apr 2020 07:41:39 +0100

On 24/04/2020 18:41, Sebastian Moeller wrote:
> Hi Gorry,

That's indeed not an unreasonable goal, but alas still not always the case.

Speaking still as-an-individual:  I am not sure whether we need this as 
a formal requirement, it's surely nice.

> 	[SM] This is not an artifact, but a consequence of the speed of light, control loops of different temporal length have different temporal fidelity, I see no real way around that short of not playing that game at all.
:-).
> As Koen observed, FQ-AQMs can equally share bandwidth between flows of wildly different RTTs, so we have a situation that is both not really fixable from the endpoints, but demonstrably can be solved by the bottleneck AQM. So why are we even contemplating to fix an isolation failure in a different AQM by requiring the end-points to attempt the impossible?

And I see scheduling as a separate (complimentary) function that has 
merits and drawbacks. So I think this should be a separate conscious 
choice to the AQM selection.

> 	[SM] My sentiment as well. Maybe a re-scrutinization of the MUST/SHOULD classification in the L4S drafts is in order?
Yes, if there are other sentences that can't be "standardised" we should 
look at them to see the intention. That will happen anyway when the 
shepherd does his pass on the document, but it is good to spot those.
> 	Sebastian
Gorry

(still wearing no hats and simply commenting on things).

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Sat, 25 Apr 2020 22:38:52 +0000

Hi Sebastian,

Agreed, you have a point here, thanks. This should indeed be specified more specific towards the goal which is the safety aspect. It is only a safety problem if the base RTT is much lower than the typical queue delay target, so it MUST limit the steady state rate increase due to and for the lower RTT range. This is why I proposed at least something like f(rtt) = max(rtt, 15ms). We don't intent to have a MUST requirement to eliminate rate decrease due to higher RTTs. This can be an optional performance improvement if desired. No need to require this at all as it is not a safety concern. I agree we should rephrase this, making sure it cannot be interpreted this strict.

The goal of this requirement was to replace the function of the "big" Classic queue latency that evened extreme RTT differences between flows. If this network buffer is really small the RTT-unfairness is too extreme as well. This is why classic AQMs still have around or more than 15ms queue latency targets. In an FQ the target could be stretched further as RTT unfairness is overruled by the flow scheduler. 

This is not a problem of DualQ. It is a problem with 1/RTT dependent congestion controls sharing a very low latency queue. So this is not a network problem, but an end system problem (which for example BBR does not have). The end system principle is to prefer end-system solutions anyway, as it typically scales better. There is an end-system solution, so why not using it?

In our current Prague version the RTT-independence default is off. The goal is of course to enable this by default (maybe even to remove the option).

We are also not trying to avoid FQ per se. L4S works with FQ, we only want to make sure that it will not become a requirement to deploy FQ to get L4S-like benefits.

Regards,
Koen.

-----Original Message-----
From: Sebastian Moeller <[email-masked]> 
Sent: Friday, April 24, 2020 7:22 PM
To: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>
Cc: Wesley Eddy <[email-masked]>; tsvwg IETF list <[email-masked]>
Subject: Re: [tsvwg] RTT independent Prague results and code

Hi Koen,

Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 26 Apr 2020 11:18:31 +0200

Hi Koen,

> On Apr 26, 2020, at 00:38, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> Hi Sebastian,
> 
> Agreed, you have a point here, thanks.

	[SM] I note that this is not a new point, I have mentioned the discrepancy between the claim and the results from early on IIRC. It is that disconnect between hype and actual data that make me severely doubt L4S suitability for the real world. In civil engineering there is the concept of keeping a safety margin to allow for unexpected complications and conditions, in the whole L4S effort I am missing that kind of defensive thinking/design and that concerns me a lot.

> This should indeed be specified more specific towards the goal which is the safety aspect.

	[SM] But that is not actually a new insight. What is new is that I insist upon comparing what L4S promises with what it actually delivers.

> It is only a safety problem if the base RTT is much lower than the typical queue delay target, so it MUST limit the steady state rate increase due to and for the lower RTT range.

	[SM] But now you have the problem that a transport specification needs to operate against a completely unknown target, what latency-under-load-increase to account for? Go for the widely deployed 5ms to be expected of default codel/fq_codel, or the 10 or 15ms of different PIE instances (DOCSIS seems to mandate 10ms, L4S 15ms, can you guys not even make your own mind up?), or the exact 15ms of the not yet widely deployed L4S AQM? IMHO this again supports a design in which the side-effect of a newly introduced AQM is solved inside that AQM instead of requiring to tailor transport protocols to conform to that AQM. Because, if you truly set out for co-evolving AQM and transport protocol, I have a few ideas for you to contemplate that will leave L4S as currently designed in the dust, like explicitly telling the AQM the RTT (in which case )

> This is why I proposed at least something like f(rtt) = max(rtt, 15ms).

	[SM] BTW, the function obviously should be sum(rtt, 15), assuming that you expect that 15ms is the one canonical latency-under-load-increase for all time. max(rtt, 15) is again playing your old game of stacking the cards in favour of L4S traffic (for RTT > 15ms), but I digress. And no, I do not consider thisto be out-of-line, as I see a consistent and systematic bias in the L4S drafts to choose the option better for L4S traffic if forced to choose.

> We don't intent to have a MUST requirement to eliminate rate decrease due to higher RTTs.

	[SM] But that is exactly how a naive reader understands the text in the RFC, as a generic RTT-debiasing requirement, which is both desirable and likely impossible. 

> This can be an optional performance improvement if desired.

	[SM] Since when is dumbing down/making something react more sluggish considered a performance improvement? If this fudge truly would improve performance, I am sure you would not have stopped at max(rtt,15); rather something like fixed(500) to also include satellite links.
	Either I am missing something, or you are trying too hard to dress up an actually undesirable requirement as "feature". That is what I requested as "calling a spade a spade" and pretending this is a "performance improvement" is not an objective and factual statement (as it requires a non standard definition of performance).

> No need to require this at all as it is not a safety concern. I agree we should rephrase this, making sure it cannot be interpreted this strict.

	[SM] Yes, please. Bonus points for explicitly stating that this is only required because the dual queue coupled AQM has an open flank there and needs external help.

> The goal of this requirement was to replace the function of the "big" Classic queue latency that evened extreme RTT differences between flows.
> If this network buffer is really small the RTT-unfairness is too extreme as well. This is why classic AQMs still have around or more than 15ms queue latency targets.

	[SM] Pardon me? Codel has a default target of 5ms (which I consider considerable lower than 15ms), as I have been repeating ad nauseam. I also proposed multiple times to test a target of 5ms for your classic queue, but so far all I got was "crickets". At which point we would have reduced the worst case imabance from 15:1 to 5:1, not great, but certainly a big improvement independent of whether end-points play along or not. Will I be happy and consider the issue solves with a 5ms target? No, I will not, as I consider 1:1 to be the goal and would accept 2:1-1:2, others will see this differently though.

> In an FQ the target could be stretched further as RTT unfairness is overruled by the flow scheduler. 

	[SM] "unfairness is overruled by the flow scheduler"? That indicates a slight misunderstanding what FQ is. Serving queues equally simply side steps the issue of RTT bias, assuming there is enough buffering to allow each flow its pBDP amount of buffering to avoid stalls/under-utilization. As I said, the one way to avoid suffering from the pesky fact that the speed of light results in a control loop tightness inversely proportional to the RTT is to not play that game. The end points are in no position to solve the RTT-bias issue, but your AQM is. 
	Why can't your AQM do this then, and why should the world deploy an AQM that clearly falls behind the state of the art in this arguably important dimension?

> This is not a problem of DualQ. 

	[SM] Pardon me? The dual queue coupled AQM forces 15ms of additional queueing on one traffic class but not the other, therefore the observed imbalances are direct consequences of this AQM's actions. I do not want to play semantic games here, but it is clear that this issue is a side-effect of your AQM design choices. I do not believe it to be unreasonable to expect you not to create these side-effects in the first place.
	My take on this is like the following. At its core DualQ is a scheduler with a 1~6:1 throughput split between the L4S- and the non-L4S-queues, that also employs a number of heuristics (coupling) that for a number of conditions ameliorate the 16:1 imbalance into something more equitable (albeit almost always with the L4S-queue getting a bit more throughput). Instead of fixing the AQM to have a better/more equitable worst-case sharing behavior you propose to make the problematic situations more unlikely in which the heuristics mis-fire. I do not consider this to be a safe design method.

> It is a problem with 1/RTT dependent congestion controls sharing a very low latency queue.

	[SM] One of these Doctor it hurts when I do X issues, with the obvious "don't do X then" "solutions".

> So this is not a network problem, but an end system problem (which for example BBR does not have).

	[SM] Sorry, BBR seems to have an inverse RTT-bias, see https://arxiv.org/pdf/1706.09115.pdf. Again the speed of light and path's of different length lead to issues with having control loops converge. It seems you can decide which side to prefer, but equitable sharing is not in the books.
	But, BBR is still pretty rare in the wild, and in version 1 will also ignore ECN and will be replaced by v2 in the near future?.

> The end system principle is to prefer end-system solutions anyway, as it typically scales better. 

	[SM] Assuming a problem can be solved by the end systems... Which in this case clearly is not the case. Why expect the end system control loops to clean up the mess you create with an unsuitable AQM in the fist place?

> There is an end-system solution, so why not using it?

	[SM] Because the claim "There is an end-system solution" does not fit with your agreement that you have not solved RTT-dependence for TCP Prague in general?

> In our current Prague version the RTT-independence default is off. The goal is of course to enable this by default (maybe even to remove the option).

	[SM] Not exposing this as a parameter will at least avoid people abusing this to gain an advantage to the end-users eyeballs, but it does not solve the conceptual issue, that this is fixing a self-inflicted wound at the wrong place/layer.

> 
> We are also not trying to avoid FQ per se.

	[SM] You might want to discuss this with Bob first, your messaging on that point is all over the place. My take, you really want L4S sound like it gives FQ performance while not being bound to any of the guarantees users would expect from a FQ system. Why you do that is completely opaque to me though.

> L4S works with FQ, we only want to make sure that it will not become a requirement to deploy FQ to get L4S-like benefits.

	[SM] Sorry, I am not buying that. Bob clearly despises FQ with a vengeance (just look at his publications, with titles like "Flow Rate Fairness: Dismantling a Religion" there is no ambiguity left). Your position is be more open and un-biased, granted, but that does not IMHO counter Bob's clear bias. And do not get me wrong, FQ is not a panacea and the solution for everything, but if you claim your solution is just as good/better than FQ expect to be measured against what an FQ system can deliver. This is especially relevant here as with SQM for years now there is a system out in the wild which already successfully reduces the median queueing delay into the 5ms range, that is based on FQ and works quite well in the real world already. So you are not competing only against a theoretical FQ-AQM, but against deployed reality, and from my vantage point things are not looking good for L4S, too little too late, with too many undesirable side-effects. Which is sad, because I can happily get behind the drive for higher temporal fidelity (lower amplitude) congestion feed-back signaling, I just do not see the L4S approach as currently designed/implemented as the only or best approach to get such signaling into the internet.

Regards
	Sebastian

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Sun, 26 Apr 2020 14:01:28 +0000

Hi Sebastian,

Seems you try to sell FQ again. I don't think there is a discussion on whether FQ can give (approximal) equal rates to flows (almost independent from the RTT), as it is part of its design with all its con's as well. So don't expect for DualQ to do the same, as it is an impossible goal for the network, without actually doing FQ. As long as the goal is that the network should be able to process packets without knowing each packet's flow id and RTT, this endpoint problem need to be solved within the endpoints. 

Koen.

-----Original Message-----
From: Sebastian Moeller <[email-masked]> 
Sent: Sunday, April 26, 2020 11:19 AM
To: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>
Cc: Wesley Eddy <[email-masked]>; tsvwg IETF list <[email-masked]>
Subject: Re: [tsvwg] RTT independent Prague results and code

Hi Koen,

Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 26 Apr 2020 22:40:46 +0200

Hi Koen,

more below in-line.

> On Apr 26, 2020, at 16:01, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> Hi Sebastian,
> 
> Seems you try to sell FQ again.

	[SM] I am sorry, if I made you think I was trying to sell something here. Not my intent. As I said, I try to measure L4S implemented reality against its RFC text. I note where these differ, not more not less. I bring in FQ when IMHO appropriate as an example. If you can solve all the relevant issues without resorting to traditional FQ techniques, good for you good for us. 

> I don't think there is a discussion on whether FQ can give (approximal) equal rates to flows (almost independent from the RTT), as it is part of its design with all its con's as well. So don't expect for DualQ to do the same,

	[SM] The problem is, that your RFCs describe the dual queue coupled AQM in terms that imply similar isolation quality as traditional FQ would offer, so expect your AQM to be measured against an FQ reference. If you change the RFC to better match reality/design of L4S implementation, my comparison might not be as negative for L4S.
	Take as an example the whole pirouette we just took with "decreasing RTT-dependence"; you ostensibly argue for decreasing RTT-bias as a means to improve equitable sharing between traffic classes, a position which makes most sense if you think that equitable sharing (over different RTTs) is desirable. But once you do that and realize that there is an existing technique that robustly delivers on that promise, and consciously decide against using that technique, your calls to increase RTT-independence ring hollow. But then you actually called for decreasing RTT-dependence to not get worse than the non-AQM status quo...

> as it is an impossible goal for the network, without actually doing FQ.

	[SM] See we are getting somewhere, you now conceded that the network CAN solve this issue*, and it is your CHOICE to not do this in the network. That is quite different from your claim until now, that this would be an end-point issue and needed to be solved in the end-points. 

*) Unlike the end-points; short of inventing faster than light travel/information transfer, really end points CAN NOT solve this in general, the AQM can. I accept that you do not want your AQM to do this for a number of reasons, but that is different from your initial claim that this issue would be naturally solved at the end-points.

But again this is a bit of a red herring, as all you are actually doing is call for this to fix up your broken AQM design by making the transport layer less likely to trigger the broken conditions. 

> As long as the goal is that the network should be able to process packets without knowing each packet's flow id and RTT, this endpoint problem need to be solved within the endpoints. 

	[SM] Well, if you insist on a set of goals that are mutually exclusive you are in for unsatisfactory solutions. But nobody hinders you from re-evaluation the whole issue and come to different conclusions.
	I would say, if RTT-invariance is a desired/required property then L4S and your choice of AQM again proves itself to be unsuitable for the task or your description of the desired property of the AQM might be simply misleading. The thing is, instead of improving on the state of RTT-independence, you actually require RTT-dependence to be decreased for your AQM to not regress over the status quo. 

Best Regards
	Sebastian

---

From: "Tilmans, Olivier (Nokia - BE/Antwerp)" <[email-masked]>
Date: Tue, 28 Apr 2020 09:40:52 +0000

> Just to point you to the presentation update with more results than on the
 > previous interim, and code: https://l4steam.github.io/rtt-independence/.
 > There is also a link to the video demonstrated at the interim.

As a follow-up, the scripts used to generate the above plots (i.e., network
setup, tests, and plotting) are now also available, either as a link at the
bottom of the page, or directly there:
https://github.com/L4STeam/l4steam.github.io/tree/master/rtt-independence/experiments

Best,
Olivier