Subject: Re: [tsvwg] FW: path forward on L4S issue #16

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 9 Jun 2020 07:33:49 +0000

Jonathan..

" I think some concrete evidence of this possibility was provided by the
> experimental adaptation of SCReAM to use L4S signalling, which was
> discussed here fairly recently.  The implementation took a very bare-bones
> approach, copying only the use of the ECT(1) marker and the DCTCP
> response algorithm from whatever original source they used.  The result
was
> summarily declared a success by its author without any analysis of how it
> might interact with conventional traffic; I assume he relied on the L4S
team
> proper to have done their homework in that respect.
"

For the record the thread is
https://mailarchive.ietf.org/arch/msg/tsvwg/FFQH_XG0-wbLIt7JRJ33dH8Fyfc/ 
and the caveat was clearly visible in the post 
"Even though it is more than a magnitude difference in queue delay between
CoDel-ECN and L4S, it is fair to say that these simple simulations should of
course be seen as just a snapshot."

/Ingemar

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 9 Jun 2020 13:09:36 +0300

> On 9 Jun, 2020, at 10:33 am, Ingemar Johansson S <[email-masked]> wrote:
> 

Not to mention a magnitude difference in throughput.

 - Jonathan Morton

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 9 Jun 2020 14:11:11 +0000

Jonathan, you kick in open doors..

I urge interested people (if any?) to follow the discussion thread instead
to get an understanding what causes the lower throughput with L4S.
https://mailarchive.ietf.org/arch/msg/tsvwg/FFQH_XG0-wbLIt7JRJ33dH8Fyfc/

/Ingemar

> -----Original Message-----
> From: Jonathan Morton <[email-masked]>
> Sent: den 9 juni 2020 12:10
> To: Ingemar Johansson S <[email-masked]>
> Cc: [email-masked]
> Subject: Re: path forward on L4S issue #16
> 
> > On 9 Jun, 2020, at 10:33 am, Ingemar Johansson S
> <[email-masked]> wrote:
> >
> > "Even though it is more than a magnitude difference in queue delay
> > between CoDel-ECN and L4S, it is fair to say that these simple
> > simulations should of course be seen as just a snapshot."
>

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 9 Jun 2020 16:21:07 +0200

Ingemar,

I have a feeling that this discussion thread is based on a disconnect. I believe that most here agree that Scream is a nice and interesting project worth pursuing. The only contentious point is whether scream working well with current L4S AQM behavior over a single simulated hop, can make up for the apparent lack of testing under even mildly adversarial conditions (aka real internet conditions) of the core L4S implementation and design. If you have made such measurements in the context of scream's development, please do not hesitate to present them here.

incomplete listing of mildly adversarial conditions:
1) bi-directionally saturating traffic
2) asymmetric links
3) long networks paths, >>5 hops
4) paths with parallel segments 
5) unfriendly greedy traffic with less than expected yielding behavior
...

Data demonstrating L4S immunity against negative behaviour under any of these conditions, is IMHO quite helpful for the continuing discussion.

Best Regards
	Sebastian

> On Jun 9, 2020, at 16:11, Ingemar Johansson S <ingemar.s.johansson=[email-masked]> wrote:
>

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 9 Jun 2020 14:43:55 +0000

Hi
Why don't you run the testing yourself to get the answers you look for?. 

You can download and build the SCReAM BW test tool (follow the steps below).
I can guarantee that you'll find cases where it does not work In a
satisfactory way. But as with most other congestion control work it is work
in progress, sometimes that progress is good, sometimes not so good due to
other projects that are more important.

/Ingemar
==============================
How to get and build the SCReAM BW test tool on Linux PC's

cd ~/somewhere 
git clone https://github.com/EricssonResearch/scream.git
cd code/bw-test-tool
cmake .
make

What needs to be installed in advance is
git
cmake 
make
gcc (or g++ ?) I install both

Read the SCReAM-BW-test-tool.pptx
==============================

> From: Sebastian Moeller <[email-masked]>
> Sent: den 9 juni 2020 16:21
> To: Ingemar Johansson S
> <ingemar.s.johansson=[email-masked]>
> Cc: Jonathan Morton <[email-masked]>; Ingemar Johansson S
> <[email-masked]>; [email-masked]
> Subject: Re: [tsvwg] path forward on L4S issue #16
> 
> Ingemar,
> 
> I have a feeling that this discussion thread is based on a disconnect. I
believe
> that most here agree that Scream is a nice and interesting project worth
> pursuing. The only contentious point is whether scream working well with
> current L4S AQM behavior over a single simulated hop, can make up for the
> apparent lack of testing under even mildly adversarial conditions (aka
real
> internet conditions) of the core L4S implementation and design. If you
have
> made such measurements in the context of scream's development, please
> do not hesitate to present them here.
> 
> 5) unfriendly greedy traffic with less than expected yielding behavior ...
> 
> Data demonstrating L4S immunity against negative behaviour under any of
> these conditions, is IMHO quite helpful for the continuing discussion.
> 
> Best Regards
> 	Sebastian
> 
> 
> 
> > On Jun 9, 2020, at 16:11, Ingemar Johansson S
> <ingemar.s.johansson=[email-masked]> wrote:
> >
> > I urge interested people (if any?) to follow the discussion thread
> > instead to get an understanding what causes the lower throughput with
> L4S.
> > https://mailarchive.ietf.org/arch/msg/tsvwg/FFQH_XG0-
> wbLIt7JRJ33dH8Fyf
> > c/
> >
> > /Ingemar
> >

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 9 Jun 2020 16:56:20 +0200

Hi Ingemar,

> On Jun 9, 2020, at 16:43, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi

	[SM] Good question. But I believe you are taking this as a scream question, while I consider this to be an L4S question. I am fine with scream being a research project and all, not having gone out of its way to test all possible adversarial traffic patterns, after all nobody is forced to use scream.
	With the envisioned roll-out of L4S as the "mandatory" CC of the future of the whole internet, the situation looks quite different IMHO. Mandatory here does not mean that everybody is forced to actively use L4S-aware transport protocols, but that all traffic traversing a L4S hops will suffer the consequences.

> other projects that are more important.

	[SM] +1; please accept my apology, I did not want to diss scream in any way here, I really just want to re-focus the discussion on the issues I see with L4S and its lack of sufficient realistic testing. Again, for scream as mostly a research vehicle this is fine, and asking me to roll up my sleeves and get into testing is good advice. Now, if L4S would be just a research vehicle we would not even have this discussion.

Best Regards
	Sebastian

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 9 Jun 2020 18:01:12 +0300

> On 9 Jun, 2020, at 5:43 pm, Ingemar Johansson S <[email-masked]> wrote:
> 

Because we are looking for cases where SCReAM (and other L4S transports) fail - and we have already found sufficient such cases to raise objections.  We don't need to look further.

The burden of proof is now on you (and the rest of the L4S proponents) to show that these problems have been overcome.    This will require *changes* to L4S, and testing of an implementation of that changed specification.

Sebastian merely listed a few general scenarios where we know L4S does badly.  Those are tests which you should run internally as part of your R&D cycle, and then present results of, in order to give us confidence that L4S can safely be deployed.

 - Jonathan Morton

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Tue, 9 Jun 2020 08:24:36 -0700 (PDT)

> other projects that are more important.

Is there any plan to update RFC8298 to include using L4S w/Scream? 

> /Ingemar

Does this code include the modifications that use ECT(1) marking?
Is it possible to turn on/off that marking so that useful comparision
tests can easily be created?  

For the benifit of the others in this thread, SCReAM is
an experiment as documented by RFC8298, with the following
in section 5 Discussion, a list of items:
o  Support for alternate ECN semantics: This specification adopts the
      proposal in [ALT-BACKOFF] to reduce the congestion window less
      when ECN-based congestion events are detected.  Future work on Low
      Loss, Low Latency for Scalable throughput (L4S) may lead to
      updates in a future document that describes SCReAM support for
      L4S.

-- 
Rod Grimes                                                 [email-masked]

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 9 Jun 2020 18:24:35 +0200

Hi Jonathan,

> On Jun 9, 2020, at 17:01, Jonathan Morton <[email-masked]> wrote:
> 
> Sebastian merely listed a few general scenarios where we know L4S does badly.  

	[SM] Well, to be generous, these are important scenarios where we know way too little about L4S' robustness and reliability. The recent testing indicates that L4S has ample opportunity to improve its performance/safety under these conditions.
	Now, I have a hard time believing that none of the engineers and companies that opted for L4S did this based purely on the obviously overly-optimistic promises in the L4S drafts and the RITE project. So, I would really appreciate if data that was acquired in the process of seeing whether L4S was/is fit for roll-out and implementation in silicon (if that actually happened at all) could be presented here on the list.

Best Regards
	Sebastian

> Those are tests which you should run internally as part of your R&D cycle, and then present results of, in order to give us confidence that L4S can safely be deployed.

	[SM] +1.

Sebastian

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 9 Jun 2020 18:58:12 +0000

Hi
Just for my understanding, do you seek evidence that L4S can be safely
deployed before the experiment actually starts or is the collection of
evidence a part of the experiment?

To take 3GPP networks as an example. I don't expect that initial experiments
will run across the entire internet. And as this experiment carries on I see
a likelihood that RFC3168 ECN capable AQM where they may be will/can be
upgraded to support L4S as well ? Meanwhile one will learn things from the
experiments and will be able to refine congestion control algorithms. Does
it sound all too unrealistic?

This boils down to the two questions that I had in another thread.
1) How widely are  RFC3168 ECN capable AQMs deployed _and_ enabled ?
2) If they are deployed _and_ enabled, can/will they be updated ?

/Ingemar

> Sent: den 9 juni 2020 18:25
> To: Jonathan Morton <[email-masked]>
> Cc: Ingemar Johansson S <[email-masked]>;
> [email-masked]
> >> On 9 Jun, 2020, at 5:43 pm, Ingemar Johansson S
> > Because we are looking for cases where SCReAM (and other L4S
> transports) fail - and we have already found sufficient such cases to
raise
> objections.  We don't need to look further.
> >
> > The burden of proof is now on you (and the rest of the L4S proponents)
to
> show that these problems have been overcome.    This will require
> *changes* to L4S, and testing of an implementation of that changed
> specification.
> >
> 	[SM] Well, to be generous, these are important scenarios where we
> know way too little about L4S' robustness and reliability. The recent
testing
> indicates that L4S has ample opportunity to improve its performance/safety
> under these conditions.
> 	Now, I have a hard time believing that none of the engineers and
> companies that opted for L4S did this based purely on the obviously
overly-
> optimistic promises in the L4S drafts and the RITE project. So, I would
really
> appreciate if data that was acquired in the process of seeing whether L4S
> was/is fit for roll-out and implementation in silicon (if that actually
happened
> at all) could be presented here on the list.
> 
> Best Regards
> 	Sebastian
> 
> 
> > Those are tests which you should run internally as part of your R&D
cycle,
> and then present results of, in order to give us confidence that L4S can
safely
> be deployed.

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 9 Jun 2020 19:11:02 +0000

Hi

Yes, RFC8298 will eventually be updated, it is still an experimental RFC and
the target is to make it standards track in the end. 
There is already an implementation of L4S support in running code
(https://github.com/EricssonResearch/scream) but I don't really see that as
up to the task. More work is needed, it is perhaps possible that one can do
away with the split between the network congestion control part and the rate
control part. One problem that we currently see is that it may back off a
little too easy when we run it in our proprietary 5G network simulator with
L4S support.

The SCReAM BW test tool can be run on both RFC3168 style mode with the -ect
0 option, in which case it does something that resembles ABE. It runs in L4S
mode with the -ect 1 option.

/Ingemar
 
> From: Rodney W. Grimes <[email-masked]>
> Sent: den 9 juni 2020 17:25
> To: Ingemar Johansson S
> Cc: Sebastian Moeller <[email-masked]>; Ingemar Johansson S
> > satisfactory way. But as with most other congestion control work it is
> > work in progress, sometimes that progress is good, sometimes not so
> > good due to other projects that are more important.
> 
> > git clone
> > https://protect2.fireeye.com/v1/url?k=70acc563-2e0c7f0d-70ac85f8-861d4
> > 1abace8-e225af5cf515fe3e&q=1&e=20992ad0-7a3c-4843-b436-
> e719baf77c27&u=
> > https%3A%2F%2Fgithub.com%2FEricssonResearch%2Fscream.git
> > cd code/bw-test-tool
> > cmake .
> > make
> 
tests can
> easily be created?
> 
> For the benifit of the others in this thread, SCReAM is an experiment as
> documented by RFC8298, with the following in section 5 Discussion, a list
of
> items:
> > What needs to be installed in advance is git cmake make gcc (or g++ ?)
> > I install both
> >
> > > that most here agree that Scream is a nice and interesting project
> > > worth pursuing. The only contentious point is whether scream working
> > > well with current L4S AQM behavior over a single simulated hop, can
> > > make up for the apparent lack of testing under even mildly
> > > adversarial conditions (aka
> > real
> > > made such measurements in the context of scream's development,
> > > please do not hesitate to present them here.
> > >
...
> > >
> > > Data demonstrating L4S immunity against negative behaviour under any
> > > of these conditions, is IMHO quite helpful for the continuing
discussion.
> > > >>> "Even though it is more than a magnitude difference in queue
> > > >>> delay between CoDel-ECN and L4S, it is fair to say that these
> > > >>> simple simulations should of course be seen as just a snapshot."
> > > >>

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 9 Jun 2020 21:41:27 +0200

Hi Ingemar,

> On Jun 9, 2020, at 20:58, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi

	[SM] That depends on what you mean with experiment. Yes, I want to see that data BEFORE L4S is awarded IETF RFC status above informational... As far as I understood Gory, the intended experiment for granting L4S experimental RFCs (first) is how to roll-out L4S safely. IMHO describing the intended behaviour for the few scenarios I outlined and then demonstrating that the actual implementation meets those goals, should be an uncontentious "conditio sine qua non", but as this whole process demonstrates this does not seem to be the consensus position.

> it sound all too unrealistic?

	[SM] No, that sounds fine, but you do not need IETF approval for such localized experiments to begin with, it is really the "across the entire internet" part in the L4S RFCs that makes it IMHO a prerequisite to demonsatrate sufficient safety before roll-out. I note that none of the conditions I outline could not be easily tested in a more or less private experiment still spanning the width of the internet, just set-up three test nodes, say, one in Frankfurt, one in Atlanta and one in Tokyo each with an L4S AQM and go wild with testing traffic between them, no RFC required for that. 
	But here we are, working hard to get IETF approval for L4S even before it is clear that L4S can actually deliver its promises under normal internet conditions. 

> 2) If they are deployed _and_ enabled, can/will they be updated ?

	[SM] Just looking at the IPv4/6 "transition" the answer to the second question is, "yes" and "with glacial speed". I can understand the desire for the existing internet to follow one's own idea of optimality, but realistically the existing internet has a lot of momentum and and thing that requires a flag day, pretty much is doomed (unless as clearly beneficial to everybody as the introduction of congestion handling, but L4S' gains are rather modest in comparison).

 Best Regards
	sebastian

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 10 Jun 2020 06:44:18 +0000

Hi Sebastian

Please see inline
/Ingemar

> Sent: den 9 juni 2020 21:41
> Cc: Jonathan Morton <[email-masked]>; [email-masked]
> > On Jun 9, 2020, at 20:58, Ingemar Johansson S
> 	[SM] That depends on what you mean with experiment. Yes, I want
> to see that data BEFORE L4S is awarded IETF RFC status above
informational...
> As far as I understood Gory, the intended experiment for granting L4S
> experimental RFCs (first) is how to roll-out L4S safely. IMHO describing
the
> intended behaviour for the few scenarios I outlined and then demonstrating
> that the actual implementation meets those goals, should be an
> uncontentious "conditio sine qua non", but as this whole process
> demonstrates this does not seem to be the consensus position.

[IJ] That boils down to an assessment how severe the issues you point out
really are, and to some extent it boils  down to my questions #1 and #2
below. 

> 
> >
> > To take 3GPP networks as an example. I don't expect that initial
> > experiments will run across the entire internet. And as this
> > experiment carries on I see a likelihood that RFC3168 ECN capable AQM
> > where they may be will/can be upgraded to support L4S as well ?
> > Meanwhile one will learn things from the experiments and will be able
> > to refine congestion control algorithms. Does it sound all too
unrealistic?
> 
> 	[SM] No, that sounds fine, but you do not need IETF approval for
> such localized experiments to begin with, it is really the "across the
entire
> internet" part in the L4S RFCs that makes it IMHO a prerequisite to
> demonsatrate sufficient safety before roll-out. I note that none of the
> conditions I outline could not be easily tested in a more or less private
> experiment still spanning the width of the internet, just set-up three
test
> nodes, say, one in Frankfurt, one in Atlanta and one in Tokyo each with an
> L4S AQM and go wild with testing traffic between them, no RFC required for
> that.

[IJ] Then one need to define the meaning of "localized experiments" but I
skip that part. I guess it is not IETFs intention to leave up to other SDO's
like 3GPP to do what they like with the ECN bits until IETF has made up its
mind. Or is it ?, this is at least not how I read the message in TSVWG.

> 	But here we are, working hard to get IETF approval for L4S even
> before it is clear that L4S can actually deliver its promises under normal
> internet conditions.
> 
> 
> >
> 	[SM] Just looking at the IPv4/6 "transition" the answer to the
second
> question is, "yes" and "with glacial speed". I can understand the desire
for
> the existing internet to follow one's own idea of optimality, but
realistically
> the existing internet has a lot of momentum and and thing that requires a
> flag day, pretty much is doomed (unless as clearly beneficial to everybody
as
> the introduction of congestion handling, but L4S' gains are rather modest
in
> comparison).

[IJ] OK, I tend to use the this IPv4/6 analogy every now and then, but that
does not at all apply here. 
IPv4 is widely deployed and I don't even use IPv6. I seriously wonder how
widely RFC3168 is deployed and used in AQMs
So here my questions are (repeated):
1) How widely are  RFC3168 ECN capable AQMs deployed _and_ enabled ? 
   IPv4 and RFC3168 ECN capable AQMs seem to have very little in common when
one look at deployment and use, so how commonplace is it that nodes on the
internet have RFC3168 ECN capable AQMs deployed _and_ enabled, 1%, 10% ???? 

2) If they are deployed _and_ enabled, can/will they be updated ?
   You gave your personal view on this matter.

I guess it can be a good exercise for the opponents to the L4S to come with
verifiable answers to these two questions. It is important to know so we
don't waste a lot of energy on a problem that diminishes among all other
problems.

> >>> Sebastian merely listed a few general scenarios where we know L4S
> >>> does
> >> badly.
> >>
> >> indicates that L4S has ample opportunity to improve its
> >> performance/safety under these conditions.
> >> appreciate if data that was acquired in the process of seeing whether
> >> L4S was/is fit for roll-out and implementation in silicon (if that
> >> actually
> > happened

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 10 Jun 2020 09:52:16 +0200

Hi Ingemar,
more below in-line.

> On Jun 10, 2020, at 08:44, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi Sebastian
> 
> Please see inline
> /Ingemar
> 
> 

	[SM] Honestly, I disagree, while we are currently focussing on RFC3168 compatibility this is by far not the only realistic internet scenario in which L4S falls painfully short of its promises. Let me remind you on the abysmal sharing behavior between the two queues at short RTTs. I keep harping on this for the following reasons:

a) Fixing this design error in the dual queue coupled AQM in the transport layer is clearly the wrong place, given the fact that TCP Prague will not be the only transport envisioned to carry ECT(1), as far as I can tell you have made so far no tests in scream to see whether you are affected by the same issue (and truth be told, on an end-user mobile link a certain priority/dominance for real-time video streams might actually be in the users interest, as a user I would prefer some sort of opt-in though).

b) the short RTT/few hop scenario is the only one that has seen some modicum of testing in the L4S effort, and it is also the only one where decreasing the RTT by a few dozens milliseconds will actually translate into significant RTT/jitter reductions.

c) the fact that team L4S constantly tries to kill the messenger and to conflate the DualQ mis-design with RTT-independence and other red herrings, indicates a lack of solid engineering principles, which I for one find disheartening in what, by its own proponents, seems to be designed to be the future of congestion control in the internet.

In almost every corner we looked, the current L4S implementation revealed gaping holes and undesirable behavior, and the L4S team did sped most of its time not fixing the issues, but trying to arfgue the issues away. I see your point #1 as one of these attempts to argue a hard problem away instead of either solving it, or re0-designing L4S to not have that issue in the first place, which, obviously, is my personal subjective opinion.

> mind. Or is it ?, this is at least not how I read the message in TSVWG.

	[SM] I still maintain, that even for internet scale experiments no SDO needs to be involved, unless one attempts to "volunteer" one's own iusers into the experiment. But I consider that to be bad style and questionable. Also such an "experiment" should only be performed once the scenarios I outlined have been tested and the to be rolled-out mechanism survives these tests satisfactorily.
	Heck, with a modicum of cooperation, mobile carriers all over the world could run tests between their respective development labs, giving world-wide coverage without the need to put any usaer in harms way.

> widely RFC3168 is deployed and used in AQMs

	[SM] I see, partly a disconnect on my side, I am not laser focussed on your #1 and #2, as I do not believe that this an exhaustive enumeration of L4S issues that need fixing, I am looking at this more from the perspective of L4S's implicit claim to be the future of congestion control to replace all classic congesyion control. Under that perspective the IPv4/6 analogy seems quite apt to me.

> internet have RFC3168 ECN capable AQMs deployed _and_ enabled, 1%, 10% ???? \

	[SM] From a safety perspective, I still believe this is the wrong question to ask, as it implies a course of action of simply ignoring the existing deployment, trying to assess importance simply be sheer number, which is not a good proxy.

>   You gave your personal view on this matter.

	[SM] You keep saying that like you and team L4S deals in objective truth statements, to which I take offense. Also I believe that the question is ill-posed without adding a time horizon for potential updates.

> verifiable answers to these two questions.

	[SM] I would love that, but I venture a guess, that we will see more personal views on that matter. I have, as you might have noticed, asked repeatedly for cold hard data, and so far we have not seen that much, and most of if it from the SCE effort.

Best Regards
	Sebastian

> It is important to know so we

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 10 Jun 2020 08:35:11 +0000

Hi

Coming back to my questions. Is it possible to answer these first?, then we
can move on to the other questions, isn't this reasonable ?

I mean there should be an interest in getting some indication as regards to
the level of severity to the problem that we try to solve, right ?

> Sent: den 10 juni 2020 09:52
> > On Jun 10, 2020, at 08:44, Ingemar Johansson S
> >>> Just for my understanding, do you seek evidence that L4S can be
> >>> safely deployed before the experiment actually starts or is the
> >>> collection of evidence a part of the experiment?
> >>
> >> see that data BEFORE L4S is awarded IETF RFC status above
> > informational...
> >> intended behaviour for the few scenarios I outlined and then
> >> demonstrating that the actual implementation meets those goals,
> >> should be an uncontentious "conditio sine qua non", but as this whole
> >> process demonstrates this does not seem to be the consensus position.
> >
> > [IJ] That boils down to an assessment how severe the issues you point
> > out really are, and to some extent it boils  down to my questions #1
> > and #2 below.
> 
> 	[SM] Honestly, I disagree, while we are currently focussing on
> RFC3168 compatibility this is by far not the only realistic internet
scenario in
> which L4S falls painfully short of its promises. Let me remind you on the
> abysmal sharing behavior between the two queues at short RTTs. I keep
> harping on this for the following reasons:
> 
> a) Fixing this design error in the dual queue coupled AQM in the transport
> layer is clearly the wrong place, given the fact that TCP Prague will not
be the
> only transport envisioned to carry ECT(1), as far as I can tell you have
made so
> far no tests in scream to see whether you are affected by the same issue
> (and truth be told, on an end-user mobile link a certain
priority/dominance
> for real-time video streams might actually be in the users interest, as a
user I
> would prefer some sort of opt-in though).
> 
> b) the short RTT/few hop scenario is the only one that has seen some
> modicum of testing in the L4S effort, and it is also the only one where
> decreasing the RTT by a few dozens milliseconds will actually translate
into
> significant RTT/jitter reductions.
> 
> c) the fact that team L4S constantly tries to kill the messenger and to
conflate
> the DualQ mis-design with RTT-independence and other red herrings,
> indicates a lack of solid engineering principles, which I for one find
> disheartening in what, by its own proponents, seems to be designed to be
> the future of congestion control in the internet.
> 
> In almost every corner we looked, the current L4S implementation revealed
> gaping holes and undesirable behavior, and the L4S team did sped most of
its
> time not fixing the issues, but trying to arfgue the issues away. I see
your
> point #1 as one of these attempts to argue a hard problem away instead of
> either solving it, or re0-designing L4S to not have that issue in the
first place,
> which, obviously, is my personal subjective opinion.
> 
> 
> >
> >>
> >>>
> >>> experiment carries on I see a likelihood that RFC3168 ECN capable
> >>> AQM where they may be will/can be upgraded to support L4S as well ?
> >>> Meanwhile one will learn things from the experiments and will be
> >>> able to refine congestion control algorithms. Does it sound all too
> > unrealistic?
> >>
> >> demonsatrate sufficient safety before roll-out. I note that none of
> >> the conditions I outline could not be easily tested in a more or less
> >> private experiment still spanning the width of the internet, just
> >> set-up three
> > test
> >> nodes, say, one in Frankfurt, one in Atlanta and one in Tokyo each
> >> with an L4S AQM and go wild with testing traffic between them, no RFC
> >> required for that.
> >
> > [IJ] Then one need to define the meaning of "localized experiments"
> > but I skip that part. I guess it is not IETFs intention to leave up to
> > other SDO's like 3GPP to do what they like with the ECN bits until
> > IETF has made up its mind. Or is it ?, this is at least not how I read
the
> message in TSVWG.
> 
> 	[SM] I still maintain, that even for internet scale experiments no
SDO
> needs to be involved, unless one attempts to "volunteer" one's own iusers
> into the experiment. But I consider that to be bad style and questionable.
> Also such an "experiment" should only be performed once the scenarios I
> outlined have been tested and the to be rolled-out mechanism survives
> these tests satisfactorily.
> 	Heck, with a modicum of cooperation, mobile carriers all over the
> world could run tests between their respective development labs, giving
> world-wide coverage without the need to put any usaer in harms way.
> 
> 
> >
> >> before it is clear that L4S can actually deliver its promises under
> >> normal internet conditions.
> >>
> >>
> >>>
> >> the existing internet has a lot of momentum and and thing that
> >> requires a flag day, pretty much is doomed (unless as clearly
> >> beneficial to everybody
> > as
> > [IJ] OK, I tend to use the this IPv4/6 analogy every now and then, but
> > that does not at all apply here.
> > IPv4 is widely deployed and I don't even use IPv6. I seriously wonder
> > how widely RFC3168 is deployed and used in AQMs
> 
> 	[SM] I see, partly a disconnect on my side, I am not laser focussed
on
> your #1 and #2, as I do not believe that this an exhaustive enumeration of
> L4S issues that need fixing, I am looking at this more from the
perspective of
> L4S's implicit claim to be the future of congestion control to replace all
classic
> congesyion control. Under that perspective the IPv4/6 analogy seems quite
> apt to me.
> 
> 
> >   IPv4 and RFC3168 ECN capable AQMs seem to have very little in common
> > when one look at deployment and use, so how commonplace is it that
> > nodes on the internet have RFC3168 ECN capable AQMs deployed _and_
> > enabled, 1%, 10% ???? \
> 
> 	[SM] From a safety perspective, I still believe this is the wrong
> question to ask, as it implies a course of action of simply ignoring the
existing
> deployment, trying to assess importance simply be sheer number, which is
> not a good proxy.
> 
> >
> 	[SM] You keep saying that like you and team L4S deals in objective
> truth statements, to which I take offense. Also I believe that the
question is
> ill-posed without adding a time horizon for potential updates.
> 
> 
> >
> > I guess it can be a good exercise for the opponents to the L4S to come
> > with verifiable answers to these two questions.
> 
> 	[SM] I would love that, but I venture a guess, that we will see more
> personal views on that matter. I have, as you might have noticed, asked
> repeatedly for cold hard data, and so far we have not seen that much, and
> most of if it from the SCE effort.
> 
> Best Regards
> 	Sebastian
> 
> 
> 
> > don't waste a lot of energy on a problem that diminishes among all
> > other problems.
> >
> >>
> >> Best Regards
> >> 	sebastian
> >>
> >>
> >>>
> >>> /Ingemar
> >>>
> >>>>>> Why don't you run the testing yourself to get the answers you
> >>>>>> look
> >>> for?.
> >>>>>
> >>>> appreciate if data that was acquired in the process of seeing
> >>>> whether L4S was/is fit for roll-out and implementation in silicon
> >>>> (if that actually
> >>> happened

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 10 Jun 2020 15:58:11 +0300

> On 10 Jun, 2020, at 11:35 am, Ingemar Johansson S <[email-masked]> wrote:
> 

Enough for the effects to be visible on about 0.1-1.0% of current Internet traffic - and rising steadily.  That's some combination of endpoints negotiating ECN support, and middleboxes implementing an AQM, but not necessarily both on the same path.  Deployment of ECN-enabled AQM is very high across a small number of forward-thinking ISPs, including a very large one in France and a major one in Norway.

Basically every endpoint out there today implements RFC-3168 ECN, and every Linux endpoint runs an ECN-enabled AQM by default on each Ethernet interface.  If Linux and/or Microsoft had decided to switch it on by default a decade ago, rather than hiding it behind a non-default configuration flag, we would not be having this conversation, because the deployment numbers would be too big and obvious to ignore.  It's been a long, slow chicken-and-egg situation which the major consumer ISPs have essentially refused to help resolve.

But now there are commercial, off-the-shelf CPE products which anyone can buy and install, and which implement the sort of bufferbloat mitigation measures that most ISPs have not.  I've found at least one of those on the physical retail shelf, here in Finland.  I don't have sales numbers, but they must account for most of the deployment on ISPs that haven't deployed it themselves.

> 2) If they are deployed _and_ enabled, can/will they be updated ?

Some can and will.  But some is baked in hardware so would be difficult to update, and some is managed by people/vendors who do not know or care about L4S, so would tend to keep using the old system which is RFC-compliant today.  When was the last time your CPE devices got a firmware update?  So L4S has to be able to cope with the existence of such deployments in at least the medium term.

Frankly, if you believe that tunnels are an intractable problem, then this is just as bad.  So please finally drop this argument; you're not convincing anyone who matters.

 - Jonathan Morton

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 10 Jun 2020 13:59:16 +0000

Hi

Do you have any public sources that confirm the numbers you quote below ?. I
tried to look up data on this but it surely is not easy. 
Which foras are the vendors that manufacture CPEs active in (if any) ?. 

As regards to endpoints implementing RFC3168, do you refer to servers and
clients or something else?. My interpretation is servers and clients and I
don't believe that they are central  to this discussion, or do I miss
something ?. I understand that traffic shaping on outgoing interfaces can be
applied in a Linux host but don't see why they become a problem especially
as there are qdiscs that support dualQ.

/Ingemar

PS. I am not yet trying to convince anybody, I am just trying to find the
answers.

> Sent: den 10 juni 2020 14:58
> Cc: Sebastian Moeller <[email-masked]>; [email-masked]
> > On 10 Jun, 2020, at 11:35 am, Ingemar Johansson S
> Enough for the effects to be visible on about 0.1-1.0% of current Internet
> traffic - and rising steadily.  That's some combination of endpoints
negotiating
> ECN support, and middleboxes implementing an AQM, but not necessarily
> both on the same path.  Deployment of ECN-enabled AQM is very high
> across a small number of forward-thinking ISPs, including a very large one
in
> France and a major one in Norway.
> 
> Basically every endpoint out there today implements RFC-3168 ECN, and
> every Linux endpoint runs an ECN-enabled AQM by default on each Ethernet
> interface.  If Linux and/or Microsoft had decided to switch it on by
default a
> decade ago, rather than hiding it behind a non-default configuration flag,
we
> would not be having this conversation, because the deployment numbers
> would be too big and obvious to ignore.  It's been a long, slow
chicken-and-
> egg situation which the major consumer ISPs have essentially refused to
help
> resolve.
> 
> But now there are commercial, off-the-shelf CPE products which anyone can
> buy and install, and which implement the sort of bufferbloat mitigation
> measures that most ISPs have not.  I've found at least one of those on the
> physical retail shelf, here in Finland.  I don't have sales numbers, but
they
> must account for most of the deployment on ISPs that haven't deployed it
> themselves.
> 
> Some can and will.  But some is baked in hardware so would be difficult to
> update, and some is managed by people/vendors who do not know or care
> about L4S, so would tend to keep using the old system which is RFC-
> compliant today.  When was the last time your CPE devices got a firmware
> update?  So L4S has to be able to cope with the existence of such
> deployments in at least the medium term.
> 
> Frankly, if you believe that tunnels are an intractable problem, then this
is
> just as bad.  So please finally drop this argument; you're not convincing
> anyone who matters.
> 
>  - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 10 Jun 2020 16:34:46 +0200

Hi Ingemar,

to gently push back on some details.

> On Jun 10, 2020, at 15:59, Ingemar Johansson S <[email-masked]> wrote:
> 
> [...]I understand that traffic shaping on outgoing interfaces can be
> [...]

	There seems to be a single out-of-the-mainline-Linux-tree repository (https://github.com/L4STeam/linux) for both the dual queue coupled AQM and TCP Prague. 
	I would not call that prrof of sufficient existence of "qdiscs that support dualQ" to allow Linux system admins to switch over to dualqand I do not see how even inclusion into the mainline kernel* would this solves the issue for currently deployed Linux machines, which often use vendor kernels which do not necessarily track mainline closely, especially for server distributions.
	I would respectfully argue that for safety considerations one should look at the current state of the internet and not potential less problematic states one would like to find the internet in...

Best Regards
	Sebastian

*) As far as I can tell there have been no attempts at upstreaming the dual queue coupled AQM yet, so it is not clear what/if survives the contact with the linux kernel maintainers.

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 10 Jun 2020 17:15:58 +0000

Hi

As regards to dualQ (below), do you see any specific reason why it would not
be possible to upstream (complexity/memory whatever) it or is your argument
that it is just not done yet ?

Also, do you have any comments to my three other questions, please refer to
earlier email in the thread for the context.
1) Do you have any public sources that confirm the numbers you quote below
?. I tried to look up data on this but it surely is not easy. 
2) Which foras are the vendors that manufacture CPEs active in (if any) ?.
3) As regards to endpoints implementing RFC3168, do you refer to servers and
clients or something else?. My interpretation is servers and clients and I
don't believe that they are central  to this discussion, or do I miss
something ?.

/Ingemar

> Sent: den 10 juni 2020 16:35
> > On Jun 10, 2020, at 15:59, Ingemar Johansson S
> > applied in a Linux host but don't see why they become a problem
> > especially as there are qdiscs that support dualQ.
> > [...]
> 
> 	There seems to be a single out-of-the-mainline-Linux-tree repository
> (https://protect2.fireeye.com/v1/url?k=e33cc533-bd9c7f5d-e33c85a8-
> 869a14f4b08c-0ec6a27e7722e722&q=1&e=29721776-06f8-43e4-a1e6-
> 67f0d2c15283&u=https%3A%2F%2Fgithub.com%2FL4STeam%2Flinux) for
> both the dual queue coupled AQM and TCP Prague.
> 	I would not call that prrof of sufficient existence of "qdiscs that
> support dualQ" to allow Linux system admins to switch over to dualqand I
do
> not see how even inclusion into the mainline kernel* would this solves the
> issue for currently deployed Linux machines, which often use vendor
kernels
> which do not necessarily track mainline closely, especially for server
> distributions.
> 	I would respectfully argue that for safety considerations one should
> look at the current state of the internet and not potential less
problematic
> states one would like to find the internet in...
> 
> Best Regards
> 	Sebastian
> 
> 
> *) As far as I can tell there have been no attempts at upstreaming the
dual
> queue coupled AQM yet, so it is not clear what/if survives the contact
with
> the linux kernel maintainers.

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Wed, 10 Jun 2020 13:00:02 -0700 (PDT)

Hello Ingmar,
Inlinde below

Regards,
Rod
> that it is just not done yet ?

It would be a mistake to upstream something into Linux, or FreeBSD or
any other publically distributed software that has not been accepted
as a solution by the IETF, *unless* that code was disabled with
#ifdef that would require a custom compilation.  To do anything less
to protect the internet from potentially harmful software would be
irresposible.

> don't believe that they are central  to this discussion, or do I miss

I do not actually believe your 2, or now 3 questions are actually
the right questions to be asking.  The fact is that RFC3168 and handful
of other AQM RFC's are infact the accepted standards for internet
operation today, and though you can go declare them "non deployed",
you can not declare them non standard, and until you do so one
should considered them deployed.

If you want to base L4S forward progress on the lack of RFC3168 AQM's
then you *MUST* first get the IETF to revoke the standard status of
RFC3168 and all related AQM's.

5 years ago when L4S started that might of been possible, but
given the push in the last 5 years of people actively deploying
RFC3168 and new AQM's based on it that possibility is most likely
no longer possible.  And even if it was I suspect the long tail
would put you 10 years down the road.

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 10 Jun 2020 22:04:41 +0200

Hi Ingemar,

more below in-line.

> On Jun 10, 2020, at 19:15, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi
> 

	[SM] Caveat: I am not involved in the Linux kernel and hence can not predict with any certainty dualQ's likelihood of getting included. BUT as you know I am arguing for some time now, that there is a discrepancy between the dualQ claims and its actual performance. In other words it is not done, and given the responses from the L4S team to this issue (like the 15ms hack in TCP Prague's CC response) I am more and more of the opinion, that this is not simply a case of dualQ just not being finished, but rather that the current dualQ design simply can not meet its promises and someone would need to go back to the drawing board. 

> earlier email in the thread for the context.

	[SM] I snipped these out of my reply since I had nothing meaningful to add to those.

> ?. I tried to look up data on this but it surely is not easy. 

	[SM] I do not know where Jonathan's numbers come from. But https://datatracker.ietf.org/meeting/98/materials/slides-98-maprg-tcp-ecn-experience-with-enabling-ecn-on-the-internet-padma-bhooma-00 has some numbers from Apple, I believe Bob cited these numbers multiple times in the past.
Given the fact that 3gpp contains quite a lot of large carriers maybe that would be a forum to ask for numbers?

> 2) Which foras are the vendors that manufacture CPEs active in (if any) ?.

	[SM] I believe that OpenWrt certainly supports rfc3168 behaviour, and there are CPE that run on modified OpenWrt, so the OpenWrt forum might be a decent starting point?

> don't believe that they are central  to this discussion, or do I miss

	[SM] Well, it is these end-points that are going to suffer, when L4S gets it wrong (when, not if). So these numbers give you an estimate of the potential fall-out area.

Best Regards
	Sebastian

---

From: Ingemar Johansson S <[email-masked]>
Date: Thu, 11 Jun 2020 08:23:32 +0000

Hi

Please see inline [IJ]
/Ingemar

> Sent: den 10 juni 2020 22:05
> > On Jun 10, 2020, at 19:15, Ingemar Johansson S
> > As regards to dualQ (below), do you see any specific reason why it
> > would not be possible to upstream (complexity/memory whatever) it or
> > is your argument that it is just not done yet ?
> 
> 	[SM] Caveat: I am not involved in the Linux kernel and hence can not
> predict with any certainty dualQ's likelihood of getting included. BUT as
you
> know I am arguing for some time now, that there is a discrepancy between
> the dualQ claims and its actual performance. In other words it is not
done,
> and given the responses from the L4S team to this issue (like the 15ms
hack
> in TCP Prague's CC response) I am more and more of the opinion, that this
is
> not simply a case of dualQ just not being finished, but rather that the
current
> dualQ design simply can not meet its promises and someone would need to
> go back to the drawing board.

[IJ] Yes, and understand that there are widely differing views about the
severity of this possible problem, if it really is a problem.

> 
> >
> > Also, do you have any comments to my three other questions, please
> > refer to earlier email in the thread for the context.
> 
> > 1) Do you have any public sources that confirm the numbers you quote
> > below ?. I tried to look up data on this but it surely is not easy.
> 
> 	[SM] I do not know where Jonathan's numbers come from. But
> https://datatracker.ietf.org/meeting/98/materials/slides-98-maprg-tcp-ecn-
> experience-with-enabling-ecn-on-the-internet-padma-bhooma-00 has some
> numbers from Apple, I believe Bob cited these numbers multiple times in
the
> past.

[IJ] OK, that is the same reference that I have found, I was thinking that
there is perhaps something more up to date available? As regards to 3GPP
access then I can very safely say that we can stop worrying, there is AFAIK
no ECN marking (RFC3168 or anything else) in 3GPP base station nodes so
there are no interop concerns here. There is a remote possibility that ECN
marking can be enabled in e.g. industrial Cisco LTE routers I tried this a
while back but had problems with ECN bleaching which is a configuration
matter in 3GPP networks. 

> 	[SM] I believe that OpenWrt certainly supports rfc3168 behaviour,
> and there are CPE that run on modified OpenWrt, so the OpenWrt forum
> might be a decent starting point?

[IJ] OK, thanks, I was of the impression, that OpenWrt is on the table in
the implementation of fq-codel and the like, is this correct?, if that is
true then it perhaps fits bets with L4S issue #17 but perhaps it is relevant
for #16 and #17?. 
Has this extended beyond the early adopter do-it-your-self community?. I
recall from earlier in this long tread that you found off the shelf routers
in stores, were these shipped and ready with RFC3168 support or was an
update necessary to make it RFC3168 capable?.

> 
> > 3) As regards to endpoints implementing RFC3168, do you refer to
> > servers and clients or something else?. My interpretation is servers
> > and clients and I don't believe that they are central  to this
> > discussion, or do I miss something ?.
> 
> 	[SM] Well, it is these end-points that are going to suffer, when L4S
> gets it wrong (when, not if). So these numbers give you an estimate of the
> potential fall-out area.

[IJ] OK, then I understand. I was a bit mislead as I assumed that we
discussed network nodes that do RFC3168 ECN marking, not end hosts that
negotiate ECN. I would like to see these as separate parts of the problem. 

> both
> >> the dual queue coupled AQM and TCP Prague.
> >> not see how even inclusion into the mainline kernel* would this
> >> solves the issue for currently deployed Linux machines, which often
> >> use vendor
> > kernels
> >> which do not necessarily track mainline closely, especially for
> >> server distributions.

---

From: Ingemar Johansson S <[email-masked]>
Date: Thu, 11 Jun 2020 08:51:13 +0000

Hi Rodney 
Please see inline
/Ingemar

> Sent: den 10 juni 2020 22:00
> To: Ingemar Johansson S
> It would be a mistake to upstream something into Linux, or FreeBSD or any
> other publically distributed software that has not been accepted as a
solution
> by the IETF, *unless* that code was disabled with #ifdef that would
require a
> custom compilation.  To do anything less to protect the internet from
> potentially harmful software would be irresposible.

[IJ] I guess a lot of things are upstreamed to the Linux kernel without the
blessing from the IETF, the TCP stack is one example which is ahead of IETF
standardization

> I do not actually believe your 2, or now 3 questions are actually the
right
> questions to be asking.  The fact is that RFC3168 and handful of other AQM
> RFC's are infact the accepted standards for internet operation today, and
> though you can go declare them "non deployed", you can not declare them
> non standard, and until you do so one should considered them deployed.
> 
> If you want to base L4S forward progress on the lack of RFC3168 AQM's then
> you *MUST* first get the IETF to revoke the standard status of
> 5 years ago when L4S started that might of been possible, but given the
push
> in the last 5 years of people actively deploying
> RFC3168 and new AQM's based on it that possibility is most likely no
longer
> possible.  And even if it was I suspect the long tail would put you 10
years
> down the road.

[IJ] My intention with this was to first and foremost get a feeling about
the severity of the problem stated in issue #16. Given what this gives in
hand it is perhaps possible to move forward (or not). 
But we somehow need to see the numbers and hard facts on the table,
otherwise we are stuck in this discussion forever.

> > > > [...]I understand that traffic shaping on outgoing interfaces can
> > > > be applied in a Linux host but don't see why they become a problem
> > > 	I would respectfully argue that for safety considerations one
> > > should look at the current state of the internet and not potential
> > > less
> > problematic

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 11 Jun 2020 12:40:25 +0200

Hi Ingemar,

as usual more below in-line.

> On Jun 11, 2020, at 10:23, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi
> 
> Please see inline [IJ]
> /Ingemar
> 

	[SM] Sorry, not accepting that starving nonECT(1) flows at short RTTs is a severe problem seems to be a position not even team L4S took. It has been demonstrated from early on (it is visible in allmost all L4S papers and in the measurements from team SCE) that the coupling heuristic that is an essential component of dualQ's design simply does not give the required isolation guarantees needed to save flows in the non-LL queue from the consequences of its ~15:1 LL:no-LL queue scheduler. At its core dualQ is a ~16:1 scheduler that by applying a number of heuristics will give rough equivalence between the queues under a number of realistic conditions, unfortunately it fails to do so under a number of different equally realistic conditions. Team L4S went so far to propose an absolute ugly hack in TCP Prague to paper over this mis-design in dualQ, accepting that this is a severe problem.
	So are you really taking the position that this is not really a problem? Especially since, as i mentioned before, L4S really only has seen testing for short RTT/few hop scenarios, and is expected to be rolled-out/used by typically close by CDNs?

> there is perhaps something more up to date available?

	[SM] I believe the problem is, that almost nobody (except maybe for the ripe atlas project) has the required testing nodes deployed required to assess the likelihood/prevalence on CE marking on the internet. Ideally you have a network of widely distributed testing nodes in leaf networks and try to run saturating ECN enabled transfers between them, while recording the number of CE marks per path (pair of testing nodes), nobody in academia really has such a network, but a few major carriers should be able to set something like this up easily, given the big names in support of L4S, I would find it marvelous if these could cooperate to get the CE measurements done, before proceeding with the L4S drafts. 

> As regards to 3GPP

	[SM] I meant asking the 3GPP members to help in setting up an internet wide trawl for CE marked packets, for this leaf/edge or close to the leaf/edge mesurement nodes seem required, and hence operators/carrieres that connect end-users would be in a good position to measure CE mark probabilities, no?

> the implementation of fq-codel and the like, is this correct?,

	[SM] SQM-scripts in OpenWrt allows to choose between cake and fq_codel, but cake can be run in single queue mode. QOS-scripts in OpenWrt IIRC defaults to fq_codel.

> if that is

	[SM] I believe that Pete Heist's recent data indicates that thanks to the birthday paradoxon, stochastic fq as used in fq_codel and cake is not a full remedy for L4S's changed response to CE-marks, so maybe that separation needs to be reverted again.

> update necessary to make it RFC3168 capable?.

	[SM] There is Evenroute's IQrouter, that you can order at amazon, that runs a modified OpenWrt and does use fq (I think it defaults to cake nowadays, and cake is an rfc3168 AQM), then there is the Turris Omnia (and also the MOX I believe) that use fq_codel for their guest networks. And since both of these are OpenWrt based they will default to fq_codel for all interfaces.

> [IJ] OK, then I understand.

	[SM] Please note, that I can not speak for Jonathan, so this was just my opinion on that matter.

> I was a bit mislead as I assumed that we

	[SM] Fair enough. To assess the total risk from L4S's changed CE-response however, we need to look at all these things in integrated form.

Best Regards
	Sebastian

---

From: Ingemar Johansson S <[email-masked]>
Date: Thu, 11 Jun 2020 11:07:00 +0000

Hi
Inline...
/Ingemar

> Sent: den 11 juni 2020 12:40
> > On Jun 11, 2020, at 10:23, Ingemar Johansson S
> >> know I am arguing for some time now, that there is a discrepancy
> >> between the dualQ claims and its actual performance. In other words
> >> it is not
> > done,
> >> dualQ design simply can not meet its promises and someone would need
> >> to go back to the drawing board.
> >
> > [IJ] Yes, and understand that there are widely differing views about
> > the severity of this possible problem, if it really is a problem.
> 
> 	[SM] Sorry, not accepting that starving nonECT(1) flows at short
RTTs
> is a severe problem seems to be a position not even team L4S took. It has
> been demonstrated from early on (it is visible in allmost all L4S papers
and in
> the measurements from team SCE) that the coupling heuristic that is an
> essential component of dualQ's design simply does not give the required
> isolation guarantees needed to save flows in the non-LL queue from the
> consequences of its ~15:1 LL:no-LL queue scheduler. At its core dualQ is a
> ~16:1 scheduler that by applying a number of heuristics will give rough
> equivalence between the queues under a number of realistic conditions,
> unfortunately it fails to do so under a number of different equally
realistic
> conditions. Team L4S went so far to propose an absolute ugly hack in TCP
> Prague to paper over this mis-design in dualQ, accepting that this is a
severe
> problem.
> 	So are you really taking the position that this is not really a
problem?
> Especially since, as i mentioned before, L4S really only has seen testing
for
> short RTT/few hop scenarios, and is expected to be rolled-out/used by
> typically close by CDNs?
>

[IJ] I believe that this discussion belongs to issue #28 so I wont delve
into it in this thread. 

> >> Given the fact that 3gpp contains quite a lot of large carriers maybe
> >> that would be a forum to ask for numbers?
> >
> > [IJ] OK, that is the same reference that I have found, I was thinking
> > that there is perhaps something more up to date available?
> 
> 	[SM] I believe the problem is, that almost nobody (except maybe for
> the ripe atlas project) has the required testing nodes deployed required
to
> assess the likelihood/prevalence on CE marking on the internet. Ideally
you
> have a network of widely distributed testing nodes in leaf networks and
try
> to run saturating ECN enabled transfers between them, while recording the
> number of CE marks per path (pair of testing nodes), nobody in academia
> really has such a network, but a few major carriers should be able to set
> something like this up easily, given the big names in support of L4S, I
would
> find it marvelous if these could cooperate to get the CE measurements
done,
> before proceeding with the L4S drafts.
> 
> 
> > As regards to 3GPP
> > access then I can very safely say that we can stop worrying, there is
> > AFAIK no ECN marking (RFC3168 or anything else) in 3GPP base station
> > nodes so there are no interop concerns here. There is a remote
> > possibility that ECN marking can be enabled in e.g. industrial Cisco
> > LTE routers I tried this a while back but had problems with ECN
> > bleaching which is a configuration matter in 3GPP networks.
> 
> 	[SM] I meant asking the 3GPP members to help in setting up an
> internet wide trawl for CE marked packets, for this leaf/edge or close to
the
> leaf/edge mesurement nodes seem required, and hence operators/carrieres
> that connect end-users would be in a good position to measure CE mark
> probabilities, no?

[IJ] OK, my misunderstanding. I leave this uncommented as I don't personally
know of any such activities. 

> 
> 
> >
> >>
> >>
> >>> 2) Which foras are the vendors that manufacture CPEs active in (if
> >>> any)
> > ?.
> >>
> > [IJ] OK, thanks, I was of the impression, that OpenWrt is on the table
> > in the implementation of fq-codel and the like, is this correct?,
> 
> 	[SM] SQM-scripts in OpenWrt allows to choose between cake and
> fq_codel, but cake can be run in single queue mode. QOS-scripts in OpenWrt
> IIRC defaults to fq_codel.
> 
> 
> > if that is
> > true then it perhaps fits bets with L4S issue #17 but perhaps it is
> > relevant for #16 and #17?.
> 
> 	[SM] I believe that Pete Heist's recent data indicates that thanks
to
> the birthday paradoxon, stochastic fq as used in fq_codel and cake is not
a
> full remedy for L4S's changed response to CE-marks, so maybe that
> separation needs to be reverted again.
> 
> > Has this extended beyond the early adopter do-it-your-self community?.
> > I recall from earlier in this long tread that you found off the shelf
> > routers in stores, were these shipped and ready with RFC3168 support
> > or was an update necessary to make it RFC3168 capable?.
> 
> 	[SM] There is Evenroute's IQrouter, that you can order at amazon,
> that runs a modified OpenWrt and does use fq (I think it defaults to cake
> nowadays, and cake is an rfc3168 AQM), then there is the Turris Omnia (and
> also the MOX I believe) that use fq_codel for their guest networks. And
since
> both of these are OpenWrt based they will default to fq_codel for all
> interfaces.

[IJ] Thanks, looked up the Turris Omnia, it mentions automatic updates with
rationale to protect against cyber threats. I interpret this nice feature as
a possibility to update also for support L4S later on. This is course
pending interest from the developer community. But atleast there are options
to avoid the effects of issue #16 and #17 and avoid some of the issues with
legacy only RFC3168 compatible AQMs, right ?.
Something similar seem to apply to the Evenroute quote "Over-the-air
firmware updates to stay current" .

> >> gets it wrong (when, not if). So these numbers give you an estimate
> >> of the potential fall-out area.
> >
> 	[SM] Please note, that I can not speak for Jonathan, so this was
just
> my opinion on that matter.
> 
> 
> > I was a bit mislead as I assumed that we discussed network nodes that
> > do RFC3168 ECN marking, not end hosts that negotiate ECN. I would like
> > to see these as separate parts of the problem.
> 
> 	[SM] Fair enough. To assess the total risk from L4S's changed CE-
> response however, we need to look at all these things in integrated form.

[IJ] Sure, but it is still important to keep these things separate, after
all one can alter the argument and say that end hosts can update to use L4S
with BBRv2 or Prague..

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 11 Jun 2020 13:55:42 +0200

Hi Ingemar,

> On Jun 11, 2020, at 13:07, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi
> Inline...
> /Ingemar
> 

	[SM] Fair enough, now whether #28 will ever see a discussion at all is a different question.

> know of any such activities. 

	[SM] Nor do I, but if exact numbers of rfc3168 AQMs in the internet would be a potential major hurdle for my project, and I had major ISPs/mobile carriers in my team, this is an activity I would start... According to Bob, Ericsson, Sprint, Google, Nokia Networks, AT&T, Vodafone are all behind L4S so getting these numbers fresh out of the internet should not be an insurmountable obstacle, no?

> legacy only RFC3168 compatible AQMs, right ?.

	[SM] Yes, both turris and evenroute are good players that offer new firmwares quite often that fix issues, so are in a position to also do this for L4S. Mind you that response could well be to bleach ECT(1) unconditionally... Because we are putting the cart before the horse here, until it is demonstrated that L4S overall improves things bleaching ECT(1) seems to be the safer position, the onus really is on L4S as the new kid on the block, I would say...

> firmware updates to stay current" .

	[SM] +1; both vendors are really good at keeping their promises about updates, but both do not force updates on their users, so having the machinery for automatic OTA updates is nice, but it will not guarantee that all deployed instances actually partake in that service.
	And then there is the long-tail issue with OpenWrt, people on older less beefy routers often see to stick to oudated versions since newer versions require more resources and can in some cases simply not run on old devices, so there will be legacy OpenWrt deployments in the field that require a fork-lift updata (CPE replacement) that is hard to schedule from outside, no? (This does not affect turris nor evenroute, as far as I understanf both still support all their ever deployed devices, but that is IMHO just a question of time until they will EOL/EOS).

Best Regards
	Sebastian

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 11 Jun 2020 16:03:55 +0300

> On 11 Jun, 2020, at 11:23 am, Ingemar Johansson S <[email-masked]> wrote:
> 

Shipped and ready.  They're using essentially the same underlying mechanisms as OpenWRT, just with a more refined user configuration interface, which includes a setup wizard which walks the user through choosing appropriate shaper parameters to go with the AQM.

In fact I understand quite a few commercial retail products actually run OpenWRT off the shelf now, just with a custom user interface.  All of these implement RFC-3168, but may require hand configuration to make it effective.  I'm talking specifically about a subset of these which mention Smart Queue Management or an equivalent term on the retail packaging, and have therefore made this feature easily accessible to the end user.

As Sebastian mentioned, the IQrouter fits in this category.  It is an outstanding example which actually probes the line over a period of several days, to determine a shaping schedule which matches diurnal variations in line capacity.  Under the covers, it is OpenWRT with some customisations, running on a normal CPE wifi router.

Understand that products like this are needed only because most ISPs and modem vendors have made absolutely no effort to tackle bufferbloat at more appropriate locations in the network - not even using packet-dropping, ECN-ignorant AQMs.  I note that ISPs' voices, as opposed to those of equipment vendors belonging to one particular industry organisation, were conspicuously absent from those calling for L4S to move forward here.

 - Jonathan Morton

---

From: Ingemar Johansson S <[email-masked]>
Date: Thu, 11 Jun 2020 13:26:12 +0000

Hi Jonathan 

As mentioned in response to Sebastian. There is a possibility that these
routers can be updated to support L4S later on so this mean that there is a
possible way out of issue #16 at least for these devices. 
Of course it takes a look in a well polished crystal ball to see if that
will happen.

Regards
/Ingemar

> Sent: den 11 juni 2020 15:04
> > On 11 Jun, 2020, at 11:23 am, Ingemar Johansson S
> Shipped and ready.  They're using essentially the same underlying
> mechanisms as OpenWRT, just with a more refined user configuration
> interface, which includes a setup wizard which walks the user through
> choosing appropriate shaper parameters to go with the AQM.
> 
> In fact I understand quite a few commercial retail products actually run
> OpenWRT off the shelf now, just with a custom user interface.  All of
these
> implement RFC-3168, but may require hand configuration to make it
> effective.  I'm talking specifically about a subset of these which mention
> Smart Queue Management or an equivalent term on the retail packaging,
> and have therefore made this feature easily accessible to the end user.
> 
> As Sebastian mentioned, the IQrouter fits in this category.  It is an
> outstanding example which actually probes the line over a period of
several
> days, to determine a shaping schedule which matches diurnal variations in
> line capacity.  Under the covers, it is OpenWRT with some customisations,
> running on a normal CPE wifi router.
> 
> Understand that products like this are needed only because most ISPs and
> modem vendors have made absolutely no effort to tackle bufferbloat at
> more appropriate locations in the network - not even using
packet-dropping,
> ECN-ignorant AQMs.  I note that ISPs' voices, as opposed to those of
> equipment vendors belonging to one particular industry organisation, were
> conspicuously absent from those calling for L4S to move forward here.
> 
>  - Jonathan Morton

---

From: Ingemar Johansson S <[email-masked]>
Date: Thu, 11 Jun 2020 13:30:00 +0000

OK, thanks. 
I believe that I will now close this thread from my side.

/Ingemar

> Sent: den 11 juni 2020 13:56
> > On Jun 11, 2020, at 13:07, Ingemar Johansson S
> >>>>> would not be possible to upstream (complexity/memory whatever) it
> >>>>> or is your argument that it is just not done yet ?
> >>>>
> >>>> 	[SM] Caveat: I am not involved in the Linux kernel and hence can
> >>>> not predict with any certainty dualQ's likelihood of getting
included.
> >>>> BUT as
> >>>> dualQ design simply can not meet its promises and someone would
> >>>> need to go back to the drawing board.
> >>>
> >> is a severe problem seems to be a position not even team L4S took. It
> >> has been demonstrated from early on (it is visible in allmost all L4S
> >> papers
> > and in
> >> the measurements from team SCE) that the coupling heuristic that is
> >> an essential component of dualQ's design simply does not give the
> >> required isolation guarantees needed to save flows in the non-LL
> >> queue from the consequences of its ~15:1 LL:no-LL queue scheduler. At
> >> its core dualQ is a
> >> ~16:1 scheduler that by applying a number of heuristics will give
> >> rough equivalence between the queues under a number of realistic
> >> conditions, unfortunately it fails to do so under a number of
> >> different equally
> > realistic
> >> conditions. Team L4S went so far to propose an absolute ugly hack in
> >> TCP Prague to paper over this mis-design in dualQ, accepting that
> >> this is a
> > severe
> >> problem.
> > [IJ] I believe that this discussion belongs to issue #28 so I wont
> > delve into it in this thread.
> 
> 	[SM] Fair enough, now whether #28 will ever see a discussion at all
is
> a different question.
> >>>>> 1) Do you have any public sources that confirm the numbers you
> >>>>> quote below ?. I tried to look up data on this but it surely is not
easy.
> >>>> https://datatracker.ietf.org/meeting/98/materials/slides-98-maprg-t
> >>>> cp
> >>>> -ecn-
> >>>> Given the fact that 3gpp contains quite a lot of large carriers
> >>>> maybe that would be a forum to ask for numbers?
> >>>
> >>> [IJ] OK, that is the same reference that I have found, I was
> >>> thinking that there is perhaps something more up to date available?
> >>
> >> to run saturating ECN enabled transfers between them, while recording
> >> the number of CE marks per path (pair of testing nodes), nobody in
> >> academia really has such a network, but a few major carriers should
> >> be able to set something like this up easily, given the big names in
> >> support of L4S, I
> > would
> >>> access then I can very safely say that we can stop worrying, there
> >>> is AFAIK no ECN marking (RFC3168 or anything else) in 3GPP base
> >>> station nodes so there are no interop concerns here. There is a
> >>> remote possibility that ECN marking can be enabled in e.g.
> >>> industrial Cisco LTE routers I tried this a while back but had
> >>> problems with ECN bleaching which is a configuration matter in 3GPP
> networks.
> >>
> >> leaf/edge mesurement nodes seem required, and hence
> >> operators/carrieres that connect end-users would be in a good
> >> position to measure CE mark probabilities, no?
> >
> > [IJ] OK, my misunderstanding. I leave this uncommented as I don't
> > personally know of any such activities.
> 
> 	[SM] Nor do I, but if exact numbers of rfc3168 AQMs in the internet
> would be a potential major hurdle for my project, and I had major
> ISPs/mobile carriers in my team, this is an activity I would start...
According to
> Bob, Ericsson, Sprint, Google, Nokia Networks, AT&T, Vodafone are all
> behind L4S so getting these numbers fresh out of the internet should not
be
> an insurmountable obstacle, no?
> 
> 
> >
> >>
> >>
> >>>
> >>>>
> >>>>
> >>>> and there are CPE that run on modified OpenWrt, so the OpenWrt
> >>>> forum might be a decent starting point?
> >>>
> >>> [IJ] OK, thanks, I was of the impression, that OpenWrt is on the
> >>> table in the implementation of fq-codel and the like, is this
> >>> correct?,
> >>
> >> fq_codel, but cake can be run in single queue mode. QOS-scripts in
> >> OpenWrt IIRC defaults to fq_codel.
> >>
> >>
> >>> if that is
> >>> I recall from earlier in this long tread that you found off the
> >>> shelf routers in stores, were these shipped and ready with RFC3168
> >>> support or was an update necessary to make it RFC3168 capable?.
> >>
> >> that runs a modified OpenWrt and does use fq (I think it defaults to
> >> cake nowadays, and cake is an rfc3168 AQM), then there is the Turris
> >> Omnia (and also the MOX I believe) that use fq_codel for their guest
> >> networks. And
> > since
> > [IJ] Thanks, looked up the Turris Omnia, it mentions automatic updates
> > with rationale to protect against cyber threats. I interpret this nice
> > feature as a possibility to update also for support L4S later on. This
> > is course pending interest from the developer community. But atleast
> > there are options to avoid the effects of issue #16 and #17 and avoid
> > some of the issues with legacy only RFC3168 compatible AQMs, right ?.
> 
> 	[SM] Yes, both turris and evenroute are good players that offer new
> firmwares quite often that fix issues, so are in a position to also do
this for
> L4S. Mind you that response could well be to bleach ECT(1)
unconditionally...
> Because we are putting the cart before the horse here, until it is
> demonstrated that L4S overall improves things bleaching ECT(1) seems to be
> the safer position, the onus really is on L4S as the new kid on the block,
I
> would say...
> 	[SM] +1; both vendors are really good at keeping their promises
> about updates, but both do not force updates on their users, so having the
> machinery for automatic OTA updates is nice, but it will not guarantee
that all
> deployed instances actually partake in that service.
> 	And then there is the long-tail issue with OpenWrt, people on older
> less beefy routers often see to stick to oudated versions since newer
> versions require more resources and can in some cases simply not run on
old
> devices, so there will be legacy OpenWrt deployments in the field that
> require a fork-lift updata (CPE replacement) that is hard to schedule from
> outside, no? (This does not affect turris nor evenroute, as far as I
understanf
> both still support all their ever deployed devices, but that is IMHO just
a
> question of time until they will EOL/EOS).
> 
> 
> Best Regards
> 	Sebastian
> 
> 
> >
> >>
> >>
> >>>
> >>>>
> >>>>> servers and clients or something else?. My interpretation is
> >>>>> servers and clients and I don't believe that they are central  to
> >>>>> this discussion, or do I miss something ?.
> >>>>
> >>>> 	[SM] Well, it is these end-points that are going to suffer, when
> >>>> L4S gets it wrong (when, not if). So these numbers give you an
> >>>> estimate of the potential fall-out area.
> >>>
> >>> I was a bit mislead as I assumed that we discussed network nodes
> >>> that do RFC3168 ECN marking, not end hosts that negotiate ECN. I
> >>> would like to see these as separate parts of the problem.
> >>
> > [IJ] Sure, but it is still important to keep these things separate,
> > after all one can alter the argument and say that end hosts can update
> > to use L4S with BBRv2 or Prague..

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Thu, 11 Jun 2020 15:33:16 -0500

For what it's worth ...

> [email-masked]> wrote:

snipped down to

> below
> https://datatracker.ietf.org/meeting/98/materials/slides-98-maprg-tcp-ecn-experience-with-enabling-ecn-on-the-internet-padma-bhooma-00
> has some numbers from Apple, I believe Bob cited these numbers multiple
> times in the past.

I have my suspicions about the likelihood of getting numbers from 3GPP
carriers, but I note that the 3GPP-IETF coordination team is becoming more
active. If someone wants to ask formally, I suppose sending a liaison
statement to 3GPP before the 3GPP plenary meetings in July would be one way
to do that.

If TSVWG thinks that's attractive, I believe the TSVWG co-chairs know more
about that process than I do, so I'll let them take it from here.

Most of my experience with asking for information like this from wireless
operators was actually with GSMA, not 3GPP itself, during and after the IAB
MARNEW workshop (https://www.iab.org/activities/workshops/marnew/ ) in
2015, and my understanding was that operators weren't willing to disclose
operational measurements to other operators, let along anyone else, because
of concerns that competitors would use that information to attract
customers away from the disclosing operator.

But maybe this could end differently in 2020.

Best,

Spencer

---

From: "Holland, Jake" <[email-masked]>
Date: Mon, 15 Jun 2020 18:35:48 +0000

Hi Ingemar,

I posted some summary numbers from some observations I made here:
https://mailarchive.ietf.org/arch/msg/tsvwg/2tbRHphJ8K_CE6is9n7iQy-VAZM/

I didn't say anything about France or Norway, but the numbers I have
seen were roughly in line with the numbers Jonathan gave.

The "growing" claim also looks tentatively well-supported, because I've
seen 2 new ASNs since then with a high count of CE-marking paths.

HTH.

Best regards,
Jake

On 6/10/20, 6:59 AM, "Ingemar Johansson S" <ingemar.s.johansson=[email-masked]> wrote:

Hi

/Ingemar

negotiating
in
default a
we
chicken-and-
help
they
is

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 17 Jun 2020 13:20:05 +0000

Hi Jake

And thanks for the input. 
Given all the input sofar my impression is that It is difficult to say how serious issue 16 really is. 
Surely there is a possibility that L4S flows can get an potentially unfair advantage over a congested node that is  RFC3168 capable (and ECN enabled). The question is here when things are deemed as unfair and when starvation is a fact, opinions differ. 
Another question is to how large extent there will be legacy RFC3168 ECN capable (and ECN enabled ) bottlenecks that will "never" be updated or replaced. This question is currently difficult to answer. 

Yet another aspect is how successful it will be to implement RFC3168 bottleneck detection e.g. in TCP Prague, I am quite confident that these will not always be successful. But then, given the current deployment situation and possibility to update nodes in the networks (OpenWrt routers appear to support automatic updates) I am not convinced that these need to be 100% bulletproof either.

/Ingemar

> From: Holland, Jake <[email-masked]>
> Sent: den 15 juni 2020 20:36
> To: Ingemar Johansson S
> <ingemar.s.johansson=[email-masked]>; Jonathan Morton
> <[email-masked]>
> I didn't say anything about France or Norway, but the numbers I have seen
> were roughly in line with the numbers Jonathan gave.
> 
> The "growing" claim also looks tentatively well-supported, because I've seen
> 2 new ASNs since then with a high count of CE-marking paths.
> 
> HTH.
> 
> Best regards,
> Jake
> 
> On 6/10/20, 6:59 AM, "Ingemar Johansson S"
> Do you have any public sources that confirm the numbers you quote below
> clients or something else?. My interpretation is servers and clients and I don't
> believe that they are central  to this discussion, or do I miss something ?. I
> understand that traffic shaping on outgoing interfaces can be applied in a
> Linux host but don't see why they become a problem especially as there are
> qdiscs that support dualQ.
> 
> /Ingemar
> 
> > Enough for the effects to be visible on about 0.1-1.0% of current
> > Internet traffic - and rising steadily.  That's some combination of
> > endpoints
> negotiating
> > every Linux endpoint runs an ECN-enabled AQM by default on each
> > Ethernet interface.  If Linux and/or Microsoft had decided to switch
> > it on by
> default a
> > But now there are commercial, off-the-shelf CPE products which anyone
> > can buy and install, and which implement the sort of bufferbloat
> > mitigation measures that most ISPs have not.  I've found at least one
> > of those on the physical retail shelf, here in Finland.  I don't have
> > sales numbers, but
> they
> > must account for most of the deployment on ISPs that haven't deployed
> > it themselves.
> >
> > Some can and will.  But some is baked in hardware so would be
> > difficult to update, and some is managed by people/vendors who do not
> > know or care about L4S, so would tend to keep using the old system
> > which is RFC- compliant today.  When was the last time your CPE
> > devices got a firmware update?  So L4S has to be able to cope with the
> > existence of such deployments in at least the medium term.
> >
> > just as bad.  So please finally drop this argument; you're not
> > convincing anyone who matters.
> >
> >  - Jonathan Morton
>

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 17 Jun 2020 15:26:52 +0200

Hi Ingemar,

more below in-line.

> On Jun 17, 2020, at 15:20, Ingemar Johansson S <ingemar.s.johansson=[email-masked]> wrote:
> 
> Hi Jake
> 
> Yet another aspect is how successful it will be to implement RFC3168 bottleneck detection e.g. in TCP Prague, I am quite confident that these will not always be successful.

	[SM] +1

> But then, given the current deployment situation and possibility to update nodes in the networks (OpenWrt routers appear to support automatic updates)

	[SM] Please accept my apology for not being clear on that topic in the past. Stock OpenWrt does not offer automatic updates, not even notification of update availability at all. There are some vendors, like Evenroute and Turris, that base their own products on top of OpenWrt that do offer automatic updates and as far as I can tell also notifications. But I have no reliable numbers on the prevalence of either stock OpenWrt or update-enabled versions in the field. I also add that a tiny bit of research should allow to confirm this.

> I am not convinced that these need to be 100% bulletproof either.

	[SM] 100% is not achievable, but what we currently have is an effort just designed to get past the objections int the WG instead of actually trying to tackle the issue for real.

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 17 Jun 2020 16:59:04 +0300

> On 17 Jun, 2020, at 4:20 pm, Ingemar Johansson S <[email-masked]> wrote:
> 

When questions such as these are "difficult to answer", the correct design choice is to assume pessimistically, unless and until the question can be answered more definitively.  Therefore, the L4S design must account for the possibility of RFC-3168 AQMs in the network, and actively avoid unfairness in that case.

> Yet another aspect is how successful it will be to implement RFC3168 bottleneck detection e.g. in TCP Prague, I am quite confident that these will not always be successful.

I think we can safely say that the detection mechanism currently in TCP Prague is not successful.  David Black also pointed out that this would always be a fragile mechanism, difficult to implement correctly, and thus likely to be implemented wrongly by other transports even if a workable version was somehow developed.

With these facts established, I do not see how L4S can proceed without changing the signalling mechanism it uses.  This is something we intuitively realised more than a year ago, when the SCE project was started, but now we have actual evidence to back up that intuition.

 - Jonathan Morton

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 17 Jun 2020 14:08:41 +0000

Hi
Sebastian 
Please see inline
/Ingemar

> Sent: den 17 juni 2020 15:27
> To: Ingemar Johansson S
> Cc: Holland, Jake <[email-masked]>; Jonathan Morton
> <[email-masked]>; Ingemar Johansson S
> > On Jun 17, 2020, at 15:20, Ingemar Johansson S
> > Surely there is a possibility that L4S flows can get an potentially unfair
> advantage over a congested node that is  RFC3168 capable (and ECN
> enabled). The question is here when things are deemed as unfair and when
> starvation is a fact, opinions differ.
> > Another question is to how large extent there will be legacy RFC3168 ECN
> capable (and ECN enabled ) bottlenecks that will "never" be updated or
> replaced. This question is currently difficult to answer.
> >
> > Yet another aspect is how successful it will be to implement RFC3168
> bottleneck detection e.g. in TCP Prague, I am quite confident that these will
> not always be successful.
> 
> 	[SM] +1
> 
> 
> > But then, given the current deployment situation and possibility to
> > update nodes in the networks (OpenWrt routers appear to support
> > automatic updates)
> 
> 	[SM] Please accept my apology for not being clear on that topic in the
> past. Stock OpenWrt does not offer automatic updates, not even notification
> of update availability at all. There are some vendors, like Evenroute and
> Turris, that base their own products on top of OpenWrt that do offer
> automatic updates and as far as I can tell also notifications. But I have no
> reliable numbers on the prevalence of either stock OpenWrt or update-
> enabled versions in the field. I also add that a tiny bit of research should allow
> to confirm this.

[IJ] Sounds like a contradiction here to me. I was asking for a list of off the shelf routers with OpenWrt support, all these seem to support automatic updates of course limited by some EOL constraints. To me this seems like what the average John or Jane Smith would consider to buy. On top of this we have the few enthusiastic do-it-yourself people who implement OpenWrt in Raspberry PI but that is of lesser interest because a) they are very few and b) they are in the forefront and are thus likely to upgrade pretty fast. 

> 	[SM] 100% is not achievable, but what we currently have is an effort
> just designed to get past the objections int the WG instead of actually trying
> to tackle the issue for real.
> 
> >
> > /Ingemar
> >
> >> https://protect2.fireeye.com/v1/url?k=d34bc422-8deb7e4c-d34b84b9-
> 8660
> >> 38973a15-6559e7e079a7505b&q=1&e=7a17ee77-c385-4248-836a-
> 193e64d78d5c&
> >>
> u=https%3A%2F%2Fmailarchive.ietf.org%2Farch%2Fmsg%2Ftsvwg%2F2tbRH
> phJ8
> >> K_CE6is9n7iQy-
> >> VAZM/
> >>
> >> Do you have any public sources that confirm the numbers you quote
> >> As regards to endpoints implementing RFC3168, do you refer to servers
> >> and clients or something else?. My interpretation is servers and
> >> clients and I don't believe that they are central  to this
> >> discussion, or do I miss something ?. I understand that traffic
> >> shaping on outgoing interfaces can be applied in a Linux host but
> >> don't see why they become a problem especially as there are qdiscs that
> support dualQ.
> >>
> >> /Ingemar
> >>
> >> PS. I am not yet trying to convince anybody, I am just trying to find
> >> the answers.
> >>
> >>
> >>>> 1) How widely are  RFC3168 ECN capable AQMs deployed _and_
> enabled
> >> ?
> >>>
> >>> ECN support, and middleboxes implementing an AQM, but not
> >>> necessarily both on the same path.  Deployment of ECN-enabled AQM is
> >>> very high across a small number of forward-thinking ISPs, including
> >>> a very large one
> >> in
> >>> Basically every endpoint out there today implements RFC-3168 ECN,
> >>> and every Linux endpoint runs an ECN-enabled AQM by default on each
> >>> would not be having this conversation, because the deployment
> >>> numbers would be too big and obvious to ignore.  It's been a long,
> >>> slow
> >> chicken-and-
> >>> But now there are commercial, off-the-shelf CPE products which
> >>> anyone can buy and install, and which implement the sort of
> >>> bufferbloat mitigation measures that most ISPs have not.  I've found
> >>> at least one of those on the physical retail shelf, here in Finland.
> >>> I don't have sales numbers, but
> >> they
> >>> must account for most of the deployment on ISPs that haven't
> >>> deployed it themselves.
> >>>
> >>> difficult to update, and some is managed by people/vendors who do
> >>> not know or care about L4S, so would tend to keep using the old
> >>> system which is RFC- compliant today.  When was the last time your
> >>> CPE devices got a firmware update?  So L4S has to be able to cope
> >>> with the existence of such deployments in at least the medium term.
> >>>

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 17 Jun 2020 16:16:05 +0200

Hi Ingemar,

more below in-line.

> On Jun 17, 2020, at 16:08, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi
> Sebastian 
> Please see inline
> /Ingemar
> 
> [IJ] Sounds like a contradiction here to me.

	[SM] Why, it is an clarification.

> I was asking for a list of off the shelf routers with OpenWrt support, all these seem to support automatic updates of course limited by some EOL constraints. To me this seems like what the average John or Jane Smith would consider to buy. On top of this we have the few enthusiastic do-it-yourself people who implement OpenWrt in Raspberry PI but that is of lesser interest because a) they are very few and b) they are in the forefront and are thus likely to upgrade pretty fast. 

	[SM] I disagree, we have no numbers to figure out whether assumption a) is true*, and for b) we will first need to demonstrate that any change is actually desirable. L4S is severely lacking in b) so far.

Best Regards
	Sebastian

*) I doubt this is correct, as quite a number of OpenWrt users stay on older releases as newer versions will not run well on under-powered devices. So there is going to be a long tail of legacy versions even among OpenWrt users. I base this assessment on supporting users on the OpenWrt forum.

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Wed, 17 Jun 2020 07:19:12 -0700 (PDT)

> With these facts established, I do not see how L4S can proceed without changing the signalling mechanism it uses.  This is something we intuitively realised more than a year ago, when the SCE proje…

And I would like to add to that, this is evidence the L4S team itself
should of identified, characterized and docuemented years ago.  I
see no reason that it took team SCE to disect the failure modes
of L4S so that they could be addressed.

IMHO the fact that L4S is years into the process and only done
positive test results presentations leads one to believe it
is simply being pushed through the process with little to no
care about the actual internet community.

The proponents seem to genuily be a) unwilling to acknowledge
that there a problems they should be working on, and b) presenting
that L4S is ready for internet deployment inlight of the fact
that data showing clear and present problems exist.

---

From: Wesley Eddy <[email-masked]>
Date: Wed, 17 Jun 2020 12:23:56 -0400

> care about the actual internet community.

I don't suspect that it will soothe you, but will just say that I 
volunteer as a co-chair and shepherd this particular work out of care 
for the "actual internet community" you mention.  The desire to 
experimentally roll-out low-latency services is aimed at serving that 
community.

It has not been recklessly "pushed through the process" as you are 
concerned with.  There are itemized issues to be resolved in order to 
have confidence that this can be safely deployed (and if more emerge, 
they can be dispositioned and added if needed).  This particular thread 
is about agreeing on a path forward on the one labelled "issue #16" in 
the tracker.  Suggestions and discussion about that are good and 
welcome; repeatedly accusing others of bad intentions is not.

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 17 Jun 2020 16:27:21 +0000

Hi

OK, I see that this discussion is only going in circles, nearly giving up on
this.
So far I see only one firm evidence that RFC3168 AQMs are deployed and
enabled and these are related to the OpenWrt project, manifested in a few
upgradeable devices (and some not upgradeable) that can be bought in retail
stores. The rest is unconfirmed.

This leads me to a look at the time line for L4S and how it correlates with
the fq-codel work and as a consequence the OpenWrt project.

1) L4S was first brought up late 2013-early 2014 in TSVWG. The ECN
experimentation draft (now RFC8311) was a draft -00 in sept 2016 and I know
that this draft was preceeded by a discussion in TSVWG before that 

2) The FQ-codel work was finalized early 2016, not quite a perfect overlap
with the 00 version of the ECN experimentation work but definitely an
overlap with the preceeding L4S and dualQ discussions and references to the
RITE project which happened around 2014-2015.

With that in mind, shouldn't it have been on the table to consider the
possible future existence of L4S in the different realizations of fq-codel
(and OpenWrt) development? 

Issue #16 is pretty much a consequence of the above timeline, I see that
there was ample time to react but that reaction did not come until 2019 (or
late 2018?), was any assessment done as regards to how L4S could coexist
with fq-codel before the debate started in TSVWG?.

/Ingemar

> Sent: den 17 juni 2020 16:19
> Cc: Ingemar Johansson S <[email-masked]>; Ingemar
> Johansson S <ingemar.s.johansson=[email-masked]>;
> [email-masked]
> > > On 17 Jun, 2020, at 4:20 pm, Ingemar Johansson S
> > > Given all the input sofar my impression is that It is difficult to say
how
> serious issue 16 really is.
> > When questions such as these are "difficult to answer", the correct
design
> choice is to assume pessimistically, unless and until the question can be
> answered more definitively.  Therefore, the L4S design must account for
the
> possibility of RFC-3168 AQMs in the network, and actively avoid unfairness
in
> that case.
> > I think we can safely say that the detection mechanism currently in TCP
> Prague is not successful.  David Black also pointed out that this would
always
> be a fragile mechanism, difficult to implement correctly, and thus likely
to be
> implemented wrongly by other transports even if a workable version was
> somehow developed.
> >
> > With these facts established, I do not see how L4S can proceed without
> changing the signalling mechanism it uses.  This is something we
intuitively
> realised more than a year ago, when the SCE project was started, but now
> we have actual evidence to back up that intuition.
> 
should of
> identified, characterized and docuemented years ago.  I see no reason that
it
> took team SCE to disect the failure modes of L4S so that they could be
> addressed.
> 
> IMHO the fact that L4S is years into the process and only done positive
test
> results presentations leads one to believe it is simply being pushed
through
> the process with little to no care about the actual internet community.
> 
> The proponents seem to genuily be a) unwilling to acknowledge that there a
> problems they should be working on, and b) presenting that L4S is ready
for
> internet deployment inlight of the fact that data showing clear and
present
> problems exist.

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 17 Jun 2020 19:54:26 +0300

> On 17 Jun, 2020, at 7:27 pm, Ingemar Johansson S <[email-masked]> wrote:
> 

Codel and its progeny were developed on the assumption that RFC-3168 was not going to be obliterated by an incompatible protocol overlaid upon the same codepoints.  A reasonable assumption, given that RFC-3168 was well established as an RFC at that time and widely implemented (if not very widely used in practice), and that DCTCP was explicitly known and accepted to be unsafe to use outside of controlled environments.

And I have to say, Codel represented a genuine step forward in the state of the art for AQM.  It is far easier to configure correctly than most of its predecessors, and does not over-react to brief bursts of traffic (ie. shorter than the estimated RTT), unlike most RED derivatives.  I think it's entirely fair to credit the present increase in RFC-3168 ECN deployment to Codel's existence.  I note in passing that Codel was explicitly designed to match TCP AIMD behaviour, not high-fidelity 1/p responses.

IIRC, when L4S first appeared, the idea of deploying DCTCP over the Internet was rightly dismissed as being completely crackpot.  So everyone in the Codel community simply ignored the RITE project as irrelevant, when it became clear that was the approach they were taking.  Unfortunately, the IETF took them more seriously, and now here we are.

If and when the L4S team manage to put forward a workable solution to the basic problem, I'm sure we will be able to have a more fruitful discussion.  For the time being, we can only eliminate certain classes of solution as unworkable.

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 17 Jun 2020 19:10:38 +0200

Hi Ingemar,

more below in-line...

> On Jun 17, 2020, at 18:27, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi
> 

	The ACM codel paper is from 2012... And fq_codel was deployed well before the matching RFC was published by the IETF. By virtue of being rdf3168 compliant deploy first standardize later was a viable way forward for codel/fq_codel. If only L4S would have selected an incrementally roll-out-able path, we would not be here right now.

> (and OpenWrt) development? 

	[SM] -ENOTIMEMACHINE, at the current time, L4S is not standardized and hence there is neither a known working implementation (tested internet wide) nor an accepted standard to work against. 

> 
> Issue #16 is pretty much a consequence of the above timeline,

	[SM] That seems wishful thinking to me. #16 is a direct consequence of L4S decision to violate precedence and redefine the meaning of CE, plain and simple. The only thing where fq_codel got involved is in being a pretty good all around qdisc for tcp-friendly traffic and hence seeing decent deployment in Linux (and as I might add in Macs as it seems). Yes, that attractiveness has increased L4S' predicament, but it is NOT the root cause, let us please not confuse this here.

> I see that

	I note that for the longest time rfc3168 compatibility was on the todo list of L4S. Only when team L4S was trying to get that requirement dropped* instead of actually tackling the issue the topic came to the fore on the tsvwg mailing list and meetings.

Best Regards
	Sebastian

---

From: "Black, David" <[email-masked]>
Date: Wed, 17 Jun 2020 17:25:59 +0000

Ingemar,
[posting as an individual]

> with fq-codel before the debate started in TSVWG?.

The question seems relevant in the other direction - shouldn't L4S have taken coexistence RFC 3168 ECN more seriously in that timeframe?

To my recollection, the overall L4S approach to this topic in that timeframe is captured by this text that is still in the l4s-id draft (Appendix B):

      1.  It is quite unusual to experience queuing at more than one
          bottleneck on the same path (the available capacities have to
          be identical).

Thanks, --David

> From: tsvwg <[email-masked]> On Behalf Of Ingemar Johansson S
> Sent: Wednesday, June 17, 2020 12:27 PM
> To: Rodney W. Grimes; Jonathan Morton; Sebastian Moeller
> Cc: Ingemar Johansson S; [email-masked]

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 17 Jun 2020 17:28:53 +0000

Ok. So I take it that you dismissed L4S as a crackpot idea and gave it a blind eye for a few years. That was your decision and now you stand with equipment that only speaks RFC3168, you could have corrected that issue a few years ago but chose not to do it just because you ignored ongoing work.
Please forgive me but I fail to see why your ignorance is a problem for IETF?
/Ingemar

Hämta Outlook för iOS<https://aka.ms/o0ukef>
________________________________
Från: Jonathan Morton <[email-masked]>
Skickat: Wednesday, June 17, 2020 6:54:26 PM
Till: Ingemar Johansson S <[email-masked]>
Kopia: Rodney W. Grimes <[email-masked]>; Sebastian Moeller <[email-masked]>; [email-masked] <[email-masked]>
Ämne: Re: [tsvwg] path forward on L4S issue #16

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 17 Jun 2020 20:19:23 +0200

Hi David,

> On Jun 17, 2020, at 19:25, Black, David <[email-masked]> wrote:
> 
> Ingemar,

	[SM] Which again is consisten with my hypothesis that L4S was really only ever intended, designed, and tested for a single use-case, short RTT, low hop-count downlink paths from content/CDNs to consumers. In that context the above assessment, while not strictly correct, is a decent fit for the common condition. 
	I have been arguing for some time now, that L4S should be tested for more realistic internet-wide conditions, BEFORE its roll-out is approved by the IETF to demonstrate its fitness for that novel use-case, it was never optimized or even properly tested for. And I am truly puzzled that the L4S camp and all the voices that supported ECT(1) as input just recently keep staying silent about that question.

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 17 Jun 2020 21:25:32 +0200

Hi Wes,

> On Jun 17, 2020, at 18:23, Wesley Eddy <[email-masked]> wrote:
> 
> I don't suspect that it will soothe you, but will just say that I volunteer as a co-chair and shepherd this particular work out of care for the "actual internet community" you mention.  The desire to experimentally roll-out low-latency services is aimed at serving that community.

	[SM] Please elaborate why you believe that for experimenting with low-latency services in the actual internet, an IETF RFC seems required? The bufferbloat/fq_codel/SQM crowd has been busy doing exactly that in the real live since before 2012's Codel ACM article. Codel and fq_codel both grew proper RFC AFTER demonstrating their safety, reliability and robustness in several implementations in real code in the real internet. Why is L4S such a special case that will be given a pass on demonstrating safety, reliability and robustness BEFORE being codified in an RFC? 
As I have argued multiple times in the past L4S offer precious little over the current state of the art at a considerable cost, so is far from a slam dunk solution the whole internet is just waiting for. There are a few interesting ideas in L4S all worth exploring properly, but nothing really requires timely standardization from a technical perspective (marketimg might be a different game altogether, but I believe not the IETF's main concern)

> It has not been recklessly "pushed through the process" as you are concerned with.  

	[SM] Not my words, but the framing of the L4S or SCE decision as ECT(1) as input/output question squandered a lot of good will, needlessly. (Why needlessly, because without first answering the can ECT(!) as input be safe enough question making a decision how/if to re-use the ECT(1) codepoint seemed premature).

> There are itemized issues to be resolved in order to have confidence that this can be safely deployed

	[SM] L4S issue is not the lack of enumerated lists that "improve" IETF meeting over IETF meeting, but a lack of real hard testing under realistic internet conditions. The items in the tracker try to give points to hang-up the discussion and results of these tests, but so far that amount of real-world data from the L4S proponents* has been scarce.

> (and if more emerge, they can be dispositioned and added if needed).  This particular thread is about agreeing on a path forward on the one labelled "issue #16" in the tracker.  Suggestions and discussion about that are good and welcome; repeatedly accusing others of bad intentions is not.

	[SM] In the spirit of progress, how about team L4S either accepts rfc3168 compatibility to be a hard requirement and we start to discuss what kind of properties the L4S solution needs to achieve (degree of flow imbalance, temporal fidelity of the detection, false negative rates, ...) or team L4S lays out a plan how to assess the position Ingemar (and Bob in the past) seem to take that rfc3168 AQMs are sufficiently rare in the real world to allow ignoring them. I would expect that we then discuss how to measure this** and decide on what thresholds we consider for significance before hand.

Best Regards
	Sebastian

*) This is a call to action not only for the core L4S team, but really to everybody who voiced support or opposition to L4S, people staying completely mum in the debate so far are excused ;)

**) And yes we also need to assess who needs to make these measurements.

---

From: Paul Vixie <[email-masked]>
Date: Wed, 17 Jun 2020 20:50:27 +0000

On Wednesday, 17 June 2020 18:19:23 UTC Sebastian Moeller wrote:
> > ...
> 
> 	[SM] ... my hypothesis that L4S was really only
> ever intended, designed, and tested for a single use-case, short RTT, low
> hop-count downlink paths from content/CDNs to consumers. ...

that's not really a hypothesis, but rather an observation. by the time a 
packet has traversed a metropolitan area, the scale of its transit time will 
be greater than the difference in scales between the dual queues of dualq.

to those who were confused when i referred to L4S as a datacenter protocol, 
this is why: getting into the microsecond vs. millisecond queue in an edge 
switch or router at the far-end campus or home or LAN won't change the lived 
experience of your application or user in any way since the packet is already 
a half dozen or more milliseconds from where it was born.

-- 
Paul

---

From: Wesley Eddy <[email-masked]>
Date: Wed, 17 Jun 2020 16:51:53 -0400

(I'm at least for now ignoring the parts of this not related to this 
thread on issue #16.)

> 	[SM] In the spirit of progress, how about team L4S either accepts rfc3168 compatibility to be a hard requirement and we start to discuss what kind of properties the L4S solution needs to achieve (…

This is already a requirement.  In the L4S I-D draft, there is a 
requirement:

A.1.4 Fall back to Reno-friendly congestion control on classic ECN
           bottlenecks

    Description: A scalable congestion control needs to react to ECN
    marking from a non-L4S, but ECN-capable, bottleneck in a way that
    will coexist with a TCP Reno congestion control [RFC5681].

Again, the point of this thread is to figure out what the working group 
needs to do to achieve this.  Looking at it as a co-chair, I'm seeing 
basic agreement that the classic bottleneck detection algorithm will 
likely never be perfect.  I don't think there's agreement (or even very 
specific suggestions) about how good it needs to be.  There might be 
agreement that in any case, operators should not expect that all flows 
using ECT(1) will succeed in being nice to classic flows.  Along the 
lines of some of Paul Vixie's comments in earlier threads, protection 
from unfriendly or lying flows is important, and this should be true 
even without regard to whether an operator deploys L4S in their network, 
or if traffic is using ECT(1).  David has suggested to focus on 
operational guidelines that would help to avoid ECT(1) bleaching, and I 
think many agree that such bleaching would be undesirable.

So, it seems at the moment like we should be working on this in parallel 
to any improvements on classic bottleneck detection, because it will be 
important in any case.

---

From: Wesley Eddy <[email-masked]>
Date: Wed, 17 Jun 2020 17:00:08 -0400

> a half dozen or more milliseconds from where it was born.

There are a number of URLLC scenarios that do not involve datacenters.

---

From: "Holland, Jake" <[email-masked]>
Date: Wed, 17 Jun 2020 21:12:12 +0000

Hi Ingemar,

One clarification:

> stores. The rest is unconfirmed.

If you're referring to the observations I sent, they are not linked to
OpenWRT.  I was looking at Akamai production traffic that saw ECE marks
while delivering content to end users.

Although many of the marking devices were likely CPE devices deployed
by individual end users, there's reason to believe that several ISPs
(what looks like a growing number) have some kind of managed deployment
that has a much greater prevalence within their ASN than can be easily
explained by individual users installing a custom CPE.

I don't have any information on whether the marking devices inside these
wider deployments within specific ASNs are predominantly CPE devices or
whether they're one of the many routers or access devices that support
CE-marking that they have now chosen to enable, after historically
running it disabled.  (This is not implausible, because the amount of
traffic that will respond to the marks has changed substantially in the
last few years.)

HTH.

Best regards,
Jake

---

From: Ingemar Johansson S <[email-masked]>
Date: Thu, 18 Jun 2020 05:15:13 +0000

Hi

Thanks Jake, this clarifies things in this respect. 
I believe/hope that larger content providers and ISPs can be easier to address with operator guidelines that Wesley (David?) suggested, and that can perhaps work as we see a reasonably good industry support behind L4S. And after all a decision to bleach ECT(1) need to be preceded by some kind of investigation about the options ?
Individuals that install custom CPEs are a more uncertain element of course. We discussed this a bit in this thread, there are possibilities as well as challenges here.

With that said, I believe that it is now fair that I do a multiplicative decrease and give this group a chance to breathe 😊

/Ingemar

> Sent: den 17 juni 2020 23:12
> To: Ingemar Johansson S
> <ingemar.s.johansson=[email-masked]>; Rodney W. Grimes
> <[email-masked]>; Jonathan Morton <[email-masked]>;
> Sebastian Moeller <[email-masked]>
> On 6/17/20, 9:27 AM, "Ingemar Johansson S"
> > enabled and these are related to the OpenWrt project, manifested in a
> > few upgradeable devices (and some not upgradeable) that can be bought
> > in retail stores. The rest is unconfirmed.
> 
> Although many of the marking devices were likely CPE devices deployed by
> individual end users, there's reason to believe that several ISPs (what looks
> like a growing number) have some kind of managed deployment that has a
> much greater prevalence within their ASN than can be easily explained by
> individual users installing a custom CPE.
> 
> whether they're one of the many routers or access devices that support CE-
> marking that they have now chosen to enable, after historically running it
> disabled.  (This is not implausible, because the amount of traffic that will
> respond to the marks has changed substantially in the last few years.)
> 
> HTH.
> 
> Best regards,
> Jake
>

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 18 Jun 2020 09:27:38 +0200

Hi Wes,

More below in-line.

> (I'm at least for now ignoring the parts of this not related to this 

       [SM] Certainly an option, it will not make the objections go away though, but you know that.

>>> (and if more emerge, they can be dispositioned and added if needed).
> This particular thread is about agreeing on a path forward on the one
> labelled "issue #16" in the tracker.  Suggestions and discussion about
> that are good and welcome; repeatedly accusing others of bad intentions
> is not.
>> 	[SM] In the spirit of progress, how about team L4S either accepts
> rfc3168 compatibility to be a hard requirement and we start to discuss
> what kind of properties the L4S solution needs to achieve (degree of
> flow imbalance, temporal fidelity of the detection, false negative
> rates, ...) or team L4S lays out a plan how to assess the position
> Ingemar (and Bob in the past) seem to take that rfc3168 AQMs are
> sufficiently rare in the real world to allow ignoring them. I would
> expect that we then discuss how to measure this** and decide on what
> thresholds we consider for significance before hand.
> 

       [SM] You seem to be ignoring Bob and Ingemar's push to simply have that requirement voided due to perceived insignificance of the existing rfc3168 deployment in spite of the fact that a) nobody has deployment numbers and b) even with number it would be hard to assess significance. E.g. there are far fewer inter-AS router's than there are home router's so even if all peering router's would employ rfc3168 AQM's this would result in a relative tiny number, but I assume everybody would agree that that would be a highly significant deployment, no?
Do I understand your position correctly that this requirement will not be relaxed/given up?

> specific suggestions) about how good it needs to be.  

       [SM] Hence my proposal for the WG to work out the requirements in terms what fraction of an RTT the detection needs to trigger by and what level of false negatives (misclassified true rfc3168 paths) is deemed acceptable.
	I will start out with a proposal that the L4S team should like:
temporal response period: 1ms
correct detection rate: 99.999

But realistically how about a more relaxed:

temporal response period:  <= 4 RTTs after congestion signals appeared

rationale: it takes <= 1 RTT to feed back the first CE to the sender so the sender has 3 RTTs to figure out whether the path is likely rfc3168 compliant or not. That limits the fallout of misclassifications significantly, if later the transport layer accumulated sufficient evidence to declare the detection a false positive, it should be allowed to re-evaluate its decision and try 1/p CE interpretation again, still with a 4 RTT fall-back window.
Ideally TCP Prague SHOULD start out with assuming all paths are rfc3168 and switch to 1/p style CE-interpretation only after sufficient evidence has been gathered. But that would amount to safety by design, an approach clearly not preferred by the L4S effort.

correct detection rate: 99.9 % (and no persistent wrong classification of individual paths)

rationale: these numbers will be arbitrary, but for all intents and purposes 99.9 should sufficiently take the sting out false negatives. BUT that only holds if we do not permit persistent mis-classifications as that can easily lead to starvation of non-1/p flows traversing the same rfc3168-compliant hop.

I am happy to discuss other explicit proposals.

> There might be 
> using ECT(1) will succeed in being nice to classic flows. 

	[SM] It is worth mentioning that typically we take adversarial traffic into account that maliciously tries to game the system for its advantage, but here we are talking about technically well-meaning flows that are simply hobbled by their underlaying technologies short comings. Which is ironic, given that none of this is deployed yet and hence the short-coming might simply be engineered away pro-actively instead.

> Along the 

	[SM] As undesirable as it is unavoidable giving the approach to testing that has/is currently used. If you only leave one remedy, please do not be sad when the real world uses that one remedy. As an operator, would you really consider it to be good advice to ignore ECT(1) packets in your RFC3168 AQM and still happily pass on such bobby-trapped packets to your paying customers, or would you pro-actively clean up that mess, since you need to touch all of these packets anyway? Bleaching ECT(1) (or bleaching both ECN bits if no ECN is used by an ISP) surely is the trustworthy safe by default method.

> to any improvements on classic bottleneck detection, because it will be

	[SM] Well, I am really interested ot read proposals how that would look like and how that would work out with operators that are not constantly reading internet drafts.

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 18 Jun 2020 09:38:56 +0200

Hi Paul,

> On Jun 17, 2020, at 22:50, Paul Vixie <[email-masked]> wrote:
> 
> that's not really a hypothesis, but rather an observation.

	[SM] Good point, I did not want to claim that I could divine the L4S intent and design goals, but I agree that for testing this hypothesis is based on observed facts.

> by the time a 

	[SM] +1; even with a delay target of 15ms the non-LL queue of the dual queue coupled AQM  will have chopped off the bad multiple dozens or hundreds of milliseconds queueing delay that is the real problem for latency bound services on the internet. Which is nicely demonstrated by how well already deployed combinations of HTB+fq_codel or cake work in the real world. Getting 1/p congestion signalling is still a valid goal IMHO, but let's keep the cost in perspective.

> a half dozen or more milliseconds from where it was born.

	[SM] Playing advocatus diaboli here, a reduction in RTT variance has the potential to make the work of RTT-equalization methods more effective; methods that are used e.g. in VoIP de-jitter buffers and in on-line games to offer a level playing field to all agents. But the idea that a single hop is typically responsible for all delay variation seems like too strong a claim to me, so I agree that there will be diminishing returns the longer the path gets. I like your description "datacenter protocol", as that better manages expectations that "low-latency-for-all"*.

Best Regards
	Sebastian

*) Which ignores that not all latency is queueing delay, hence reducing queueing delay will not magically make all paths low-latency.

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 18 Jun 2020 11:07:39 +0300

> On 18 Jun, 2020, at 10:38 am, Sebastian Moeller <[email-masked]> wrote:
> 
> …the idea that a single hop is typically responsible for all delay variation seems like too strong a claim to me…

It's quite easy to construct a counterexample, too: any network path including two wifi hops.  Wifi is inherently a significant source of delay variation due to its shared-medium, half-duplex, CSMA/CA with ARQ architecture.  So even if the rest of that network path introduces no delay variation at all, there are still two hops independently introducing delay variations.

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 18 Jun 2020 10:15:28 +0200

Hi Jonathan,

good point below, which basically adds one more really important test condition to the already long list of missing tests for L4S:

A) Saturated multi-hop paths (individually saturated for both down- and upload direction)
B) Bi-directionally saturated path
C) asymmetric paths (including A) and B) conditions)
D) tests over real internet paths, coast to coast in the US, as well as trans-european and trans-asia paths
E) paths with wifi links
F)...

But in all likelihood we will just see more tests for those conditions L4S is known to work reasonably well for, short RTT and/or low hop count, tests of the download direction with at best the download direction experiencing congestion...

Best Regards
	Sebastian

> On Jun 18, 2020, at 10:07, Jonathan Morton <[email-masked]> wrote:
>