Subject: [tsvwg] path forward on L4S issue #16

From: Wesley Eddy <[email-masked]>
Date: Thu, 4 Jun 2020 16:53:50 -0400

I think we should discuss the path forward on L4S issue #16 and what 
people are working on, planning to do, or expecting to see in this regard.

This is the issue on interaction with RFC 3168 ECN AQMs in the network.

I think this is one of the more important ones in many recent 
discussions, so would like to make sure we're agreeing on what it will 
take to complete or what success will look like.

The classic bottleneck detection work is a key part of this.  In that 
vein, I'd like to subsume issue #29 (that it looks like Pete Heist 
opened) under this one and close #29.  Issue #29 shows some cases where 
an L4S queue is misidentified as a classic one.  I think fixing this is 
part of making the classic queue detection and end-host performance 
robust in as many cases as possible when trying to take advantage of L4S 
in the network, but doesn't lead to a safety issue.

Are there any plans or thoughts on how to proceed on issue #16 that 
people can share?

---

From: "Black, David" <[email-masked]>
Date: Fri, 5 Jun 2020 00:36:17 +0000

Internal Use - Confidential

> I think fixing this is part of making the
> classic queue detection and end-host performance robust in as many cases as
> possible when trying to take advantage of L4S in the network, but doesn't lead to a
> safety issue.
>
> Are there any plans or thoughts on how to proceed on issue #16 that people can share?

[Writing as an individual, not WG chair]
I have an alternate perspective on this issue, and suggest an alternate approach to addressing it.

Under the heading of TCP Prague, an enormous amount of work has gone into endpoint detection of RFC 3168 ECN bottlenecks and endpoint-based problem avoidance.   Of the test results that have been posted, Pete Heist's scenario 8 results (https://mailarchive.ietf.org/arch/msg/tsvwg/-gx8R1jcwiCsX1_5cdnvl3kKlOs/) ought to give everyone pause, as they show TCP Prague flows getting into trouble competing among themselves.

This TCP Prague work on bottleneck detection and problem avoidance has been going on for a year, and while it's produced some very interesting results and I'm sure that more progress will be made, I'm skeptical that relying on it to close issue #16 will yield much more than rounds of finding new problems in the next five versions of TCP Prague.   Beyond that, I'd expect to see quite a bit of L4S low latency traffic from protocols whose endpoint implementations have not been scrutinized to anything approaching the degree of attention that has been recently devoted to TCP Prague, and hence are at more risk of getting into trouble.

As an alternative, I suggest starting from Neil Cardwell's network and operator perspective (from https://mailarchive.ietf.org/arch/msg/tsvwg/kyDZ7vYLeOk1V15nZfNXGmaUNxk/):

	- L4S flows potentially causing unfairness in RFC3168 ECN bottlenecks has
	been mentioned as a potential concern. However, a robust RFC3168 ECN
	bottleneck should already have a mechanism to avoid unfairness caused by
	flows that are marked as ECT(0|1) and yet not performing RFC3168 responses.
	In particular, many of the large sources of known deployments of  RFC3168
	--  Linux fq_codel and cake -- are already deployed with fair queueing. In
	such bottlenecks L4S traffic should not cause harm to other non-L4S flows.
	Furthermore, if there really are ISPs with deployments of RFC3168
	bottlenecks that have neither FQ nor any other protection from
	non-RFC3168-ECT(1) flows, then they can bleach incoming ECT(1) code points
	to Not-ECT and treat L4S as Not-ECT (ISPs typically already transform the
	DSCP byte at their ingress anyway). So I do not see harm to RFC3168 ECN
	bottlenecks as a prohibitive concern.

	- More generally, if there is any problem discovered with the L4S
	experiment, either the algorithm or particular implementations, bottlenecks
	can easily identify L4S traffic and bleach it into Not-ECT, and treat it
	like Reno/CUBIC traffic.

TCP Prague is not only not mentioned in the text quoted above, it appears nowhere in the entire email from which that was extracted.  Nonetheless, the email lays out an approach to L4S safety, although as initially proposed, it invites operator bleaching of ECT(1), which would be unfortunate, as ECT(1) not being subject to bleaching is among the key reasons for using it.  Refining that approach might produce something more robust that doesn't invite ECT(1) bleaching, and that recognizes that FQ is not a 100% solution to flow interference (e.g., courtesy of the birthday paradox for the number of hash buckets in an FQ implementation).  As noted before, please don't cast aspersions in Neal's direction for introducing the notion of operator bleaching of ECT(1).

So, I would suggest focusing attention on networks and operators as an alternative approach to focusing on endpoints.

Thanks, --David

> -----Original Message-----
> From: tsvwg <[email-masked]> On Behalf Of Wesley Eddy
> Sent: Thursday, June 4, 2020 4:54 PM
> To: [email-masked]
> Subject: [tsvwg] path forward on L4S issue #16
> 
> 
> [EXTERNAL EMAIL]
> 
> I think we should discuss the path forward on L4S issue #16 and what people are
> working on, planning to do, or expecting to see in this regard.
> 
> I think this is one of the more important ones in many recent discussions, so would
> like to make sure we're agreeing on what it will take to complete or what success
> will look like.
> 
> The classic bottleneck detection work is a key part of this.  In that vein, I'd like to
> subsume issue #29 (that it looks like Pete Heist
> opened) under this one and close #29.  Issue #29 shows some cases where an L4S
> queue is misidentified as a classic one.  I think fixing this is part of making the
> classic queue detection and end-host performance robust in as many cases as
> possible when trying to take advantage of L4S in the network, but doesn't lead to a
> safety issue.
> 
> Are there any plans or thoughts on how to proceed on issue #16 that people can
> share?

---

From: "Black, David" <[email-masked]>
Date: Mon, 8 Jun 2020 13:32:23 +0000

Without stupid confidential tag this time ...

Thanks, --David

> I think fixing this is part of making the classic queue detection and 
> end-host performance robust in as many cases as possible when trying 
> to take advantage of L4S in the network, but doesn't lead to a safety 
> issue.
>

[Writing as an individual, not WG chair] I have an alternate perspective on this issue, and suggest an alternate approach to addressing it.

Thanks, --David

> opened) under this one and close #29.  Issue #29 shows some cases 
> where an L4S queue is misidentified as a classic one.  I think fixing 
> this is part of making the classic queue detection and end-host 
> performance robust in as many cases as possible when trying to take 
> advantage of L4S in the network, but doesn't lead to a safety issue.
>

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 8 Jun 2020 17:30:56 +0300

> On 8 Jun, 2020, at 4:32 pm, Black, David <[email-masked]> wrote:
> 

I think some concrete evidence of this possibility was provided by the experimental adaptation of SCReAM to use L4S signalling, which was discussed here fairly recently.  The implementation took a very bare-bones approach, copying only the use of the ECT(1) marker and the DCTCP response algorithm from whatever original source they used.  The result was summarily declared a success by its author without any analysis of how it might interact with conventional traffic; I assume he relied on the L4S team proper to have done their homework in that respect.

So we can indeed expect endpoints to behave like TCP Prague or like SCReAM, within the L4S ecosystem as presently defined.  Network operators will be expected to deal with the fallout.  I think we should also expect most network operators to be blissfully unaware of anything we're doing here; most, indeed, will be fully occupied in simply maintaining reliable connectivity to their own little patch of the Pacific, or Africa, or Asia Minor.

That is the context in which L4S must find a solution to proceed.  The solution must be demonstrated, tested, and documented in the appropriate drafts before it can be accepted.  I think it is a Hard Problem™, requiring genuine further innovation to solve - or, alternatively, a change to the signalling method in order to remove the ambiguity at the root of the problem.

 - Jonathan Morton

---

From: Steven Blake <[email-masked]>
Date: Mon, 08 Jun 2020 16:38:47 -0400

On Mon, 2020-06-08 at 13:32 +0000, Black, David wrote:

> As an alternative, I suggest starting from Neil Cardwell's network
> and operator perspective (from 
> https://mailarchive.ietf.org/arch/msg/tsvwg/kyDZ7vYLeOk1V15nZfNXGmaUNxk/):
> 

My cursory examination turned up zero evidence that any major vendor
has implemented any sort of AQM/ECN "penalty box" fairness enforcement
mechanisms in carrier gear, at least above the access edge. If anyone
has some evidence, please speak up.

> 	can easily identify L4S traffic and bleach it into Not-ECT, and treat it

If there is even a non-trivial probability that this could be the
outcome, how can anyone argue in good faith that L4S traffic should not
be classified by DSCP?

Regards,

// Steve

---

From: "[email-masked]" <[email-masked]>
Date: Mon, 8 Jun 2020 23:47:37 +0000 (UTC)

If we're now discussing ways to make L4S more safe, it seems to me that it might be useful to attempt to enumerate all the
possibilities. I think there are a few that have not been previously mentioned; most have difficulties, but perhaps some of the experts here can a way forward:

1) As frequently mentioned, if L4S marks were distinguishable from RFC3168 marks, L4S endpoints would be able to react more appropriately. 
Since the group is now persuing ECT(1) as network input, this approach is currently without an codepoint, although there has been some speculation that DSCP could be used as an output. This is sufficient for safety, but perhaps it is not necessary. However the 'safe by construction' 
attributes are obviously attractive. 

2) RFC3168 detection, as proposed by Bob's working paper. The immediate difficulty seems to be in defining a set of test cases sufficient to convince the group.

3) Change DUALQ such that marks from a DUALQ can be more easily distinguished from RFC3168 marks.
I raise this as a possibility in order to see if any aspect of "safe by construction" can be incorporated into the RFC3168 detection scheme.
I can see two possibilities:
 a) DUALQ to never mark certain packets, comprising a known,  substantial % of packets.
 b) DUALQ redefined to always mark consecutive packets from a flow

Both of these have their difficulties. 

In case a), if a CE were observed on one of the forbidden packets, then it could be assumed to be RFC3168 and fallback should occur. If the % of forbidden packets were known, and these are independent RFC31680 CE marking, then the statistics of when fallback should occur will only be determined by the rate of RFC3168 CE marking. I think this would get us a lot closer to 'safe by construction', as we can decide if the tail distribution of fallback time looks okay. It would be necessary to see how much this degrades the L4S congestion signal.

In case b), if a lone CE were observed then it would be clear an RFC3168 AQM was present, and fallback would occur. The statistics of this are more complicated because they depend on the rate of DUALQ marking as well as RFC3168 marking, since the proportion of RFC3168 marks masked by L4S marks is less obvious. It would be more obvious if DUALQ marking were capped at 50 or 75% instead of 100%. I don't know 
how that would affect the L4S goals.

Both of these have the advantage that the mechanism for detecting RFC3168 AQMs is simple and predictable. Unfortunately there are other obstacles. 

In the case of a), it would seem to require some codepoint that could be used as another input to the IP layer, to inform DUALQ which packets are forbidden. If those were easy to find, we wouldn't be having this whole discussion. So we are left looking for one that, for some reason, couldn't be used for the L4S  ID, but could be used here. There are a few options, with various difficulties:

i) In previous work, it's been proposed to use an IPv6 destopt as a network input (rfc7837). I'm not sure if that was ever deployed, and obviously it only works for ipv6.
ii) The LSB of TTL. The sender could modulate the LSB of TTL. Although the actual value would not be be visible at the AQM, it's probably had some constant subtracted from it. So if the AQM refrains from marking when TTL.0==1, the receiver could work out fairly swiftly whether the constant offset was odd or even. However, this would not work inside tunnels as the inner TTL is not propagated to the outer on encapsulation
iii) The LSB of payload length (or, the 3rd LSB of payload length). This has the distinct disadvantage that modulating payload length might not be possible just in the TCP implementation, but also involve the application layer. The 3rd LSB could be modulated by adding TCP options, but those might be intefered with by middleboxes. 

The difficulty of b) is more obvious - it requires DUALQ to be flow aware, even more so that FQ, since binning would not be sufficient. It wouldn't need to keep the flow state very long (just long enough to mark the next packet) but I would guess this is still too much.

The open question here is whether there is any such modification of DUALQ that doesn't have the above difficulties.
There might be ways to modify the statistics of DUALQ marks to make them more distinguishable  from RFC3168 marks, without requiring either another input or FQ.  This brings me to:

c) We could change the marking function to in introduce a discontinuity, so that the L queue never adopts a marking rate between 0% and (say) 20%. (It would still be linear between 20% and 100%).  There would still need to be a region when 0% marks occur. This is a bit odd, but it still makes good use of the available bandwidth of ECN, and conveys the same signal, so I guess it should still be scalable. With such a modification, if a lone CE mark was observed on a flow, it would be highly likely that it came from an RFC3168 AQM. However it's much less obvious (to me at least) how to write down P(RFC3168 is present | marks seen), and even less obvious what the statistics of fallback time would be  (since it would depend on how much time the DUALQ spent in the 0% marking region), although I suspect it would be a lot better than in 2). The DUALQ linkage, along with prague, would need adaption. It may prove complicated around the discontinuity region. Nevertheless this seems most in line with the spirit of the L4S design. The other issue is whether it is possible to build up a queue in both an DUALQ AQM and an RFC3168 AQM at the same time on the same path. If this was a significant possibility, the DUALQ marks at the >20% regime would would obscure the RFC3168 marks. But this state of affairs would need to persist otherwise fallback would occur when only the RFC3168 AQM was the bottlebeck.

Obviously, modifying DUALQ would be an annoyance to vendors who have already implemented it in silicon, but perhaps less of one than other options...

4) Modifying networks containing RFC3168 AQMs

Obviously this category has the disadvantage that operators not deploying L4S need to take action, but David has indicated that it may be acceptable.

Here we have
 a) Bleaching ECT(1)
 b) Modifying RFC3168 AQMs to be aware of ECT(1)

a) Has the problem that it will cause deployment problems for this and any other use of ECT(1), but has the advantage that it requires an operator to act only on the perimeter of their network
b) Requires an operator to inspect and possibly modify  every device  on their network.

Probably here it would be necessary to have a suite of options depending on what RFC3168 AQMs the operator thinks they might have, eg
c) if you know you have no RFC3168 AQMs, no action required

One option which occurs to me is to 
d) provide operators a way to signal that their network doesn't yet support L4S. What's not obvious to me is:
 i) since that requires tcp prague and all other L4S transport implementations to adhere to it, will operators trust it? Possibly they would up to the point that some transport implementation implements it incorrectly.
 ii) what's a suitable way of probing for support?

Alex

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 9 Jun 2020 04:23:31 +0300

> On 9 Jun, 2020, at 2:47 am, [email-masked] wrote:
> 

Obviously, SCE does things this way, but without an input classifier.  This makes sense, since it started with observation of this very problem in DCTCP and L4S, and sought to avoid it "by construction" as you put it.

Jake Holland made an alternative proposal which does use ECT(1) as an input, and makes ECT(0) the output from a HFCC AQM.  This retains the RFC-3168 meaning of a CE mark, and thereby unambiguously distinguishes the outputs of L4S and RFC-3168 AQMs.  I believe this has the potential to work well, and might be the single most promising solution to the problems that L4S has, under the constraint that ECT(1) is used as an input to the network.

The only substantive argument that I have noticed against this is that it might interact with tunnels poorly, due mostly to underspecification of how mismatches between ECT(0) and ECT(1) in different ECN fields belonging to the same packet should be resolved upon reassembly and/or decapsulation.

I think we need to understand the tunnel situation much more clearly, with reliable and up-to-date information about their prevalence and detailed behaviour, in order to properly judge whether the effects on Jake's proposal (or indeed on SCE) are helpful, benign, undesirable, or catastrophic.  I'm aware that both RFC-3168 and RFC-6040 have things to say about this area, and that they do not necessarily agree with each other or with actual field practice.  In principle it is possible to update these specs, but initially it is already-deployed systems that will be important here.

Intuitively, I would classify the effects of CE being an ambiguous signal as "catastrophic", while the effects of a tunnel muddling up an ECT(0/1) signal are likely to be "benign".  By the latter I mean that a HFCC signal applied to the outer header of a tunnel may be amplified or erased upon decapsulation, but that a CE mark or an unmarked packet will be decapsulated correctly.

 - Jonathan Morton

---

From: "Holland, Jake" <[email-masked]>
Date: Sat, 20 Jun 2020 01:33:12 +0000

Hi Wes and tsvwg,

On 6/4/20, 1:54 PM, "Wesley Eddy" <[email-masked]> wrote:
> The classic bottleneck detection work is a key part of this.
...(snip)...

I think a few other ideas have also been floated, so I'd ask to include
those proposals as invited parts of the discussion as well.

This thread seemed like it had a lot of branches and was kind of hard
to follow in places, so I thought I'd compile a list of the proposed
paths forward that I've seen.  I'm not sure I got everything (please
respond if anybody knows another suggestion that was left out).

So in hopes it's useful, here's my list (in no particular order):

1. a robust classic bottleneck detection mechanism

2. Changing L4S to use a 2-signal approach, using ECT(1)->ECT(0) for
   the 1/p signal and ECT(1|0)->CE as a 1/sqrt(p) signal.

3. a flag day to deprecate ECT(1)->CE marking by classic queues (instead
   treating ECT(1) as NECT if no non-3168 meaning is implemented).

4. operational considerations to recommend changing ECT(1) to NECT at
   ingress to networks that have marking classic queues deployed

5. operational considerations to recommend policing strategies that can
   solve the general case of non-compliant traffic that does not respond
   with the expected backoff to AQM congestion signaling.

I'll make another new suggestion now:

6. An experiment-linked public whitelist of participant-registered IP
   ranges that have a L4S compatible dualq in their reachability path at
   the likely bottleneck, which would be checked by endpoints before
   negotiating L4S.

To add some of my own color commentary on these:

1. a robust classic bottleneck detection mechanism
a. AFAICT this is still the method preferred by the L4S authors.
b. There's been some discussion about how technically feasible this
   goal is, and also what level of "robust" is necessary.
c. I'll respectfully suggest that we as a WG should ideally have at least
   one solid backup plan, in case this approach proves hard to reach
   consensus.

2. Changing L4S to use a 2-signal approach, using ECT(1)->ECT(0) for
   the 1/p signal and ECT(1|0)->CE as a 1/sqrt(p) signal.
a. a few people seemed interested in considering this, but several
   objections were raised, chiefly (IIRC):
   i.   late out of order packets from chained loaded dualqs, and the
        corresponding spurious retransmissions (with a side note that
        this problem worsens as dualq deployment increases)
   ii.  loss of the 1/p signal with RFC 6040 tunnel decapsulation, and
        a corresponding limited scope initially, with a long slow path
        (including standards actions) to get to ubiquitous deployment
   iii. discards the long-term L4S goal of reclaiming ECT(0) for other
        purposes
   iv.  doesn't match the desired timeline for experimental deployment
        by those who have engaged with the L4S work.  (I'm not sure any
        of the proposed paths forward will satisfy this objection, but
        IIRC this one was specifically raised in response to the 2-signal
        proposal.)

3. a flag day to deprecate ECT(1)->CE marking by classic queues (instead
   treating ECT(1) as NECT if no non-3168 meaning is implemented).
a. Interestingly, the scope of this approach seems to track on the same
   questions as the "how important is a robust classic queue detection"
   question in #1.b, because they both depend on the current and in-flight
   deployment footprint for CE-marking classic queues.  If robustness is
   not very important, then it's because classic queue deployment is low,
   so a flag day would be a low-touch event.  And if a flag day would be
   a major lift with a lot of work for operators, then good robustness
   would likewise be very important if not doing a flag day.
b. However, this presumably requires an update to RFC 3168, and I'm not
   sure whether there's a well-established process for organizing an
   event like this.  Obviously the outreach effort is much higher than
   if e.g. option #1 or 2 turned out to be feasible to get working well
   enough to satisfy rough consensus.

4. operational considerations to recommend changing ECT(1) to NECT at
   ingress to networks that have marking classic queues deployed.
a. Note that these considerations are for networks NOT participating in the
   L4S experiment, so they can't easily be folded into L4S-specific
   operational considerations.
b. This seems to me most appropriate as an add-on of a fallback position
   during the #3 (flag day), where classic queues can't be reconfigured to
   treat ECT(1) as NECT.  But I left it as a separate point because it was
   proposed independently, and maybe there's an argument that only networks
   that experience a problem would need to do this and could do it as a
   post-hoc fix when problems are encountered, rather than pre-arranging
   it with a flag day and the associated proactive outreach it would need.
   (to be clear: I am currently against that position, but I acknowledge I
   might be in the rough when consensus is checked)

5. operational considerations to recommend policing strategies that can
   solve the general case of non-compliant traffic that does not respond
   with the expected backoff to AQM congestion signaling.
a. Arguably this should be done regardless of L4S, because it seems like
   an underdeveloped piece of the puzzle for the general problem of active
   queue management in network devices.  However, solving this well and
   getting solutions widely deployed or at least available (or somehow
   deployed in conjunction with L4S endpoint enablement) could also
   potentially address issue 16.
b. There are many possible strategies here, so outlining some of the
   known ones maybe seems worthwhile.  A few examples that spring to
   mind:
   i.   FQ
   ii.  PPV (http://ppv.elte.hu/), as we saw in ICCRG 104 from Szilveszter
        Nadas, seems to have a lot of promise here
   iii. Likewise the work trying to solve a similar problem written up in
        "Fair Resource Sharing for Stateless-Core Packet-Switched Networks
        With Prioritization" by Michael Menth and Nikolas Zeitler
        (https://ieeexplore.ieee.org/document/8419697)
   iv.  There's also some good insights along these lines in the "Rationale"
        section of the docsis queue protection scheme doc:
https://tools.ietf.org/html/draft-briscoe-docsis-q-protection-00#section-5
        It says the same approach could be used in scenarios beyond dualq,
        and I think there's some applicability to codel or pie, with or
        without ECN.
   v.   Perhaps some generic guidelines that captures what many of these
        have in common--in general a policing response could be based on
        sampled monitoring (not necessarily integrated closely with the
        queues) that maintains stats on the top current and recent senders,
        and blacklists or downgrades their traffic for some time in response
        to a large enough standing queue, or overflow of an AQM.  (On the
        grounds that someone here is non-compliant since it exceeded
        expected operational bounds, and thus at least the highest volume
        recent senders have presumably failed to back off appropriately.)
   A BCP that lists these (and maybe other options) and captures a more
   generalized version of the advice from docsis-q-protection seems likely
   helpful here.
c. Also worth noting: the need for some kind of isolation for non-compliant
   senders is not inherently an ECN-related (nor L4S-related) problem--there's
   no forced reason that a sender necessarily has to respond to loss either...

6. An experiment-linked public whitelist of participant-registered IP
   ranges that have a L4S compatible dualq in their reachability path at
   the likely bottleneck, which would be checked by endpoints before
   negotiating L4S.
a. to flesh the idea out a bit, I'm imagining this as a web API with a
   known URL and a database attached, which is documented and maintained
   as part of the L4S experiment, where experiment participants who have
   deployed and enabled dualq-capable devices register the applicable
   IP ranges, and L4S-capable endpoints query the web API before opening
   connections, so that they avoid negotiating L4S support except when
   either inside a network with a registered dualq, or when connecting to
   a remote endpoint that's inside such a network, caching the answers
   for ~10-30 minutes (or according to http headers in the web api or
   something)
b. This would let innocent bystander 3168 traffic operate unmolested at
   access bottlenecks while gaining live operational experience with L4S.
c. If the rapid ubiquitous rollout goes as planned according to the L4S
   project intent, once the dualq devices are far more prevalent than classic
   ECN queues and widely available on all the bandwidth-shaping access
   technologies and the non-experiment participants are predominantly not
   doing any classic marking, the whitelist could be gradually retired, or
   turned into a blacklist of L4S on IPs that are known to have problems with
   legacy systems that haven't been upgraded (which can be discovered by
   gradually enabling L4S on non-participant paths, perhaps with A/B tests,
   and following up with those networks to ask afterward whether it caused
   problems)
d. This approach is of course operationally awkward and not a good long-term
   solution, and also comes with potential privacy concerns as the deployment
   grows, but would allow for forward progress on L4S without fixing the
   underlying incompatibility problem with the CE ambiguity, so that time and
   an ongoing outreach effort could have a chance to resolve the classic ECN
   deployment footprint questions.  This also allows for some amount of
   targeted experimentation with the classic queue detection work.

(Of these, it's maybe interesting to note that only 1, 2, and 6 do not
seem to require a standards or BCP action, which IIRC was originally meant
to be out of scope for the L4S work, outside of 8311.)

As far as the original "working on, planning to do, or expecting to see"
question:

I guess I'm expecting to see at some point the results from what the L4S
team is doing on the detection mechanisms.  But I remain not very hopeful
they will address all the concerns that have been raised, so I'm hoping
it'll be coupled with a credible outreach effort that seems likely to reach
all the networks that have deployed or are in-process of deploying shared
classic queues, at the very least.

I'm not sure about "expecting", but I'd also be very much in favor of seeing
some sort of approval-style poll conducted (maybe with a preference
weighting or rank ordering or something) of what the WG members think of the
technical viability of the different proposed approaches, so that people can
have a better idea of what others think sound like promising directions.

Outside that, I'd personally love to see further discussion on these or
other ideas, but maybe forking the thread into different threads for the
different proposals, to better avoid the confusion I've been feeling trying
to find the prior links to the preceding messages on this monster (I eventually
gave up).

I regret that I don't currently have much to offer on the "working on" or
"planning to do" front, being rather busy right now with a few other
challenging problems.

But I am supportive of efforts to improve internet latency.  Especially
those aimed at increasing the deployment (and enablement) of the currently
available 3168 AQMs and the increased default use of 3168 ECN by more
endpoint stacks, since that would have a useful impact on application level
latency in TCP connections right away.

When I see good opportunities and I'm able, I'll aim to make minor
contributions to latency-related efforts or do things like minor testing
support when possible.  So anyone engaging in that kind of thing, please do
keep the wg (or at least me) posted on progress and any opportunities to
provide useful low-effort support.  I can spend a day on this kind of thing
every once in a while (tho sadly, not necessarily every time it might be
useful), but I can't spend weeks at a time.  That's probably true for the
near forseeable future.

Best regards,
Jake

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Fri, 19 Jun 2020 20:14:04 -0700 (PDT)

Jake,
	Thank you for spending the time to collect this
detailed summary.

	I believe you left out: (adding one to your last one and listing)

 7.  Use a DSCP to seperate the experiment, leaving ECT(1) and CE as
     currently specified in the L4S draft.

 8.  Use a DSCP to classify the traffic as L4S and leave ECT(1) unused,
     altering CE semantics.

 9.  Use a DSCP to classify the traffic, and use ECT(1) as a 1/p signal,
     leaving CE semantics in place.

10.  Dropping L4S as over promising and short delivering with complexity
     that almost certainly sets it up for a failed deployment.

Note that in all 4 of these solutions bleaching is unlikely to be
used if there are problems, and the experiment is rather trivial to
terminate if there are problems.  These also keep ECT(1) avaliable
for a future non-experiment version of L4S should the experiment work,
or something else should it fail.  7 to 9 can even be started today
without an IETF consenses and some real operational data created.

On the side, IMHO, the work going into L4S would be better spent addressing:
a)  DSCP global traversal
b)  ack thinning being underspecified such that it creates protocol
    problems.  Specifically the fact it tosses out changes in reserved
    bits by thinning packets with different bit values.  This was
    identified years ago and left as a problem, it needs cleaned up.
c)  Revision to RFC6040 and other tunnel related drafts to clear
    the issues there.  Again, identified years ago and left to clean
    up.

Regards,
Rod

-- 
Rod Grimes                                                 [email-masked]

---

From: <[email-masked]>
Date: Mon, 22 Jun 2020 06:11:11 +0000

Hi Jake,

thanks for your ongoing, constructive and to me big effort on ECN deployment.

Professionally, I'm not involved in access and application related queue- and congestion management. So it's hard for me to contribute. As an individual, I like your address proposal though. That sounds as if one could also use "alto" mechanisms (you mention an API and I only have a very rough idea, what alto does, so I may be wrong).

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: tsvwg <[email-masked]> Im Auftrag von Holland, Jake
Gesendet: Samstag, 20. Juni 2020 03:33
An: [email-masked]; [email-masked]
Betreff: Re: [tsvwg] path forward on L4S issue #16

Hi Wes and tsvwg,

...(snip)...

I think a few other ideas have also been floated, so I'd ask to include those proposals as invited parts of the discussion as well.

This thread seemed like it had a lot of branches and was kind of hard to follow in places, so I thought I'd compile a list of the proposed paths forward that I've seen.  I'm not sure I got everything (please respond if anybody knows another suggestion that was left out).

1. a robust classic bottleneck detection mechanism a. AFAICT this is still the method preferred by the L4S authors.

(Of these, it's maybe interesting to note that only 1, 2, and 6 do not seem to require a standards or BCP action, which IIRC was originally meant to be out of scope for the L4S work, outside of 8311.)

I guess I'm expecting to see at some point the results from what the L4S team is doing on the detection mechanisms.  But I remain not very hopeful they will address all the concerns that have been raised, so I'm hoping it'll be coupled with a credible outreach effort that seems likely to reach all the networks that have deployed or are in-process of deploying shared classic queues, at the very least.

I'm not sure about "expecting", but I'd also be very much in favor of seeing some sort of approval-style poll conducted (maybe with a preference weighting or rank ordering or something) of what the WG members think of the technical viability of the different proposed approaches, so that people can have a better idea of what others think sound like promising directions.

Outside that, I'd personally love to see further discussion on these or other ideas, but maybe forking the thread into different threads for the different proposals, to better avoid the confusion I've been feeling trying to find the prior links to the preceding messages on this monster (I eventually gave up).

I regret that I don't currently have much to offer on the "working on" or "planning to do" front, being rather busy right now with a few other challenging problems.

But I am supportive of efforts to improve internet latency.  Especially those aimed at increasing the deployment (and enablement) of the currently available 3168 AQMs and the increased default use of 3168 ECN by more endpoint stacks, since that would have a useful impact on application level latency in TCP connections right away.

When I see good opportunities and I'm able, I'll aim to make minor contributions to latency-related efforts or do things like minor testing support when possible.  So anyone engaging in that kind of thing, please do keep the wg (or at least me) posted on progress and any opportunities to provide useful low-effort support.  I can spend a day on this kind of thing every once in a while (tho sadly, not necessarily every time it might be useful), but I can't spend weeks at a time.  That's probably true for the near forseeable future.

Best regards,
Jake

---

From: "Scharf, Michael" <[email-masked]>
Date: Mon, 22 Jun 2020 07:22:05 +0000

Regarding ALTO, I recommend to have a look at RFC 7285 "Application-Layer Traffic Optimization (ALTO) Protocol" and RFC 7971 "Application-Layer Traffic Optimization (ALTO) Deployment Considerations".

A whitelist could possibly be realized by the ALTO Map Service with a new cost type.

As alternative to a public list, ALTO also defines on-demand query services (Endpoint Cost Service or Endpoint Property Service). That solution does not require that the network operator discloses properties of the network topology. There are privacy tradeoffs between the ALTO Map Service and the ALTO Endpoint Cost Service.

ALTO was developed before YANG took off in the IETF. One could possibly realize a network service similar to ALTO using RESTCONF/YANG. But then one would probably have to solve quite a bit of the problems that are well understood and solved for ALTO.

Michael
(as co-author of RFC 7971)

> From: tsvwg <[email-masked]> On Behalf Of
> [email-masked]
> Sent: Monday, June 22, 2020 8:11 AM
> To: jholland=[email-masked]
> Cc: [email-masked]
> Subject: Re: [tsvwg] path forward on L4S issue #16
> 
> Hi Jake,
> 
> Professionally, I'm not involved in access and application related queue- and
> congestion management. So it's hard for me to contribute. As an individual, I
> like your address proposal though. That sounds as if one could also use "alto"
> mechanisms (you mention an API and I only have a very rough idea, what
> alto does, so I may be wrong).
> 
> Regards,
> 
> Ruediger
> 
> 
> I think a few other ideas have also been floated, so I'd ask to include those
> proposals as invited parts of the discussion as well.
> 
> This thread seemed like it had a lot of branches and was kind of hard to
> follow in places, so I thought I'd compile a list of the proposed paths forward
> that I've seen.  I'm not sure I got everything (please respond if anybody
> knows another suggestion that was left out).
> 
> (Of these, it's maybe interesting to note that only 1, 2, and 6 do not seem to
> require a standards or BCP action, which IIRC was originally meant to be out
> of scope for the L4S work, outside of 8311.)
> 
> 
> they will address all the concerns that have been raised, so I'm hoping it'll be
> coupled with a credible outreach effort that seems likely to reach all the
> networks that have deployed or are in-process of deploying shared classic
> queues, at the very least.
> 
> weighting or rank ordering or something) of what the WG members think of
> the technical viability of the different proposed approaches, so that people
> can have a better idea of what others think sound like promising directions.
> 
> Outside that, I'd personally love to see further discussion on these or other
> ideas, but maybe forking the thread into different threads for the different
> proposals, to better avoid the confusion I've been feeling trying to find the
> prior links to the preceding messages on this monster (I eventually gave up).
> 
> 
> But I am supportive of efforts to improve internet latency.  Especially those
> aimed at increasing the deployment (and enablement) of the currently
> useful), but I can't spend weeks at a time.  That's probably true for the near
> forseeable future.
> 
> Best regards,
> Jake
>

---

From: Wesley Eddy <[email-masked]>
Date: Mon, 22 Jun 2020 11:39:19 -0400

>     one solid backup plan, in case this approach proves hard to reach

I think your comment (c) agrees with what several others have said too, 
so seems like a good idea.

>          IIRC this one was specifically raised in response to the 2-signal

I agree with your summary.

>     enough to satisfy rough consensus.

I think this would also be a pretty long-term plan, and it's not clear 
to me (at least) if it's very useful.  It assumes L4S will be very 
successful, and if that's the case, these classic queues probably get 
upgraded anyways at some point.

>     might be in the rough when consensus is checked)

Useful perspective; thanks.

>     potentially address issue 16.

I personally agree with this.  I think it's not specific to L4S at all 
though.  How bad flows are dealt with in general will also be applicable 
to a buggy or malicious L4S flow.  So I personally think the gate for 
L4S work in this regard should simply be WG confidence that the 
situation is not made substantially worse by L4S flows (whether they are 
legitimate or not).

>     no forced reason that a sender necessarily has to respond to loss either...

I'm in agreement with your remarks here, and think this might be a good 
starting point for material in operational guidelines.  In regards to 
the totally valid concerns that operators not doing L4S won't be reading 
operational guidelines drafts for L4S, I think we should try to rely on 
things that are effective and useful anyways without regard to L4S 
deployment.  FQ is a good example, that already exists in some form in 
many places, and will continue to without regard to L4S.  So, if it 
helps to keep a buggy or maliciously-marked L4S flow in check, this is good.

>     targeted experimentation with the classic queue detection work.

This seems like it's useful for coordinating experiments and working on 
some of the research questions too.

However, in the case of malicious flows wrongly saying they're L4S, I 
don't think it does anything.  If there were some bug or issue found, it 
may create a way to exploit that.  That said, I'm not sure how much of 
an actual concern this should be.

---

From: Wesley Eddy <[email-masked]>
Date: Mon, 22 Jun 2020 11:49:11 -0400

Hi Rod, I just have a short comment from what I have seen as the WG 
interests.

I have not noticed much interest in DSCP global traversal, especially 
from the parties that would need to support it.  Since that is also 
explicitly against the DiffServ architecture that incorporates 
re-marking, suggesting it is a rather big change to all of how DiffServ 
has been defined, unless I misunderstand.  I believe any activity or 
proposals on global DSCP traversal are very interesting and totally fine 
to discuss, but can't be a gate for L4S.

---

From: Martin Duke <[email-masked]>
Date: Mon, 22 Jun 2020 10:23:31 -0700

Hi Jake,

Thank you so much for writing this up. I have been hoping that an
open-minded person would create an entry point to this conversation for
people that can't wade through thousands of emails on this topic.

In fact, what would you think about making this a findable, living document
by making it an individual draft? I wouldn't much care if it was adopted,
much less made an RFC, but it would be a great way for people to catch up,
and would create a simple way to both add new ideas and remove things if
they have no support in the WG?

Martin

On Fri, Jun 19, 2020 at 6:34 PM Holland, Jake <jholland=
[email-masked]> wrote:

> the

---

From: "Holland, Jake" <[email-masked]>
Date: Mon, 22 Jun 2020 19:15:58 +0000

Hi Wes,

Thanks for your comments, I mostly agree with your remarks.

However, there was one point I thought needed a response:

> situation is not made substantially worse by L4S flows (whether they are 

I don't think it's right to conflate legitimate and illegitimate flows
here.

There's a difference in kind (and in the reasonable set of responses)
between 1. a one-off buggy or cheating sender, vs. 2. a DDOS attack, vs.
3. a growing population of legitimate endpoints that are by design
incompatible with existing network devices.

There are a number of countermeasures[*] that are completely reasonable
for illegitimate traffic (#'s 1 and 2), but that are much less reasonable
if trouble is caused by a growing population of legitimate endpoints.

So while I agree that this issue is not specific to L4S at all, to me
the reasoning is the other way around:
    Although L4S now seems unsafe to roll out at scale because of its
incompatibility with classic queues, there could be a good set of
solutions to "overburdened AQM response" that could change matters
once deployed.  So digging in this direction might be able to unblock
L4S, in addition to improving AQM operations in general. (But on a
long timeline.)

Or to put my take on it another way:
    L4S flows *would* make the situation substantially worse if they
started rolling out now.  But it might be possible in the future to make
it so they *would not* make things worse, with good solutions to this
other problem.

Best regards,
Jake

[*] to be more specific:
For #1 (a one-off buggy or cheating sender), the easiest and most
common initial response for an operator is "ignore it, it'll go away".
I've certainly run my share of 1-off flow experiments on the internet,
and most of the time nobody notices or cares (and once in a while you
get spanked a bit).

But when you legitimize and start scaling those, it becomes hard or
impossible to distinguish #2 from #3, and it *would* be a problem to
apply some of the common #2 countermeasures to legitimate traffic
(like triggering routing black holes or suchlike).

If it comes up for the L4S case, absent a better solution, this is
exactly how the best response available to an operator ends up
landing on approach #4 (bleach ECT(1) to NECT).

But although I'm not a big fan of bleaching, I would much prefer a
consensus decision that it's the best answer to a consensus decision
that no answer is necessary (leaving bleaching as the undocumented,
yet only sane option available to operators when trouble happens,
and hoping it's the one they choose).

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Mon, 22 Jun 2020 21:27:12 -0700 (PDT)

Hello Wes,

Excuse me, but L4S itself even defined DSCP as one of the
alternative mechansims to ECT(1).  Are you saying we should
remove those alternatives from the draft?

Further on several occasions David Black, and others have asserted that
DSCP is a viable path forward for L4S.

How are these solutions suddenly not to be included in the considerations
of what may be a way to address Issue #16?  These are solutions that
can be used today, though the global traversal is not a requirement
for them to be used, and they are infact being used to conduct SCE
experimentation.  I find it unsettling that L4S proponets would not
consider this as a way to produce data that might influence the WG.

No comments inline,
Regards,
Rod

> from the parties that would need to support it.? Since that is also 
> has been defined, unless I misunderstand.? I believe any activity or

---

From: Wesley Eddy <[email-masked]>
Date: Tue, 23 Jun 2020 00:39:18 -0400

Hi Rod, I didn't say any of the things you're asking about; only that 
DSCP global traversal does not appear viable to depend on in the near term.

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 23 Jun 2020 09:25:13 +0200

Hi Wes,

Sure Global dscp traversal might not be a reality right now.
BUT that did not stop the introduction of the NQB internet draft that in essence tries to standardize a PHB/dscp that is pretty much the missing L4S dscp*. 
AND there are currently no data backing up the assertion that L4S will actually be useful in long-haul paths to beginn with**. For the only tested use case, essentially short RTT low hop count CDN to end user path, dscp traversal will not be a showstopper, just look at the rationale in the NQB draft.
And given the status of the NQB draft, this might reach maturity/publication well before the L4S drafts, so the appropriate L4S PHB will be defined although it is claimed that dscp will not work, anybody else seeing a disconnect here?
So instead of trying to optimize for universal roll-out maybe it would behoofe the L4S effort to aim a bit lower and gain some real world experience first, like walk before you run? 

Regards
        Sebastian

*) Currently the NQB draft excludes capacity seeking flows, but could be amended to allow such flows, if they use 1/p congestion signaling. As far as I can tell NQBs reason for existence is to have a method to have endpoints steer non ECT(1)-flows into potential LL-queues along the path. Such a method is also hinted at in one of the L4S drafts, so in spite of RSVPs being declared insufficient in all likelihood they will be used in the L4S ecosystem.

**) Testing with a single netem instance giving a constant delay of X milliseconds is not a realistic test of a high RTT high hop-count path, it is rather a test of a condition exceedingly rare in the real world, high RTT low hop count... I have been asking for realistic tests for some while now, and am beginning to assume that the lack of data might be an indication that the current L4S implementation and design simply does not cope well with the requested realistic conditions. With the existing testing harness used for the supporting papers testing the realistic scenarios I laid out in past emails should have been relative easy. I for one an done with giving L4S the benefit of the doubt.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.