Subject: [tsvwg] Position Statement on ECT(1)

From: "Holland, Jake" <[email-masked]>
Date: Sat, 25 Apr 2020 10:54:24 +0000

Hi tsvwg,

Since I contributed to the opening of this can of worms[IC] and have
been following its development closely, I thought I'd write up my
current position and intent regarding the ECT(1) decision the chairs
are looking for, in hopes that it's helpful to others.

I've included a "tl;dr" at the end with my current ranked preferences
for decision calls, but my position is perhaps more nuanced than some
might realize, so I ask that people not attribute positions to me on
the topic without reading through the details.

My intent is to watch the upcoming tsvwg interim and to send an update
to the list (and perhaps state it at the mic if it'll be really fast)
if anything presented substantially changes my views in any important
respects, and otherwise to try to stay out of it unless there's
something crucial unsaid.

Disclaimer:

I'm speaking as an individual.  I worry that some people have mistaken
my positions for official Akamai positions, but that is an erroneous
interpretation I would like to correct here.

As an IETF participant and an Akamai employee, I'm offering my technical
opinions in a way that I hope will benefit both the Internet and Akamai,
but I do not know of any specific Akamai position on this topic.  There
are many other technical experts employed by Akamai who may or may not
agree with me after digging into the technology.  Akamai's permission for
me to openly engage in technical discussion on this topic should not be
taken as an endorsement of my views.

Position:

ECT(1) should be a network output.

Main driving opinion behind the main position:

Based on observations to date, L4S should use DSCP as a classifier and
ECT(1) as output from the LL queue for the 1/p congestion signal.

Relevant Claims:

Taken together, the numbered claims are the pillars supporting my
position above.  The lettered sub-claims are either supporting points
of the claim or important consequences of it.  If I'm convinced there
is substantial and relevant error in these claims, I could change
my views above.

1. L4S with dualq is an important contribution.  It would be good if it
deploys and ultimately sees wide use.
1.a. The SCE-related congestion control work so far [SCE-CC] is NOT a
primary consideration for the ECT(1) network input/output decision.

2. TSVWG should not indirectly countermand IAB and IETF consensus as
currently documented in a BCP without explicitly achieving consensus
that it's best overturned.
2.a. Deploying an experiment that makes classic ECN undeployable would
indirectly countermand the BCP IAB consensus from RFC 7567.

3. Using ECT(1) as a classifier is likely to still be broken in the
implementations most likely to be rolled out, even if the currently known
problems get solved.
3.a. Congestion control is hard and stability of results is fragile to
subtle details in the control loop.
3.b. High-fidelity (1/p) marking with a ECT(1)->CE transition is inherently
ambiguous because of existing code that interprets ECT(1) according to
RFC 3168.
3.c. The proposed algorithm for classic queue detection is probably fragile
to subtle details, as well as changes to the competing traffic profiles and
network conditions.  It's inherently a tuned heuristic that cannot be made
robust to generalized conditions.
3.d. Intractable problems have been demonstrated previously [BADNESS].
Other likely scenarios that will occur in practice if deployed have been
requested and are unexamined (e.g. classic queue fairness vs. Reno,
another common competing congestion controller).  There is no reason to
believe these unexamined cases don't have other, separate intractable
problems that need different tuning.
3.e.  Misbehaving senders that failed at classic queue detection have no
means to discover their failure, so when it happens it remains persistent
unless externally corrected.

4. DSCP is a good choice for an L4S classifier.
4.a. DSCP is listed as technically viable in the L4S specs.
4.b. No available classifier is ideal, as described in the L4S specs.
4.c. DSCP is deployable in controlled environments.
4.d. It is better to start with a limited deployment that is safe than to
rush to deploy something that is likely to be broken.
4.e. L4S is arguably a class of service.  Using it requires the presence
of specific properties in the network path.
4.f. This reduces the useful deployment scope for L4S, but is more
appropriate and reversible than ECT(1) assignment for a usage as known
ambiguous signaling.

5. If DSCP is used as a classifier, ECT(1) is no longer necessary as a
classifier and can be used as a 1/p congestion signal.
5.a. This allows failover to the classic queue to use CE as a 1/sqrt(p)
signal instead of loss, which reduces the impact on competing LL traffic
in the event of malicious overuse, misconfig, or transient overload.
5.b. Reducing ambiguity of signaling increases resiliency to misconfig
and prevents most of the likely cases of fragile and unsafe behavior in
the many unexamined possible network scenarios.

Discussion:

Bob has presented the case for the utility of L4S's latency properties
with good link utilization many times, and I broadly agree.  The
performance for dualq with responsive endpoints across high-volume
aggregation-point bottlenecks provides an important step forward for
fixing latency.  And the aggregation-point bottlenecks are the most
important network location for most end-user connections where latency
needs fixing. (Hence claim #1.)

Although I still think the SCE team's congestion control related
contributions [SCE-CC] might have some value, support for that work in
shared bottlenecks is not a component of my position.  (Hence claim #1.a.)

The better applicability of dualq+L4S to aggregating devices is a critical
component of solving latency for end users, and a more important
consideration in the end.

That said, the SCE team has done outstanding work in general and their
contributions are highly relevant to this question, and by excluding the
congestion control work, I want to avoid excluding their contributions to
extensive testing and analysis that have directly impacted the known
properties of the different approaches, and call out this work as highly
relevant to this question.  I want to thank them and I encourage wg
participants to do more review and engagement with their work.

Also, their cc-related contributions would provide significant and well-
proven value wherever FQ systems can be deployed at a real bottleneck.  So
making that work deployable is a notable nice-to-have because of its help
in FQ, but not otherwise relevant to the decision, and not my primary
consideration here.  It's worth it, if necessary, to discard the excellent
work in [SCE-CC] if it means ending up with a latency solution that
handles aggregating devices well.

I wanted to make the above clear, because some of the points seem to get
conflated sometimes, and I don't think of this as a "SCE-vs-L4S" debate
so much as a "L4S: is it safe?  And if so, how necessary is its rapid
deployment?" open set of questions.

DSCP as a classifier is rejected in the L4S specs because it can't
reliably traverse the internet today.  This could be a valid reason to
use ECT(1) as a classifier if the algorithms were sufficiently robust to
all relevant network conditions.  However, this has not been demonstrated
and does not appear to be a plausible claim.

A key operational problem is that gross unfairness to competing traffic is
not visible to the offending sender, except via out of band feedback to
the sender's operators.  The failure hurts only others, not the sender
who failed at detection, and no feedback is available in the network to
indicate a problem.  This is especially dangerous when there is no easy
contact between the network operator and the sender, because the operator
may need to block or severely throttle the sender's traffic.  This makes
deployment at the endpoint particularly dangerous for major sources of
traffic who could be badly impacted by getting blocked or throttled.

It's my personal opinion that fixing DSCP is a more tractable problem than
finding a stable and robust version of the classic queue detection
algorithm.  Although it's not easy, it at least might be possible.  The
queue detection system, by contrast, could run for years successfully and
then start failing in a way invisible to the sender because of changes in
traffic patterns, because of its reliance on fragile heuristics rather
than unambiguous signaling, or any patterns that can be monitored.  As a
sender it's hard to see how it can be made robustly deployable when the
safety properties depend on this kind of algorithm.

As a related aside, it's also my personal opinion that the tsvwg should
make every reasonable effort to encourage the deployment of classic
RFC 3168 marking queues to reduce the impact of congestion in a way that's
compatible with widely deployed end hosts already.

Claim #3 implies that wide deployment of current implementations using
L4S with ECT(1) as a classifier would make classic marking single queue
AQMs undeployable in existing, otherwise compliant networks.  In
combination with Claim #2, this countermands existing BCP advice from
RFC 7567, and is therefore an inappropriate action for tsvwg to encourage
by moving forward with an approach that can be reasonably expected to
have these properties.

Adoption of ECT(1) as a network input for classification would work
against this goal, because of the ambiguity it introduces and the need
that implies for a fragile ambiguity correction mechanisms as part of
the congestion control loop.

tl;dr:
My preference ordering:
1. (best):       ECT(1) as network output
2. (acceptable): Defer the decision
3. (against):    ECT(1) as network input

References:

[IC] (Invitation to Chaos):
  - https://mailarchive.ietf.org/arch/msg/tsvwg/nZDdRYLy1588ph_ozDJvpNjwLTM/
[SCE-CC] The SCE team's congestion control work:
  - https://github.com/chromi/sce/
  - https://github.com/heistp/sce-l4s-bakeoff
[BADNESS] The charts showing outliers at 100:1 and typical cases over 10:1,
but only for Cubic. The impact on Reno or others is unknown, but theory
suggests it's expected more severe:
  - https://datatracker.ietf.org/meeting/105/materials/slides-105-tsvwg-sessa-5-l4s-presentation#page=15

---

From: Roland Bless <[email-masked]>
Date: Sat, 25 Apr 2020 14:39:59 +0200

Hi Jake,

thanks for the write up.
I agree in several points and expressed this several times
on the list already:

1. ECT(1) should be used as congestion signal (with a different
   semantics than CE). Typically, it will already trigger at
   the onset of (light) congestion. This would also be usable
   for the L4S approach, but could be used by other approaches
   as well.

2. A DSCP should be used as classifier for the L4S semantics
   (insertion into the L4S queue). Then the ECT(1) Codepoint
   should be used for marking in the L4S queue.

Regards,
 Roland

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 26 Apr 2020 02:47:22 +0100

Jake,

Thank you and double thank you for this.
It would be great if all discussion on this topic was given with 
reasoning like this. it allows a proper technical discussion.

Inline...

> indirectly countermand the BCP IAB consensus from RFC 7567.

[BB] There is nothing in RFC3168 that says an ECN AQM MUST act on ECT(1) 
if it acts on ECT(0). So anyone can implement or deploy an RFC3168 AQM 
that solely acts on ECT(0).

So, if ECT(1) is assigned to L4S, and someone wants to deploy RFC3168 ECN:
* if they believe that deployed Classic ECN Fallback algorithms in L4S 
senders are not sufficiently robust, they can apply the RFC3168 AQM to 
ECT(0) only
* if they believe otherwise, they can apply the RFC3168 AQM to both 
ECT(0) and ECT(1).

This might not be configurable with some existing 3168 implementations. 
So the tsvwg could choose to write a brief RFC to recommend that it is 
configurable, or even to recommend that 3168 AQMs only act on ECT(0), if 
we wanted to guide the industry in that direction. Or we could even 
write a standards track update to RFC3168 to mandate ECT(0)-only. 
Personally, I prefer "the market can decide".

So, an L4S experiment does not in any way have to make RFC3168 undeployable.
Then, whatever one's opinion of ECN Fallback, there's no implicit 
countermanding of the recommendation in RFC7567 to deploy ECN. It can 
either mean an RFC8311 experimental update to ECN, or RFC3168 ECN itself 
- two completely complementary options.

It is also important to note the sheer number of network operators who 
are supporting L4S ECN, and planning to deploy it. RFC3168 ECN never 
gathered such operator interest. But you're right that we have to be 
careful that L4S ECN doesn't harm the viability of its own mother (3168 
ECN).

> robust to generalized conditions.

[BB] I don't disagree with that. Working with heuristics makes me feel 
dirty. But, as the ecn-l4s-id draft says, there are not enough bits for 
the perfect solution - we have to decide where to make the compromise. 
That's the process we're going through again now.

> 3.d. Intractable problems have been demonstrated previously [BADNESS].

[BB] I've just read your interpretation of this chart from the end, 
which I'll paste here to give other readers the flow of logic, before I 
comment on it.

> but only for Cubic [snip]:

[BB] Yes, there are typical cases over 10:1 (16:1 is the worst). 
However, the "outliers at 100:1" are not what you think, from the sound 
of it. A 1-%ile value does not mean 1% of long-running flows ran with 
that rate when averaged over their duration.

The rate measurements in that chart were sampled over 1s intervals. I 
had a review comment to add that, so it's in the new version of the 
paper. So, a 1-%ile value of worse than 1:100 means 1% of the sample 
seconds were running at worse than 1/100 of the ideal rate (defined as 
half the link rate in this case with 2 flows).

In fact I believe the 3 cases that show 1:100 were actually zero 
(-infinity on a log plot), because otherwise we wouldn't have cut the 
axis off without showing them. They are v likely to be retransmission 
time outs. Even tho the Cubic flow is ECN-capable, the ECN marking rate 
can fleetingly exceed 25% at such a low BDP, which is the threshold at 
which ECN marking is disabled in favour of drop for that AQM.

In the 3 cases you're talking about, there are 2 flows in a 4Mb/s link 
with low base RTT (5, 10, 20ms). So, the congestion window is at its 
minimum of 2 in all case. And when there's a loss, such a low cwnd 
stalls until the retransmission time-out fires.

> problems that need different tuning.

[BB] Regarding figures for Reno instead of Cubic, in the early days, we 
did experiments with both. But it became tedious to double the size of 
the matrix of runs we had to explore when the Reno results were 
generally a constant factor lower than the Cubic results.

So I conducted a mini-survey around some of the major service providers, 
and established that they all used Cubic, with one Reno exception. Since 
then we've generally done Cubic-only experiments, but we can add a run 
with Reno to the "to do list" if you want some figures.

> unless externally corrected.

[BB] Very true. See later about how the Internet relies on researchers 
for this.

> classifier and can be used as a 1/p congestion signal.

[BB] The piece missing here is how to transition from experiment to full 
deployment if it succeeds.
If we used ECT(1) as the output, we would have to use DSCP for ever as 
the classifier.

If we start from a classifier that is routinely wiped at borders, it 
will lead to an experiment that remains confined to each network as an 
island, for decades.

But it's worse than that. Using a DSCP doesn't just confine the 
experiment to participating networks. Where there's DSCP wiping it 
confines the experiment solely to the on-net traffic in that network. 
Wherever a border wipes the DSCP, subsequent networks on the path will 
not be able to take part in the experiment. So if my access ISP wants to 
offer me L4S service for traffic coming towards me, a previous network 
on the path can prevent them from doing so, by just wiping the DSCP.

It is far preferable to use a classifier that remains intact e2e right 
from the start of the experiment. Network operators can still confine 
the L4S treatment to their network (by classifying on ECT(1) and IP 
addresses). But the signal itself remains intact e2e. So, with L4S, any 
network operator can join the experiment for their part of the path, 
without having to persuade their neighbours not to wipe the codepoint 
before it reaches them.

There is the possibility that operators are tempted to wipe the ECN 
field, but I think that will remain taboo given its role in e2e 
congestion control.

Also, L4S low latency is orthogonal to Diffserv bandwidth prioritization 
(see draft-briscoe-tsvwg-l4s-diffserv).
If ECT is used as the L4S classifier, that fits, because ECN is 
orthogonal to DSCP.
But if DSCP is used as the L4S classifier, we would burn twice as many 
DSCPs = two for every bandwidth prioritization class (or at least for 
most of them).

There's also the problem that RFC3168 and RFC4301 (IPSec) tunnels and 
overlays don't propagate ECT(1) if it changed from ECT(0) since it was 
encapsulated. Neither do a whole plethora of shim encapsulations (e.g. 
all the v6 over v4 ones, GRE, GTP, etc). All these overlays will be 
gradually updating to RFC6040, but these things take decades to change.

> in the event of malicious overuse, misconfig, or transient overload.

[BB] An overloaded link must not continue using ECN marking (RFC3168 and 
RFC7567 mandate this for v good reason). The sources causing the 
overload are clearly not responding, so continuing to send out CE marks 
merely shuts down the good guys, and the cause of the overload (whether 
by malice or misconfig) continues to drive down the goodies.

Loss has the dual properties of signalling congestion and shedding load. 
It has to remain as the 1/sqrt(p) signal.

Even if this was a workable proposal, it would be an extremely niche use 
of a whole valuable codepoint.

And, by using ECT(1) as an output, you burn ECT(0) and ECT(1) for ever.
Whereas with ECT(1) as input, eventually ECT(0) could be reclaimed for 
something else (perhaps sthg to do with flow startup?).

> the many unexamined possible network scenarios.

[BB] That is true. But it's one of many factors that all need to be 
balanced. Perfection in one factor that causes numerous others to be 
worse is not necessarily a good choice.

> needs fixing. (Hence claim #1.)

[BB] For the avoidance of doubt, pls can you define the term 
aggregation-point bottleneck - I assume you're talking about a BNG or a 
CMTS, etc. And perhaps you're also including a home gateway in the 
upstream direction. But in my experience 'aggregation point' is usually 
used for a switch that is not a bottleneck, so I'm just checking.

> shared bottlenecks is not a component of my position.  (Hence claim #1.a.)
[BB] Understood.

> traffic who could be badly impacted by getting blocked or throttled.

[BB] This problem is not unique to L4S. Every new CC faces this problem. 
And, although it can't easily be solved operationally, the Internet has 
always relied on non-operational solutions - researchers doing 
validation offline as well as measurements on live networks, etc etc.

> safety properties depend on this kind of algorithm.

[BB] You're right. This is inherently fragile. But this has to be seen 
in context.
* Per-"customer" scheduling will often confine any problems within each 
customer's pipe (that doesn't belittle the problem, but it does confine 
its scope).
* The consequences of incorrect detection are not dire (rather a bold 
statement - see {Note 1}).
* It is believed that shared RFC3168 bottlenecks are unlikely to be 
common (of course, where they are deployed, the affected users will 
care, irrespective of whether they are common).
* If L4S is successful, it will not be necessary to maintain these 
algorithms for ever - they are for transition (of course, that could be 
a long time, I'm just saying it's not for ever).

{Note 1} Where link rate is low, even without the algorithm, 'fairness' 
between L4S and Classic is not far off 1. It only diverges from 1 at 
higher link rates. You can see this here:
http://folk.uio.no/asadsa/ecn-fbk/results_v2.2/baseline/codel_release/
which shows the position without fallback enabled at all in a CoDel AQM 
(not FQ) at a range of RTTs and link rates, for different mixes of 
traffic flows.

IOW, the less there is to share out, the closer the least share is to 
equal. Or, unfairness only grows when there's plenty to share out, so 
the one getting least can still make good progress (not starved).

> the congestion control loop.

[BB] The fragility of the ambiguity still stands for any pre-existing 
RFC3168 shared queues. But I hope I've shown a way to deploy RFC3168 ECN 
now and in the future alongside L4S, but without this fragility.

I'd be interested to know if this and the other points I've made, 
particularly those about the 1%iles, have altered the overall balance of 
your thinking.

Again, thank you for giving reasoned arguments for your position, so we 
can have an intelligent debate. And thank you for your kind words too.

Cheers

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Jonathan Morton <[email-masked]>
Date: Sun, 26 Apr 2020 04:53:59 +0300

> On 26 Apr, 2020, at 4:47 am, Bob Briscoe <[email-masked]> wrote:
> 
> [BB] There is nothing in RFC3168 that says an ECN AQM MUST act on ECT(1) if it acts on ECT(0). So anyone can implement or deploy an RFC3168 AQM that solely acts on ECT(0).

On the contrary, in the first paragraph of section 5:

"Routers treat the ECT(0) and ECT(1) codepoints as equivalent."

 - Jonathan Morton

---

From: Greg White <[email-masked]>
Date: Sun, 26 Apr 2020 04:21:13 +0000

Well, I assume Bob meant RFC3168 as updated by RFC8311, which does explicitly allow this differential treatment.

   Section 5 of RFC 3168 specifies that "Routers treat the ECT(0) and
   ECT(1) codepoints as equivalent."

   This memo updates RFC 3168 to allow routers to treat the ECT(0) and
   ECT(1) codepoints differently, provided that the changes from RFC
   3168 are documented in an Experimental RFC in the IETF document
   stream.  The specific change to RFC 3168 is to insert the words
   "unless otherwise specified by an Experimental RFC in the IETF
   document stream" at the end of the above sentence.

-Greg

    

     - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Sun, 26 Apr 2020 08:16:14 +0300

> On 26 Apr, 2020, at 7:21 am, Greg White <[email-masked]> wrote:
> 

That still does not allow a purely RFC-3168 AQM to differentiate.  That behaviour has to be explicitly specified in *some other* RFC.  It would then no longer be an RFC-3168 AQM, but an RFC-XXXX AQM.

> [BB] There is nothing in RFC3168 that says an ECN AQM MUST act on ECT(1) if it acts on ECT(0). So anyone can implement or deploy an RFC3168 AQM that solely acts on ECT(0).

That is all I wanted to point out, for the moment.

 - Jonathan Morton

---

From: Greg White <[email-masked]>
Date: Sun, 26 Apr 2020 06:00:04 +0000

In this debate there has been a lot of discussion around maintaining compatibility with RFC3168 or (now) preserving the option to support RFC3168 going forward.   I think we need to recognize that RFC3168 is 20 year old technology that has never really caught on, in part due to the fact that the benefits of it end up being fairly marginal.

In my view, retaining the ability to support RFC3168 for now is fine, but I don't think that we need to treat it as a critical technology that needs to be preserved indefinitely.  Actually, assuming the L4S experiment concludes successfully, the IETF should consider sunsetting it.

-Greg

    

    
    
     - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Sun, 26 Apr 2020 09:02:15 +0300

> On 26 Apr, 2020, at 9:00 am, Greg White <[email-masked]> wrote:
> 

You mean, just when adoption of it is beginning to take off?  Would you apply the same criterion to IPv6?

 - Jonathan Morton

---

From: Pete Heist <[email-masked]>
Date: Sun, 26 Apr 2020 11:12:15 +0200

Regardless of RFC 3168 ECN’s benefits, I think it may have been easier to redefine the meaning of CE 5+ years ago than it is today, when there’s initiation and signaling happening more frequently. I happen to think that MD without drop is useful as long as drop is used in overload situations, and that the ABE response in RFC 8511 may increase that usefulness, but I respect that there’s a legitimate debate to be had there, when dropping packets also helps relieve congestion.

Pete

> On Apr 26, 2020, at 8:00 AM, Greg White <[email-masked]> wrote:
>

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 26 Apr 2020 12:23:37 +0100

Jonathan,

> "Routers treat the ECT(0) and ECT(1) codepoints as equivalent."

[BB] I said there is nothing that says MUST (in capitals).

You have selectively quoted one sentence without any MUST in it, which 
(correctly) says what routers are /expected/ to do, not what they MUST do.

Pls see the following section of RFC3168, which at the time was in the 
context of incremental deployment of the ECN Nonce, where the sender 
switched between ECT(0) and ECT(1): 
https://tools.ietf.org/html/rfc3168#section-11.2.1

    However, if an ECN-capable sender is
    using ECT(1), and the congested router on the path did not understand
    the ECT(1) codepoint, then the router would end up marking some of
    the ECT(0) packets, and dropping some of the ECT(1) packets, as
    indications of congestion.  Since TCP is required to react to both
    marked and dropped packets, this behavior of dropping packets that
    could have been marked poses no significant threat to the network,
    and is consistent with the overall approach to ECN that allows
    routers to determine when and whether to mark packets as they see fit
    (seeSection 5  <https://tools.ietf.org/html/rfc3168#section-5>).

So not only does RFC3168 allow implementers to do what I said it allows 
them to do, it also specifically says it poses no significant threat, 
even if the different ECT markings are within the same flow. Certainly 
there is still no conceivable danger in applying drop to ECT(1) only, 
particularly given RFC8311 recognizes that ECT(1) is not used by RFC3168 
senders, which made it possible for RFC8311 (standards track) to update 
RFC3168 for nodes taking part in an ECN experiment, as follows.

    This memo updatesRFC 3168  <https://tools.ietf.org/html/rfc3168>  to allow routers to treat the ECT(0) and
    ECT(1) codepoints differently, provided that the changes fromRFC  <https://tools.ietf.org/html/rfc3168>
    3168  <https://tools.ietf.org/html/rfc3168>  are documented in an Experimental RFC in the IETF document
    stream.

It is a common mistake to read an expected behaviour as a requirement. 
RFC8311 makes the same mistake when it relaxes this behaviour - it calls 
it a 'requirement' of RFC3168, but RFC3168 does not strictly /require/ 
it. You will also notice that there is no MUST or SHOULD in RFC8311 on 
this subject, because none is necessary.

To be clear, at the moment the standards track RFC8311 /advice/ for ECN 
routers not involved in an ECT(1) experiment is:

        Forwarding behavior as described inRFC 3168  <https://tools.ietf.org/html/rfc3168>  remains the
        preferred approach for routers that are not involved in ECN
        experiments, in particular continuing to treat the ECT(0) and
        ECT(1) codepoints as equivalent, as specified inSection 4.2  <https://tools.ietf.org/html/rfc8311#section-4.2>
        below.

I was involved in the drafting of that sentence, because RFC3168 router 
implementers were asking what they should do. I still believe it was the 
correct advice at the time, given the exact nature of the ECN 
experiments was still to be defined back then.

My take-away message is that implementers can produce standards 
compliant implementations of RFC3168 that apply drop to ECT(1), if they 
consider that will better complement an ECT(1) experiment going on 
around them. And the IETF could alter its guidance to recommend this, 
but the choice is already available to implementers anyway, so we don't 
have to go there unless we want to.

Bob

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 26 Apr 2020 13:56:21 +0200

Hi Bob,

I think we can agree that was ever intenden to consciously ignore/drop ECT(1) marked packets instead of escalating them to CE-marks. So you basically say, that after the L4S experiment started, all future RFC3168 compliant AQMs need to be changed to allow for that condition (which RFC3168 permits), correct?
Now let me ask the community, how realistic does anybody here take it that this is going to happen? As if all deployed implementations would change the very moment the ietf changes an RFC?

Best Regards
	Sebastian

> On Apr 26, 2020, at 13:23, Bob Briscoe <[email-masked]> wrote:
> 
> Jonathan,
> 
> 
> You have selectively quoted one sentence without any MUST in it, which (correctly) says what routers are /expected/ to do, not what they MUST do. 
> 
> Pls see the following section of RFC3168, which at the time was in the context of incremental deployment of the ECN Nonce, where the sender switched between ECT(0) and ECT(1): https://tools.ietf.org/html/rfc3168#section-11.2.1
>    (see 
> Section 5).
> So not only does RFC3168 allow implementers to do what I said it allows them to do, it also specifically says it poses no significant threat, even if the different ECT markings are within the same flow. Certainly there is still no conceivable danger in applying drop to ECT(1) only, particularly given RFC8311 recognizes that ECT(1) is not used by RFC3168 senders, which made it possible for RFC8311 (standards track) to update RFC3168 for nodes taking part in an ECN experiment, as follows.
> 
> 3168
>  are documented in an Experimental RFC in the IETF document
>    stream.  
> 
> 
> It is a common mistake to read an expected behaviour as a requirement. RFC8311 makes the same mistake when it relaxes this behaviour - it calls it a 'requirement' of RFC3168, but RFC3168 does not strictly /require/ it. You will also notice that there is no MUST or SHOULD in RFC8311 on this subject, because none is necessary.
> 
> To be clear, at the moment the standards track RFC8311 /advice/ for ECN routers not involved in an ECT(1) experiment is:
> 
>        Forwarding behavior as described in RFC 3168
>  remains the
>        ECT(1) codepoints as equivalent, as specified in 
> Section 4.2
> 
>        below.
> 
> 
> I was involved in the drafting of that sentence, because RFC3168 router implementers were asking what they should do. I still believe it was the correct advice at the time, given the exact nature of the ECN experiments was still to be defined back then.
> 
> 
> My take-away message is that implementers can produce standards compliant implementations of RFC3168 that apply drop to ECT(1), if they consider that will better complement an ECT(1) experiment going on around them. And the IETF could alter its guidance to recommend this, but the choice is already available to implementers anyway, so we don't have to go there unless we want to.
> 
> 
> 
> Bob
> 
> 
>>  - Jonathan Morton
> 
> --

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 26 Apr 2020 23:06:25 +0100

Sebastian

> I think we can agree that was ever intenden to consciously ignore/drop ECT(1) marked packets instead of escalating them to CE-marks. So you basically say, that after the L4S experiment started, all…

[BB] I didn't say "need to be changed", because the need is not clear - 
which is why I said it can be up to each implementer.

> Now let me ask the community, how realistic does anybody here take it that this is going to happen? As if all deployed implementations would change the very moment the ietf changes an RFC?

Jake's concern and my proposed resolution of it, was about L4S 
suppressing any incentive to deploy RFC3168 in the future.

This has nothing to do with any pre-existing already deployed RFC3168 
AQMs, which clearly don't need any further incentive to be deployed. You 
said yourself that this applies to 'future RFC3168 compliant AQMs' when 
you summed up my suggestion.

Bob
>
> Best Regards
> 	Sebastian
>
>
>> On Apr 26, 2020, at 13:23, Bob Briscoe<[email-masked]>  wrote:
>>
>> Jonathan,
>>
>>
>>>> On 26 Apr, 2020, at 4:47 am, Bob Briscoe<[email-masked]>
>>>>   wrote:
>>>>
>> Pls see the following section of RFC3168, which at the time was in the context of incremental deployment of the ECN Nonce, where the sender switched between ECT(0) and ECT(1):https://tools.ietf.org/html/rfc3168#section-11.2.1

-- 
________________________________________________________________
Bob Briscoehttp://bobbriscoe.net/

---

From: "Holland, Jake" <[email-masked]>
Date: Mon, 27 Apr 2020 00:07:47 +0000

> Thank you and double thank you for this.

My pleasure, and I'm glad to hear it's something you can engage with :)

I'm going to trim to just what I think are the parts I'm responding to,
but if anybody wants to follow the full context in case something
important is left out, I thought I'd include a link to your message for
easier reference:
https://mailarchive.ietf.org/arch/msg/tsvwg/ShTE1QTw81cDLqGfB_6-xTzmFYE/

> that solely acts on ECT(0).

A fair point for future deployments.  But I note that many people have
for example linux boxes capable of forwarding at line rate for useful
links, which have tc-pie readily available.  It's not so clear that such
evaluations might not already be ongoing, given the increased endpoint
use of ECN in recent years, and I would prefer not to tell these folks
that they'll have to wait for new updates before rolling something out
safely.

> that rate when averaged over their duration.

Thanks for the clarification.

> If we used ECT(1) as the output, we would have to use DSCP for ever as 

Either that or be open to changing the signaling after more operational
experience.  But yes, the overall point is well taken that DSCP is not
a great way to encourage wide deployment, nor a great way to establish
useful stable signaling, only an incremental step that might allow for
some interoperable production use for further examination.

My point is that this is better than committing to something that seems
like it's likely to have operational problems.

> field, but I think that will remain taboo given its role in e2e 

Yes, this is part of my worry.  If the way things work gets fragmented
in a way that ends up causing operational problems, we could easily see
ECN go the way of DSCP instead of the opposite.

> (see draft-briscoe-tsvwg-l4s-diffserv).

Also a fair point.  Again, my only response is that to me it seems better
than the alternative, though not ideal.

> gradually updating to RFC6040, but these things take decades to change.

It might be helpful to have some visibility on how this stands now, and
what kinds of tunnels are in wide use.  I'm not really clear on how big
a problem this is, and at present I assume most just black it out or
something, but really I have no idea.  I'm roughly assuming that the
tunneling deployments are more upgradeable than the queues, but that
as an endpoint we'll be dealing with weird inconsistencies that basically
never fully understood anything beyond "don't lose CE marks" and maybe
"loss is acceptable in confusing cases", if we're lucky.

I'd be interested to hear more about implementation and deployment status
on this, but it's a side question, in that it doesn't make the alternate
L4S CE semantics safely deployable.  If the tunnels can't be made
compatible with a safe answer to the signaling, it's not clear how they
can be made compatible with a safe deployment of LL.

While typing this I just noticed you mentioned mobile networks universally
using tunnels, which is an interesting point.  But I thought many that
moved to IPv6 weren't using tunnels for the IPv6 portion?  I guess I'm
interested in the scope of the tunnel deployments over shared bottlenecks
today, but I'm not sure how much it matters for this question.

> used for a switch that is not a bottleneck, so I'm just checking.

Yes, I'm intending to capture devices like CMTS, BNG, ONT, and similar
access points with many clients to the immediate south in the network and
typically bw-throttled according to the price they're paying.

> particularly those about the 1%iles, have altered the overall balance of 

Not really, sorry.  I think the main thrust of the point still stands,
though there's still some reading to do to make sure I have fully grasped
everything.

So at the moment I'm in roughly the same spot.  I recognize that my
position would not be a way to achieve a rapid rollout, but to me it
looks like one of the better paths forward, because the downside of
moving along with ambiguous signaling and having it go poorly in
deployment seems even worse, and not an unlikely outcome.

Best regards,
Jake

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 27 Apr 2020 05:16:42 +0100

Jake,

> that they'll have to wait for new updates before rolling something out

Treating ECT(1) as if it's Not-ECT is effectively changing a classifier. 
I know any small change is a major thing. However, cutting the scope of 
a classifier down from 2 codepoints to 1 is hardly rocket science. And 
implementers have as long to do this as it will take for the whole of 
L4S to be productionized.

> ECN go the way of DSCP instead of the opposite.

[BB] You might have seen an idea I posted on this list where the last 
bit of the L4S ECN field could be included in integrity coverage. 
Probably impractical to change integrity coverage, but something to mull on.

> something, but really I have no idea.

[BB] We do know that >50% of the Internet is now mobile, which is all 
compressed, encrypted, and multiply encapsulated. Mostly with GTP.

It feels like many tsvwg folks don't have a good feel for just how much 
encapsulation there is in the Internet, which might be why this issue 
tends to always be glossed over. I can't properly quantify the degree of 
encap in the Internet I'm afraid, but I can describe some of it beyond 
mobile already covered above.

For instance, in ex public telephone and telegraph companies (mostly DSL 
and PON), there's a significant amount of L2TP, at least in Europe. When 
I left BT about 5 years ago, the proportion was approaching 50% of lines 
encapsulated with L2TP. This is how "local loop unbundling" is achieved 
- the incumbent operates the lower layer as a wholesaler and the 
networks of the "unbundled" retail ISPs are encapsulated as overlays. 
Not all unbundled ISPs are encapsulated - only the bigger ones who want 
to operate more independently of the underlying wholesaler.

Don't know about satellite.

I understand DOCSIS Cable is pretty free from any encapsulation 
/deployed by the operator/.

But then, as well as the encapsulations deployed by the operator, there 
are overlays, often from host to host, or host to proxy. For instance 
for v6 over v4.

Similarly, there are all the different VPN encapsulations. Not just 
end-user VPNs, but all the overlay technologies used in and across data 
centres. Not necessarily encrypted VPNs, but at least routing overlays. 
Remember there are low latency deployment scenarios in DCs as well.

So, there must be no less than 60-70% of the internet conducted over 
tunnels or encapsulations.

> I'm roughly assuming that the
> tunneling deployments are more upgradeable than the queues,

I would have said there's no difference. Most tunnels will start or end 
on a node that's there for other reasons, not specifically to do 
tunnelling. But there are obviously tunnelling-specific boxes.

The major factor though is that the tunnel endpoints will often be in 
other networks, or operated by some entity different from the one who 
operates the bottleneck. So, the folk who operate the queue will have no 
business relationship with those you would get a tunnel upgraded.

> but that

[BB] See reply to Dave Taht just now, which pretty much confirms what 
you've just said.

All the above reasons are why L4S avoids anything that requires changes 
to encapsulations.

> can be made compatible with a safe deployment of LL.

[BB] This is a judgement call. I don't share your desire for perfection 
on the fairness front. Personally, I think 1:10 is fine as long as it 
approaches closer to 1:1 for low rate links. If it's occasionally 1:16 
on high rate links, that wouldn't bother me.

But I've been prepared to try to solve this problem for those like you 
that do care. However, it does seem that no matter how well I do it, 
I'll just be told "it might not work". So I'm beginning to wonder why 
I'm even trying to shoot for ever-moving goal posts.

> moved to IPv6 weren't using tunnels for the IPv6 portion?

AFAIK, v6 is no different. The tunnelling is for mobility and for 
encryption.

> I guess I'm

[BB] My point above in a nut-shell - most transport layer people just 
aren't really aware of all the encapsulation they are sitting over.

Tunnel deployment is hugely important.

> deployment seems even worse, and not an unlikely outcome.

[BB] The other point we haven't really talked about is TCP. Without 
precise feedback, you can't get close to ultra-low delay. And there's 
not enough space in TCP to feed back two outputs without waiting for 
years for an option to become traversible through middleboxes.

But I'm going to stop there.

Regards

Bob

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 27 Apr 2020 14:04:34 +0300

> On 27 Apr, 2020, at 7:16 am, Bob Briscoe <[email-masked]> wrote:
> 
> I don't share your desire for perfection on the fairness front. Personally, I think 1:10 is fine as long as it approaches closer to 1:1 for low rate links. If it's occasionally 1:16 on high rate links, that wouldn't bother me.

Nevertheless, there is a Y in the boxes claiming Fairness on the L4S side, which in light of both test results and the above statement, is a little questionable.  I don't think a 1:10 ratio in throughput at the same RTT (after reaching steady state) would meet that criterion, unless it were explainable by a difference in growth function at high BDPs as in CUBIC vs. Reno.

A 2:1 ratio is easier to make that claim for.  At the very least, the CodelAF approach ensures that no flow is starved, even approximately - and even if there is a difference in RTTs.

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 27 Apr 2020 12:26:39 +0100

Jonathan,

> Nevertheless, there is a Y in the boxes claiming Fairness on the L4S side, which in light of both test results and the above statement, is a little questionable.  I don't think a 1:10 ratio in thro…

[BB] The 1:10 or 1:16 figures I stated above were in the context of 
abandoning any attempt at a Fall-back algorithm.

That 'Y' for fairness in the L4S table is /with/ the Fallback algorithm.
In our tests of the algo so far, 48% is the lowest that Cubic has been 
reduced to by Prague when running /with/ the Fallback Algo.

> A 2:1 ratio is easier to make that claim for.  At the very least, the CodelAF approach ensures that no flow is starved, even approximately - and even if there is a difference in RTTs.

Yes, we haven't added a whole run with mixed RTTs yet. You can see 
that's in the plan:
http://folk.uio.no/asadsa/ecn-fbk/results_v2.2/full_heatmap_rrr/

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 27 Apr 2020 12:54:25 +0100

Jake,

Important omission: I've been focusing on tunnel decap, but there's a 
big problem with using Diffserv at encap...

I've just been told about traversal with a typical IPSec tunnel product 
(wireguard). IPSec tunnels copy ECN, but not Diffserv, to the outer on 
encap. As it says here 
https://www.wireguard.com/protocol/#diffserv-considerations:

"All transport data packets have a DSCP value of 0, because the DSCP 
value of the inner packet is /never/ copied to the outer packet, so that 
we don't leak information about the data inside the encrypted inner 
packet. However, we do copy the ECN bits to and from the inner packets, 
in accordance with the logic described in RFC6040." That logic was 
carried over from the IPSec architecture [RFC4301].

When the IETF security area prepared the new IPSec architecture they 
wanted as few knobs as possible. So, rather than add knobs for allowing 
or blocking mutable fields that could be used as covert channels, they 
decided that the 2-bit ECN field was a "manageable risk", whereas the 
6-bit DSCP was not. This was a big thing in the Security Area (saag) at 
the time, and RFC6040 was well-received, because it normalized all 
tunnels to the approach taken by IPSec.

This particular person was arguing with me for your approach until they 
discovered it wouldn't get into IPSec tunnels. Now they're in a quandry.

Bob

On 27/04/2020 05:16, Bob Briscoe wrote:
> Treating ECT(1) as if it's Not-ECT is effectively changing a 
> classifier. I know any small change is a major thing. However, cutting 
> the scope of a classifier down from 2 codepoints to 1 is hardly rocket 
> science. And implementers have as long to do this as it will take for 
> the whole of L4S to be productionized.
>
>
>>
> It feels like many tsvwg folks don't have a good feel for just how 
> much encapsulation there is in the Internet, which might be why this 
> issue tends to always be glossed over. I can't properly quantify the 
> degree of encap in the Internet I'm afraid, but I can describe some of 
> it beyond mobile already covered above.
>
> For instance, in ex public telephone and telegraph companies (mostly 
> DSL and PON), there's a significant amount of L2TP, at least in 
> Europe. When I left BT about 5 years ago, the proportion was 
> approaching 50% of lines encapsulated with L2TP. This is how "local 
> loop unbundling" is achieved - the incumbent operates the lower layer 
> as a wholesaler and the networks of the "unbundled" retail ISPs are 
> encapsulated as overlays. Not all unbundled ISPs are encapsulated - 
> only the bigger ones who want to operate more independently of the 
> underlying wholesaler.
>
> But then, as well as the encapsulations deployed by the operator, 
> there are overlays, often from host to host, or host to proxy. For 
> instance for v6 over v4.
>
> end-user VPNs, but all the overlay technologies used in and across 
> data centres. Not necessarily encrypted VPNs, but at least routing 
> overlays. Remember there are low latency deployment scenarios in DCs 
> as well.
>
> I would have said there's no difference. Most tunnels will start or 
> end on a node that's there for other reasons, not specifically to do 
> operates the bottleneck. So, the folk who operate the queue will have 
> no business relationship with those you would get a tunnel upgraded.
>
>
>> but that
> All the above reasons are why L4S avoids anything that requires 
> changes to encapsulations.
>
>>
> [BB] This is a judgement call. I don't share your desire for 
> perfection on the fairness front. Personally, I think 1:10 is fine as 
> long as it approaches closer to 1:1 for low rate links. If it's 
> occasionally 1:16 on high rate links, that wouldn't bother me.
>