Subject: [tsvwg] SCE and tunnels / encapsulations

From: Bob Briscoe <[email-masked]>
Date: Mon, 27 Apr 2020 00:57:28 +0100

Jonathan, and other SCE contributors

This is one of the factors that strongly distinguishes L4S from SCE, so 
it's important.

In the slides you have contributed for tomorrow's tsvwg meeting, the 
question put by the chairs for #11 was:
"ECT(1) propagation through current Internet (tunnels, lower layers). ":

The SCE responses regarding tunnels are:
1.    "ECT(1) traverses the Internet well, due to foresight in RFC-3168."
2.    "Update to RFC-3168 desirable to obtain full benefits on tunnel path"

These sentences are both falsehoods.

Sentence #1:
Where an AQM applies ECT(1) (SCE) marks to the outer headers of 
encapsulated packets, as an RFC3168 decapsulator strips off the outer 
header, all SCE marks will just disappear. It only propagates CE marking 
to the onward packet (which is one of the main reasons L4S solely uses 
CE as an output from the network).

Sentence #2:
Without an update to RFC3168, there are no benefits from SCE on a tunnel 
path. So an update to RFC3168 is not merely desirable, it is essential, 
to obtain /any/ SCE-related benefit on a tunnel path.

We need to get real here. This is not a research project. This is not a 
debating game. More than 50% of the Internet is now over mobile and all 
mobile networks use tunnels. Overlays and encapsulations are widely used 
over the rest of the Internet, to support v6 over v4, VPNs, retail over 
wholesale ISPs, virtual network overlays, etc, etc.

As you know, RFC6040 is the update to RFC3168 that SCE needs.
Q1. Have you tested whether SCE works over tunnels that claim to comply 
with RFC6040?
Q2. Have you attempted to measure the uptake of RFC6040 in tunnels over 
the Internet?

Regards

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Dave Taht <[email-masked]>
Date: Sun, 26 Apr 2020 17:33:13 -0700

At one point I had done a thorough survey of all the open source
network stacks in the world,
and largely found they copied both bits back and forth. But I've lost
my notes on this and do not
remember clearly.

-- 
Make Music, Not War

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-435-0729

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 27 Apr 2020 03:44:21 +0300

> On 27 Apr, 2020, at 2:57 am, Bob Briscoe <[email-masked]> wrote:
> 

No, they are not.  I think you should be careful with statements like that, lest they be reflected.

> Sentence #1:
> Where an AQM applies ECT(1) (SCE) marks to the outer headers of encapsulated packets, as an RFC3168 decapsulator strips off the outer header, all SCE marks will just disappear. It only propagates CE marking to the onward packet (which is one of the main reasons L4S solely uses CE as an output from the network).

But an ECT(1) mark applied before the tunnel encapsulation point would be preserved, correct?  L4S relies on that fact, I believe.  And CE marking is sufficient for effective congestion control; SCE is formulated as a performance enhancement whose absence or erasure does not break stuff.

As you point out, there is an RFC-6040 update in the works which will improve the situation with tunnels.  It seems likely that will get published as an RFC before SCE does.  So we are satisfied with the idea that SCE will work better with tunnels that follow the new specification than the old one, but will retain effective congestion control on the old ones too.

> Sentence #2:
> Without an update to RFC3168, there are no benefits from SCE on a tunnel path. So an update to RFC3168 is not merely desirable, it is essential, to obtain /any/ SCE-related benefit on a tunnel path.

Again, SCE is formulated as a performance enhancement, not a wholesale replacement for ECN.  An SCE node through which tunnelled traffic happens to pass will still give its benefits to other traffic, even if the tunnel erases the extra signal at its egress.  A misbehaviour which, by the way, is not mandated by RFC-3168, only under-specified.  It is likely that many implementations actually do something sensible (and useful for SCE) here.

And once again you conflate the fragment reassembly rules set out by RFC-3168 and the decapsulation rules set out by RFC-6040.  You are now talking about what *might* happen if some fragments of a packet receive an SCE mark which others do not.  As currently specified in RFC-3168, some or all of those marks might be erased on reassembly.  SCE has a closed control loop which adapts to that situation, but as I stated, an update to RFC-3168 can be formulated which improves this behaviour.

We have already had a long and argumentative debate on this very subject.  If you refuse to even understand the arguments I am presenting - whether or not you agree with them - then please just drop the subject.

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 27 Apr 2020 04:09:59 +0100

Jonathan,

> No, they are not.  I think you should be careful with statements like that, lest they be reflected.

[BB] Of course I was extremely careful of my facts before saying 
anything like that.

> But an ECT(1) mark applied before the tunnel encapsulation point would be preserved, correct?

[BB] Of course it would. That's not the point at issue, and you know it.

It is not appropriate, for such an important issue as this, to take the 
chairs' slightly mis-worded question and answer it literally in order to 
evade what you know was the real question they were asking. That's why I 
said this is not a debating game.

>   L4S relies on that fact, I believe.  And CE marking is sufficient for effective congestion control; SCE is formulated as a performance enhancement whose absence or erasure does not break stuff.

[BB] Yes. That's why the second sentence is a falsehood:

Here is the petty debating game you're playing:

* When the RFC3168 tunnel strips the SCE signal, you define the benefits 
that base ECN provides as benefits that SCE provides (even tho we 
already have those benefits without burning a codepoint for SCE).

* Then you imagine you can use the words 'full benefits' for the 
benefits SCE offers, to make it sound like SCE still offers something 
less than full benefits when an RFC3168 tunnel strips it; when actually 
it provides no benefit that was not already there.

It is not pleasant to have to call out this sort of behaviour. I had 
asked the chairs to approach you on another misleading statement during 
the process of developing these slides together. I now notice that you 
added these new 'trick words' in the last round of edits before the 
slides were frozen.

> As you point out, there is an RFC-6040 update in the works which will improve the situation with tunnels.  It seems likely that will get published as an RFC before SCE does.  So we are satisfied wi…

[BB] The timescale of changes to tunnels is typically measured in decades.

Please just state what the position is, without all the games and tricks.

To provide benefits, SCE needs changes to sender, receiver, bottleneck, 
and any tunnel endpoints that might enclose the AQM.

2-part deployment is hard enough, so if you are proposing a 4-part 
deployment, especially involving changes to a myriad of different 
encapsulations, that has to be justified or mitigated somehow.

You're asking for an extremely scarce resource to be assigned to SCE; if 
you think you can get it though ducking, diving and dodgy tricks, you 
clearly have no idea of the roasting that a proposal like this will be 
subjected to in the IETF.

> Again, SCE is formulated as a performance enhancement, not a wholesale replacement for ECN.  An SCE node through which tunnelled traffic happens to pass will still give its benefits to other traffi…

[BB] Oh my word. This is unbelievable. I'm trying to understand how 
anyone could possibly think this stuff.

RFC3168 says:

    On decapsulation, if the CE codepoint is
    set on the outside header, then the CE codepoint is also set in the
    inner header.  Otherwise, the ECN codepoint on the inner header is
    left unchanged.

So if the outer has been changed from ECT(0) to ECT(1), the inner will 
remain at ECT(0). IOW, an SCE signal will be stripped. That's neither 
under-specified nor misbehaviour. It's what was intended at the time.

> We have already had a long and argumentative debate on this very subject.  If you refuse to even understand the arguments I am presenting - whether or not you agree with them - then please just dro…
What? I didn't say anything about fragments or reassembly.

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 27 Apr 2020 04:17:19 +0100

Dave,

> my notes on this and do not

Should copy ECN field to outer on encap, but shouldn't blindly copy them 
back from outer on decap.
Forwarded ECN field should depend on combination of outer and inner (see 
table at github URL below).

Earlier this evening, Olivier and I checked all the different tunnel 
code in Linux that we could find.

Most followed RFC3168 behaviour (i.e. propagating CE but stripping off 
ECT(1))
Foo over UDP (and therefore Generic UDP Encap (GUE)) copied nothing on 
decap, just stripped the outer off.

None copy ECT(1) from the outer when there's ECT(0) on the inner.

The table of how it's meant to be done is even pasted from RFC6040 into 
comments, but the code doesn't actually do what the table says in the 
one case of an ECT(1) outer and ECT(0) inner.
https://github.com/torvalds/linux/blob/master/include/net/inet_ecn.h#L163

macOS/xnu-darwin propagates ECT(1) correctly per RFC6040.
https://github.com/apple/darwin-xnu/blob/a449c6a3b8014d9406c2ddbdc81795da24aa7443/bsd/netinet/ip_ecn.c#L105 
<https://slack-redir.net/link?url=https%3A%2F%2Fgithub.com%2Fapple%2Fdarwin-xnu%2Fblob%2Fa449c6a3b8014d9406c2ddbdc81795da24aa7443%2Fbsd%2Fnetinet%2Fip_ecn.c%23L105&v=3>

Bob

---

From: Dave Taht <[email-masked]>
Date: Sun, 26 Apr 2020 21:18:43 -0700

thank you for taking a look.

It would be good to know what cisco and juniper actually do, by
spitting these bits into them
they are all available as vms now, cisco has 3 major linux versions in their
stacks so far as I recall, pretty ancient, and for all I know do some
tunnel types in hw.

openvpn and other userspace vpns would be good to look at also.

On Sun, Apr 26, 2020 at 8:17 PM Bob Briscoe <[email-masked]> wrote:
> Should copy ECN field to outer on encap, but shouldn't blindly copy them back from outer on decap.
> Forwarded ECN field should depend on combination of outer and inner (see table at github URL below).

> Earlier this evening, Olivier and I checked all the different tunnel code in Linux that we could find.
>
> Most followed RFC3168 behaviour (i.e. propagating CE but stripping off ECT(1))

Yea, that was (I think), essentially my conclusion, long predating l4s and sce.

I am under the impression that ipsec tunneling actually doesn't do
even that much, by
default. Please feel free to correct my bad memories! but get some sleep!

> Foo over UDP (and therefore Generic UDP Encap (GUE)) copied nothing on decap, just stripped the outer off.

a problem for both of your proposals.

My explorations LONG predate foo. so far as I know that was tom
herbert and facebook's thing
Is it deployed to any extent?

> None copy ECT(1) from the outer when there's ECT(0) on the inner.

So I'm kind of confused now. But: ECT(1) on the inner does get copied
to the outer?

But: not the reverse?

I'd thought the nonce idea was sort of respected in the field,
and that NOT passing that along e2e kind of made sense. Or maybe the nonce idea
worried about the info leak too much?

> The table of how it's meant to be done is even pasted from RFC6040 into comments, but the code doesn't actually do what the table says in the one case of an ECT(1) outer and ECT(0) inner.

Sigh.

> https://github.com/apple/darwin-xnu/blob/a449c6a3b8014d9406c2ddbdc81795da24aa7443/bsd/netinet/ip_ecn.c#L105

Good to know.

Easy patches throughout, but the long tail here is a decade+ long.

-- 
Make Music, Not War

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-435-0729

---

From: Dave Taht <[email-masked]>
Date: Sun, 26 Apr 2020 22:35:52 -0700

I learned something I didn't know btw. ping -Q "value" on linux, at
least - sets the same value on the receive as the send.
they took "echo" seriously way back then.

and generating frags can be "interesting" when transiting a tunnel.
Abusing one of my wireguard tunnels
as a test:

ping -s 1500 -f -Q 1 never.you.mind.ip
ping -s 1500 -f -Q 2 never.you.mind.ip

shows that 2 can change to 3 by the aqm and be turned back into 3 at
the egress point (inner)
shows that 1 can change to 3 by the aqm and but *not* be turned back
into 3 at the egress point (inner), it stays at 1

with what? 20%? 60%? of the world transiting tunnels? this is a major
deployment problem for both l4s and sce.

--
Make Music, Not War

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-435-0729

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 27 Apr 2020 08:03:53 +0200

Hi Dave,

Good point.

But I can not help myself noting that all of this assumed that the heavily tunneled paths we are talking about will actually roll out a pist-rfc3168 AQM. I for one do not believe that the business case for doing that had changed one iota either by L4S or by SCE, so at least for initial roll-out I consider this to be a non-issue.
I also add one level of confusion I spit in Bob's argument:
In the tunneled traffic of day a complete end user link or of a VPN, how can you expect exclusively L4S traffic to be present? So I question that even if we had AQMs on tunneled paths that doing the 1/p congestion control feed back would be a safe thing to do. And leaking individual packet's ECT(1) out of a VPN is IMHO severe enough an isolation violation to stop using that VPN for good. But again, I really do not see the internet growing AQMs at the problematic places any time soon, and I assume even with ECT(1) as output we would have time to prepare not only safe but also high fidelity tunnel passage should this become a limiting step in the future. 

Best 
        Sebastian

On 27 April 2020 07:35:52 CEST, Dave Taht <[email-masked]> wrote:
>> > The table of how it's meant to be done is even pasted from RFC6040
>into comments, but the code doesn't actually do what the table says in
>the one case of an ECT(1) outer and ECT(0) inner.
>> >
>> > 1.    "ECT(1) traverses the Internet well, due to foresight in
>RFC-3168.."

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

---

From: Dave Taht <[email-masked]>
Date: Sun, 26 Apr 2020 23:43:09 -0700

ya know? I think everybody should get some sleep.

certainly a patch needs to get prepared and queued up for
linux-stable, which, btw, I think for the first time in ages,
everybody here can agree on. That's a good feeling.

On Sun, Apr 26, 2020 at 11:03 PM Sebastian Moeller <[email-masked]> wrote:
>>>>  1.    "ECT(1) traverses the Internet well, due to foresight in RFC-3168.."
>>>> ________________________________
>>>> ________________________________

-- 
Make Music, Not War

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-435-0729

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 27 Apr 2020 13:27:52 +0300

> What? I didn't say anything about fragments or reassembly.

You were calling out one of my statements which specifically talked *only* about RFC-3168.  That means you were either talking about the fragment reassembly behaviour, *or* you were deliberately using a reference to RFC-3168 to talk about RFC-6040 instead.  I've already given you my response as to RFC-6040.

We're done here.

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 27 Apr 2020 12:12:00 +0100

Dave,

> they took "echo" seriously way back then.

[BB] I knew about ping turning round the ToS field - that was introduced 
very early on... but I hadn't thought of using it to see what happens in 
a tunnel. We've always used tracebox, but that relies on the ICMP being 
returned. Duh! Using ping -Q is obvious now you point it out.

> into 3 at the egress point (inner), it stays at 1

[BB] What's your tunnel egress in this case?

>
> with what? 20%? 60%? of the world transiting tunnels?

[BB] As said last night in tsvwg thread with Jake H "Position Statement 
on ECT(1)" - running in parallel to this, my little calcu-guesti-lation 
reckons 60-70% of paths are tunnelled. Mobile is >50% of the Internet, 
and that's all tunnelled.

> this is a major
There certainly will be weirdness like you've just found above.
But in general:
* L4S relies on the tunnels in the original RFC3168 ECN spec (year 2001).
* SCE relies on the tunnels in RFC6040 (year 2010)

Also I'm sure a lot more implementers noticed RFC3168 than RFC6040.

So I think RFC3168 is widely implemented, and it would have taken extra 
effort to implement it wrongly like you've just found, so I can't see 
that would be common.

Bob

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Mon, 27 Apr 2020 05:39:47 -0700 (PDT)

Bob,
	Short inline question on numbers.  And a clarification request.
	Marked [RWG]
Regards,
Rod

> and that's all tunnelled.

 [RWG] I am having a hard time with some of your calcu-guesti-lation numbers,
especially the "Mobile is >50%."  Exactly what is the yard stick here,
is that 50% of the end points, 50% of the traffic (seriously doubt that),
50% of the 5-tupple sessions?

Can you quantify this guesstimation with further metrics.

> * SCE relies on the tunnels in RFC6040 (year 2010)

 [RWG] This kinda implies L4S is not RFC6040 compatible, whish I know you
did not mean.  Just pointing out how someone may interrpret what
is written above.

> effort to implement it wrongly like you've just found, so I can't see 
 [RWG]                               ^ missing word what?

> >> Dave T?ht
> > Dave T?ht

-- 
Rod Grimes                                                 [email-masked]

---

From: "Tilmans, Olivier (Nokia - BE/Antwerp)" <[email-masked]>
Date: Mon, 27 Apr 2020 13:45:20 +0000

> Can you quantify this guesstimation with further metrics.

Endpoints. Note that some countries/region (e.g., EMEA) tend to nearly not have 
consumer-facing fixed infrastructure, and jumped to mobile first.

With a pinch of salt, here is one of the various reports you can find on that
subject:
https://www.cisco.com/c/en/us/solutions/collateral/executive-perspectives/annual-internet-report/white-paper-c11-741490.html
Segmenting on region/user-type/device type can be useful:
https://www.cisco.com/c/en/us/solutions/executive-perspectives/annual-internet-report/air-highlights.html
[methodology] https://www.cisco.com/c/en/us/solutions/collateral/executive-perspectives/annual-internet-report/white-paper-c11-741490.html#Appendices

 > did not mean.  Just pointing out how someone may interrpret what

[re-reading this, I think I misread your statement. The short answer should
have been that RFC6040 updating RFC3168 has a precise IETF meaning, notably
on not making existing implementation non-compliant. I'll leave the, now less
relevant text for posterity]

L4S does not need RFC6040 as it works fine with pure RFC3168 encap/decap specs:
ECT(1) at ingress is pushed to outer header, tunneled (and maybe switched to CE),
and survives decapsulation (or is overwritten by CE). RFC6040 only introduces a
second decapsulation-overwrite behavior: ECT(1) on the outer header should
Overwrite ECT(0) on the inner header--which is a case that cannot happen with
L4S as the inner packet comes from the sender, i.e., is marked ECT(1) by definition.

SCE on the other hand requires RFC6040 to be able to insert ECT(1) marks mid-tunnel,
as it otherwise gets stripped by RFC3168 logic--which is unfortunately the default
behavior on Linux/FreeBSD.

You can test that today, using any linux kernel (or freebsd for that matter). That
is incidentally how I found out that while the comment in the Linux source hints
that they 'saw' RFC6040, they did not fully implement it, as I wanted to try the
various SCE AQMs on tunneled traffic--which does nothing, as the marks do not
survive decapsulation, so never causes a receiver to feed ESCE back.

I note that both L4S and SCE will benefit from the RFC6040-udpate-shim (yet to be
published, so definitely not deployed), as that one extends the wording to also
cover encapsulation with, e.g., UDP as shim layer in between the outer and inner
header, as that particular case is missing from the RFC3168 wording.

 >  [RWG]                               ^ missing word what?

[like in wireguard], as he was referring to Dave's ping test. Although checking
wireguard's source shows it only does strict RFC3168 decapsulation, so makes it
hard to understand Dave's results...
https://github.com/WireGuard/wireguard-linux/blob/1c63f82b824b2b75b9fee182cbd998ac80b19374/drivers/net/wireguard/receive.c#L388

Best,
Olivier

 > into comments, but the code doesn't actually do what the table says in the
 > > >>> https://github.com/apple/darwin-
 > xnu/blob/a449c6a3b8014d9406c2ddbdc81795da24aa7443/bsd/netinet/ip_ecn.c#L105
 > > >> Good to know.
 > > >>

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 27 Apr 2020 16:48:36 +0300

For the purposes of the meeting, I think it would be useful to summarise what the SCE team understands about SCE's interaction with tunnels and fragmentation thus far.

The ideal behaviour for us would be that any ECT(1) codepoint found on a fragment or a tunnel's outer header would be preserved upon reassembly and/or decapsulation, in the same way as but notwithstanding the similar existing requirement for the CE codepoint.  That is not, however, what is currently specified, so it is necessary for SCE to tolerate the non-ideal behaviours of existing tunnels.

SCE marks applied before or after a tunnel or a fragmented path are preserved by the current specifications.  CE marks applied anywhere on the path are also already preserved.  Since SCE transports respond normally to CE marks, effective congestion control is not compromised by non-ideal tunnel behaviour with respect to ECT(1), when SCE transports and/or middleboxes are deployed.  I don't think this is controversial.

There *is* a potential challenge in getting SCE marks applied to the outer header of a tunnel to the inner payload.  If SCE marks are erased completely, or heavily disrupted (ie. more than about 50% erased), then high fidelity congestion control will not be implemented, but effective congestion control will still be used.

Updates to RFC-3168 can resolve the fragment reassembly question.  Updates to RFC-6040 can resolve the decapsulation question.  Once specified, rollout of updated tunnel implementations will be incremental.  The benefits of supporting SCE may serve as incentives to do so.

There is also a potential challenge due to tunnels hiding the distinction between flows they carry.  If some flows in the same tunnel are SCE aware but others are not, then successfully applying SCE marks to that tunnel may cause the SCE flows to be disadvantaged in throughput.  Tunnels can assist with resolving this problem by providing information about the inner flow in an outer-visible location, such as the IPv6 Flow Label field.

It would be useful to have a good survey of the detailed behaviour of existing tunnel implementations, with a particular focus on the above challenges, and the likelihood that they will be updated to follow future specifications about tunnel behaviour.  So far I have not seen one.

 - Jonathan Morton