Subject: [tsvwg] IPsec tunnels ( was RE: Position Statement on ECT(1) )

From: "Black, David" <[email-masked]>
Date: Mon, 27 Apr 2020 13:39:27 +0000

[posting as an individual, not WG chair]

Going back to RFC 4301 (IPsec architecture), it was written against the RFCs at the time for Diffserv (e.g., RFCs 2474, 2475, 2983) and ECN (RFC 3168).  The ECN provisions of IPsec tunnels in RFC 4301 were updated by RFC 6040 to add outer->inner ECT(1) propagation at tunnel egress.  Diffserv interaction with tunnels is somewhat complicated – see RFC 2983.  For IPsec in particular, there’s a good chance that the outer header is in a different Diffserv domain and/or security domain than the inner header.  Clearing the outer DSCP to 0 on encapsulation is the only simple thing to do that is assured to be correct (or at least not cause additional problems) in all possible cases.   From a Diffserv perspective, the tunnel behaves like a link wrt the inner Diffserv domain, but this does complicate using the DSCP to signal alternate ECN semantics (see RFC 4774).

I’m reminded of Ross Callon’s farewell remarks some number of IETFs ago, to the effect one obvious place where the IETF has done poorly on Internet architecture is tunnels and encapsulations – both the plethora of them and inconsistencies among them.

Thanks, --David

From: tsvwg <[email-masked]> On Behalf Of Bob Briscoe
Sent: Monday, April 27, 2020 7:54 AM
To: Holland, Jake; tsvwg IETF list
Subject: Re: [tsvwg] Position Statement on ECT(1)

[EXTERNAL EMAIL]
Jake,

Important omission: I've been focusing on tunnel decap, but there's a big problem with using Diffserv at encap...

I've just been told about traversal with a typical IPSec tunnel product (wireguard). IPSec tunnels copy ECN, but not Diffserv, to the outer on encap. As it says here https://www.wireguard.com/protocol/#diffserv-considerations:

"All transport data packets have a DSCP value of 0, because the DSCP value of the inner packet is never copied to the outer packet, so that we don't leak information about the data inside the encrypted inner packet. However, we do copy the ECN bits to and from the inner packets, in accordance with the logic described in RFC6040." That logic was carried over from the IPSec architecture [RFC4301].

When the IETF security area prepared the new IPSec architecture they wanted as few knobs as possible. So, rather than add knobs for allowing or blocking mutable fields that could be used as covert channels, they decided that the 2-bit ECN field was a "manageable risk", whereas the 6-bit DSCP was not. This was a big thing in the Security Area (saag) at the time, and RFC6040 was well-received, because it normalized all tunnels to the approach taken by IPSec.

This particular person was arguing with me for your approach until they discovered it wouldn't get into IPSec tunnels. Now they're in a quandry.

Bob
On 27/04/2020 05:16, Bob Briscoe wrote:
Jake,

On 27/04/2020 01:07, Holland, Jake wrote:

On 4/25/20, 6:47 PM, "Bob Briscoe" <[email-masked]><mailto:[email-masked]> wrote:

Thank you and double thank you for this.
My pleasure, and I'm glad to hear it's something you can engage with :)

I'm going to trim to just what I think are the parts I'm responding to,
but if anybody wants to follow the full context in case something
important is left out, I thought I'd include a link to your message for
easier reference:
https://mailarchive.ietf.org/arch/msg/tsvwg/ShTE1QTw81cDLqGfB_6-xTzmFYE/

[BB] There is nothing in RFC3168 that says an ECN AQM MUST act on ECT(1)
if it acts on ECT(0). So anyone can implement or deploy an RFC3168 AQM
that solely acts on ECT(0).
A fair point for future deployments.  But I note that many people have
for example linux boxes capable of forwarding at line rate for useful
links, which have tc-pie readily available.  It's not so clear that such
evaluations might not already be ongoing, given the increased endpoint
use of ECN in recent years, and I would prefer not to tell these folks
that they'll have to wait for new updates before rolling something out
safely.

Treating ECT(1) as if it's Not-ECT is effectively changing a classifier. I know any small change is a major thing. However, cutting the scope of a classifier down from 2 codepoints to 1 is hardly rocket science. And implementers have as long to do this as it will take for the whole of L4S to be productionized.

[BB] Yes, there are typical cases over 10:1 (16:1 is the worst).
However, the "outliers at 100:1" are not what you think, from the sound
of it. A 1-%ile value does not mean 1% of long-running flows ran with
that rate when averaged over their duration.
Thanks for the clarification.

5. If DSCP is used as a classifier, ECT(1) is no longer necessary as a
classifier and can be used as a 1/p congestion signal.
[BB] The piece missing here is how to transition from experiment to full
deployment if it succeeds.
If we used ECT(1) as the output, we would have to use DSCP for ever as
the classifier.
Either that or be open to changing the signaling after more operational
experience.  But yes, the overall point is well taken that DSCP is not
a great way to encourage wide deployment, nor a great way to establish
useful stable signaling, only an incremental step that might allow for
some interoperable production use for further examination.

My point is that this is better than committing to something that seems
like it's likely to have operational problems.

There is the possibility that operators are tempted to wipe the ECN
field, but I think that will remain taboo given its role in e2e
congestion control.
Yes, this is part of my worry.  If the way things work gets fragmented
in a way that ends up causing operational problems, we could easily see
ECN go the way of DSCP instead of the opposite.

[BB] You might have seen an idea I posted on this list where the last bit of the L4S ECN field could be included in integrity coverage. Probably impractical to change integrity coverage, but something to mull on.

Also, L4S low latency is orthogonal to Diffserv bandwidth prioritization
(see draft-briscoe-tsvwg-l4s-diffserv).
Also a fair point.  Again, my only response is that to me it seems better
than the alternative, though not ideal.

There's also the problem that RFC3168 and RFC4301 (IPSec) tunnels and
overlays don't propagate ECT(1) if it changed from ECT(0) since it was
encapsulated. Neither do a whole plethora of shim encapsulations (e.g.
all the v6 over v4 ones, GRE, GTP, etc). All these overlays will be
gradually updating to RFC6040, but these things take decades to change.
It might be helpful to have some visibility on how this stands now, and
what kinds of tunnels are in wide use.  I'm not really clear on how big
a problem this is, and at present I assume most just black it out or
something, but really I have no idea.

[BB] We do know that >50% of the Internet is now mobile, which is all compressed, encrypted, and multiply encapsulated. Mostly with GTP.

It feels like many tsvwg folks don't have a good feel for just how much encapsulation there is in the Internet, which might be why this issue tends to always be glossed over. I can't properly quantify the degree of encap in the Internet I'm afraid, but I can describe some of it beyond mobile already covered above.

For instance, in ex public telephone and telegraph companies (mostly DSL and PON), there's a significant amount of L2TP, at least in Europe. When I left BT about 5 years ago, the proportion was approaching 50% of lines encapsulated with L2TP. This is how "local loop unbundling" is achieved - the incumbent operates the lower layer as a wholesaler and the networks of the "unbundled" retail ISPs are encapsulated as overlays. Not all unbundled ISPs are encapsulated - only the bigger ones who want to operate more independently of the underlying wholesaler.

Don't know about satellite.

I understand DOCSIS Cable is pretty free from any encapsulation /deployed by the operator/.

But then, as well as the encapsulations deployed by the operator, there are overlays, often from host to host, or host to proxy. For instance for v6 over v4.

Similarly, there are all the different VPN encapsulations. Not just end-user VPNs, but all the overlay technologies used in and across data centres. Not necessarily encrypted VPNs, but at least routing overlays. Remember there are low latency deployment scenarios in DCs as well.

So, there must be no less than 60-70% of the internet conducted over tunnels or encapsulations.

I'm roughly assuming that the
tunneling deployments are more upgradeable than the queues,

I would have said there's no difference. Most tunnels will start or end on a node that's there for other reasons, not specifically to do tunnelling. But there are obviously tunnelling-specific boxes.

The major factor though is that the tunnel endpoints will often be in other networks, or operated by some entity different from the one who operates the bottleneck. So, the folk who operate the queue will have no business relationship with those you would get a tunnel upgraded.

but that
as an endpoint we'll be dealing with weird inconsistencies that basically
never fully understood anything beyond "don't lose CE marks" and maybe
"loss is acceptable in confusing cases", if we're lucky.

[BB] See reply to Dave Taht just now, which pretty much confirms what you've just said.

All the above reasons are why L4S avoids anything that requires changes to encapsulations.

I'd be interested to hear more about implementation and deployment status
on this, but it's a side question, in that it doesn't make the alternate
L4S CE semantics safely deployable.  If the tunnels can't be made
compatible with a safe answer to the signaling, it's not clear how they
can be made compatible with a safe deployment of LL.

[BB] This is a judgement call. I don't share your desire for perfection on the fairness front. Personally, I think 1:10 is fine as long as it approaches closer to 1:1 for low rate links. If it's occasionally 1:16 on high rate links, that wouldn't bother me.

But I've been prepared to try to solve this problem for those like you that do care. However, it does seem that no matter how well I do it, I'll just be told "it might not work". So I'm beginning to wonder why I'm even trying to shoot for ever-moving goal posts.

While typing this I just noticed you mentioned mobile networks universally
using tunnels, which is an interesting point.  But I thought many that
moved to IPv6 weren't using tunnels for the IPv6 portion?

AFAIK, v6 is no different. The tunnelling is for mobility and for encryption.

I guess I'm
interested in the scope of the tunnel deployments over shared bottlenecks
today, but I'm not sure how much it matters for this question.

[BB] My point above in a nut-shell - most transport layer people just aren't really aware of all the encapsulation they are sitting over.

Tunnel deployment is hugely important.

[BB] For the avoidance of doubt, pls can you define the term
aggregation-point bottleneck - I assume you're talking about a BNG or a
CMTS, etc. And perhaps you're also including a home gateway in the
upstream direction. But in my experience 'aggregation point' is usually
used for a switch that is not a bottleneck, so I'm just checking.
Yes, I'm intending to capture devices like CMTS, BNG, ONT, and similar
access points with many clients to the immediate south in the network and
typically bw-throttled according to the price they're paying.

I'd be interested to know if this and the other points I've made,
particularly those about the 1%iles, have altered the overall balance of
your thinking.
Not really, sorry.  I think the main thrust of the point still stands,
though there's still some reading to do to make sure I have fully grasped
everything.

So at the moment I'm in roughly the same spot.  I recognize that my
position would not be a way to achieve a rapid rollout, but to me it
looks like one of the better paths forward, because the downside of
moving along with ambiguous signaling and having it go poorly in
deployment seems even worse, and not an unlikely outcome.

[BB] The other point we haven't really talked about is TCP. Without precise feedback, you can't get close to ultra-low delay. And there's not enough space in TCP to feed back two outputs without waiting for years for an option to become traversible through middleboxes.

But I'm going to stop there.

Regards

Bob

Best regards,
Jake

--

________________________________________________________________

Bob Briscoe                               http://bobbriscoe.net/

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 27 Apr 2020 15:09:05 +0100

Jake, David,

I've pasted an email below Suresh Krishnan has said I can contribute on 
his behalf. On typical IPSec behaviour with DSCP

I've also posted a 2-slide ecn-tunnelling briefing, that might help 
people get up to speed quickly:
http://bobbriscoe.net/presents/2004ietf/2020-04ecn-tunnel-brief.pdf

Bob

-------- Forwarded Message --------
Subject: 	Re: (Urgent) advice on typical IPSec practice with DSCP
Date: 	Mon, 27 Apr 2020 13:37:16 +0000
From: 	Suresh Krishnan <[email-masked]>
To: 	Bob Briscoe <[email-masked]>

Hi Bob,
   I tried to start going over the thread but I don’t think I will go 
through it in time in time to form an informed opinion. So I will give 
you my opinion on what the tunnel ingress would do. RFC 4301 requires[1] 
the tunnel ingress to copy over the DSCP field from the payload (inner) 
packet to the encapsulating (outer) packet. It also requires copying 
over the ECN fields (so CE will be copied as is).

[1] RFC4301 gives a break to the encapsulator to reset the DSCP values 
at some administrative boundary with the following text

"If the packet will immediately enter a domain for which the
  DSCP value in the outer header is not appropriate, that value
  MUST be mapped to an appropriate value for the domain”

This allows the encapsulator to rewrite the DSCP value at will, and 
setting it to 0 is pretty common. Note that this escape clause *DOES 
NOT* apply to the ECN field which has to be copied over as is.

On the decapsulator side, the CE field will be copied from the outer to 
the inner header if the inner header had ECT(0) or ECT(1) set. Hope this 
helps.

Thanks
Suresh

On 27/04/2020 14:39, Black, David wrote:
> Going back to RFC 4301 (IPsec architecture), it was written against 
> the RFCs at the time for Diffserv (e.g., RFCs 2474, 2475, 2983) and 
> ECN (RFC 3168).  The ECN provisions of IPsec tunnels in RFC 4301 were 
> updated by RFC 6040 to add outer->inner ECT(1) propagation at tunnel 
> egress.  Diffserv interaction with tunnels is somewhat complicated – 
> see RFC 2983.  For IPsec in particular, there’s a good chance that the 
> outer header is in a different Diffserv domain and/or security domain 
> than the inner header.  Clearing the outer DSCP to 0 on encapsulation 
> is the only simple thing to do that is assured to be correct (or at 
> least not cause additional problems) in all possible cases.   From a 
> Diffserv perspective, the tunnel behaves like a link wrt the inner 
> Diffserv domain, but this does complicate using the DSCP to signal 
> alternate ECN semantics (see RFC 4774).
>
> I’m reminded of Ross Callon’s farewell remarks some number of IETFs 
> ago, to the effect one obvious place where the IETF has done poorly on 
> Internet architecture is tunnels and encapsulations – both the 
> plethora of them and inconsistencies among them.
>
> Thanks, --David
>
> *From:*tsvwg <[email-masked]> *On Behalf Of *Bob Briscoe
> *Sent:* Monday, April 27, 2020 7:54 AM
> *To:* Holland, Jake; tsvwg IETF list
> *Subject:* Re: [tsvwg] Position Statement on ECT(1)
>
> [EXTERNAL EMAIL]
>
> Jake,
>
> Important omission: I've been focusing on tunnel decap, but there's a 
> big problem with using Diffserv at encap...
>
> I've just been told about traversal with a typical IPSec tunnel 
> product (wireguard). IPSec tunnels copy ECN, but not Diffserv, to the 
> outer on encap. As it says here 
> https://www.wireguard.com/protocol/#diffserv-considerations:
>
> "All transport data packets have a DSCP value of 0, because the DSCP 
> value of the inner packet is /never/ copied to the outer packet, so 
> that we don't leak information about the data inside the encrypted 
> inner packet. However, we do copy the ECN bits to and from the inner 
> packets, in accordance with the logic described in RFC6040." That 
> logic was carried over from the IPSec architecture [RFC4301].
>
> When the IETF security area prepared the new IPSec architecture they 
> wanted as few knobs as possible. So, rather than add knobs for 
> allowing or blocking mutable fields that could be used as covert 
> channels, they decided that the 2-bit ECN field was a "manageable 
> risk", whereas the 6-bit DSCP was not. This was a big thing in the 
> Security Area (saag) at the time, and RFC6040 was well-received, 
> because it normalized all tunnels to the approach taken by IPSec.
>
> This particular person was arguing with me for your approach until 
> they discovered it wouldn't get into IPSec tunnels. Now they're in a 
> quandry.
>
>
> Bob
>
>     Treating ECT(1) as if it's Not-ECT is effectively changing a
>     classifier. I know any small change is a major thing. However,
>     cutting the scope of a classifier down from 2 codepoints to 1 is
>     hardly rocket science. And implementers have as long to do this as
>     it will take for the whole of L4S to be productionized.
>
>
>
>
>
>     [BB] You might have seen an idea I posted on this list where the
>     last bit of the L4S ECN field could be included in integrity
>     coverage. Probably impractical to change integrity coverage, but
>     something to mull on.
>
>
>
>
>     [BB] We do know that >50% of the Internet is now mobile, which is
>     all compressed, encrypted, and multiply encapsulated. Mostly with
>     GTP.
>
>     It feels like many tsvwg folks don't have a good feel for just how
>     much encapsulation there is in the Internet, which might be why
>     this issue tends to always be glossed over. I can't properly
>     quantify the degree of encap in the Internet I'm afraid, but I can
>     describe some of it beyond mobile already covered above.
>
>     For instance, in ex public telephone and telegraph companies
>     (mostly DSL and PON), there's a significant amount of L2TP, at
>     least in Europe. When I left BT about 5 years ago, the proportion
>     was approaching 50% of lines encapsulated with L2TP. This is how
>     "local loop unbundling" is achieved - the incumbent operates the
>     lower layer as a wholesaler and the networks of the "unbundled"
>     retail ISPs are encapsulated as overlays. Not all unbundled ISPs
>     are encapsulated - only the bigger ones who want to operate more
>     independently of the underlying wholesaler.
>
>     But then, as well as the encapsulations deployed by the operator,
>     there are overlays, often from host to host, or host to proxy. For
>     instance for v6 over v4.
>
>     Similarly, there are all the different VPN encapsulations. Not
>     just end-user VPNs, but all the overlay technologies used in and
>     across data centres. Not necessarily encrypted VPNs, but at least
>     routing overlays. Remember there are low latency deployment
>     scenarios in DCs as well.
>
>     I would have said there's no difference. Most tunnels will start
>     or end on a node that's there for other reasons, not specifically
>     to do tunnelling. But there are obviously tunnelling-specific boxes.
>
>     The major factor though is that the tunnel endpoints will often be
>     in other networks, or operated by some entity different from the
>     one who operates the bottleneck. So, the folk who operate the
>     queue will have no business relationship with those you would get
>     a tunnel upgraded.
>
>
>
>         but that
>     [BB] This is a judgement call. I don't share your desire for
>     perfection on the fairness front. Personally, I think 1:10 is fine
>     as long as it approaches closer to 1:1 for low rate links. If it's
>     occasionally 1:16 on high rate links, that wouldn't bother me.
>
>     But I've been prepared to try to solve this problem for those like
>     you that do care. However, it does seem that no matter how well I
>     do it, I'll just be told "it might not work". So I'm beginning to
>     wonder why I'm even trying to shoot for ever-moving goal posts.
>
>
>
>     [BB] My point above in a nut-shell - most transport layer people
>     just aren't really aware of all the encapsulation they are sitting
>     over.
>
>     [BB] The other point we haven't really talked about is TCP.
>     Without precise feedback, you can't get close to ultra-low delay.
>     And there's not enough space in TCP to feed back two outputs
>     without waiting for years for an option to become traversible
>     through middleboxes.
>
> Bob Briscoehttp://bobbriscoe.net/

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 27 Apr 2020 15:20:50 +0100

Jake, David, [sorry I omitted to cc Suresh, pls reply to this one not 
the last!]

Bob

-------- Forwarded Message --------
Subject: 	Re: (Urgent) advice on typical IPSec practice with DSCP
Date: 	Mon, 27 Apr 2020 13:37:16 +0000
From: 	Suresh Krishnan <[email-masked]>
To: 	Bob Briscoe <[email-masked]>

Hi Bob,

Thanks
Suresh

---

From: Joseph Touch <[email-masked]>
Date: Mon, 27 Apr 2020 08:21:19 -0700

> On Apr 27, 2020, at 6:39 AM, Black, David <[email-masked]> wrote:
> 

I wonder how many years ago that was.

The IESG came to Mark and me to work on a tunnels draft in 2008 in Philadelphia, It took 2 years to gather even the initial information. It’s taken over a decade to have that doc start - only start - to gain momentum.

It’s not just the inconsistencies - it’s the errors. Errors in frag and reassembly, errors in signaling (as are being pointed out here), and errors in basic mechanism (e.g., IPsec putting keying before routing).

A significant challenge to that doc is what to do moving forward - there are a LOT of implications. And people keep creating new tunneling mechanisms and expecting that doc to track the changes (vs. trying to follow that doc’s recommendations). The good news is that things are starting to change - including here.

But for the IETF as a whole, it’s not just an uphill task, it’s Sisyphean.

Joe