Subject: [Idr] Magnus Westerlund's Discuss on draft-ietf-idr-tunnel-encaps-20: (with DISCUSS)

From: Magnus Westerlund via Datatracker <[email-masked]>
Date: Thu, 03 Dec 2020 02:52:22 -0800

Magnus Westerlund has entered the following ballot position for
draft-ietf-idr-tunnel-encaps-20: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-idr-tunnel-encaps/

----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

So this is really to start a discussion of how the framework approach of this
document may not be explicit enough on what combination is actually viable
combination and have existing specification for how to deal with a number of
behaviors for tunnels. So my view after having read this one is that the
signalling is specified in a two tier fashion, with an outer encapsulation that
can be IPvX, IPvX/UDP for example and then a tunnel protocol like GRE, VXLAN,
L2TPv3. So I don't believe all combinations of outer encapsulation and tunnel
protocol is actually defined. This document does not provide a table with the
reference for where the actual data plan specification for combinations are
provided. I think it would be good if there actually existed such a table/list.

The next aspect of this discuss is the difficulty in determining if the
provided sub-TLVs are sufficient. I will mention a number of potential ones
that I wonder if they are necessary to configure these combinations.

To build on Erik Kline's discuss. So are for all these combinations when IP is
the outer encapsulation is the egress ECN behavior to correctly mark or drop
inner payload well defined. If the egress is not guaranteed to do the correct,
then I think a configuration option is required.

When using UDP encapsulation combined with IPv6 there is the question if one
can safely use zero-checksum. Per RFC 6935 and RFC 6936 some consideration is
needed to determine if the inner payload is safe to combine with zero checksum.
So this requires the combination of tunnel protocol and inner payload to
determine this. So I think some of these tunnel protocols have text on this,
but I don't know which combinations have this specified. And also here arise a
question if some of these will also need a configuration option as there exist
some inner payloads that could not be safe and thus a different tunnel with
checksum enabled may be required.

When using UDP encapsulation I am wondering over source port usage. To my
knowledge some of these protocols like VXLAN do defines that source ports are
picked randomly to ensure header hashing will provide different values for
different inner flows. So is there a need in any of this cases to identify a
single source port?

So I at least are unable to determine if this specification are containing all
necessary attributes when it doesn't have identification of what combinations
it expect to work, and what behavior on the above aspects is just working.

So lets start discussing what needs to be addressed here if any.

---

From: John Scudder <[email-masked]>
Date: Thu, 3 Dec 2020 21:38:44 +0000

Hi Magnus,

Thanks for your comment. My reply below.

> On Dec 3, 2020, at 5:52 AM, Magnus Westerlund via Datatracker <[email-masked]> wrote:
> 
> [External Email. Be cautious of content]
> 
> 
> Please refer to https://urldefense.com/v3/__https://www.ietf.org/iesg/statement/discuss-criteria.html__;!!NEt6yMaO-gk!R160ShLm3-ilXKD-spxJGlwrvP9vjZX5gkLJJWWbNNKe-2FiwsoZbD19wDsUXw$
> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-ietf-idr-tunnel-encaps/__;!!NEt6yMaO-gk!R160ShLm3-ilXKD-spxJGlwrvP9vjZX5gkLJJWWbNNKe-2FiwsoZbD2i6NtMNw$
> 
> 
> 

It’s not the goal of the spec to provide an all-encompassing evaluation or framework for tunneling as a data-plane technology. It would be a worthy effort to do that work, but I don’t think IDR is the right place to do it, nor do I think this document is the right document for it. The goal of the spec is restricted to signaling; more discussion below.

It’s also not the goal of the spec to support every conceivable combination in which you might construct a tunnel, it’s pragmatically oriented to things people actually wanted to ship and deploy. See also my responses to a few of the others, where I took the position that if other feature support is desired (for example, flow label), it’s up to those wanting that support to specify it.

> then I think a configuration option is required.

I’m afraid I don’t follow this point.

> different inner flows. So is there a need in any of this cases to identify a

None that’s been raised to date.

> it expect to work, and what behavior on the above aspects is just working.

I guess it depends on what you mean by “necessary”. The spec has been found sufficient for at least three implementations to be shipped, so it’s demonstrably solving a real-world problem. (See the IDR wiki implementation report.) It’s been found sufficient for a number of other specs, largely in BESS, to base their functions on. (See the shepherd report.) I’m certain that if you compared the functionality that’s been implemented and/or depended on by other specs, with the complete matrix of potential functionality, you would find the surface has barely been scratched yet, which I think is fine. In that sense, I think of this spec as a framework (the attribute itself, and the TLV/sub-TLV structure within it, and their generic processing rules) plus a number of applications within that framework (the sub-TLVs). In my opinion, it would be problematic if the framework were deficient, but there is no expectation on the part of the authors or WG that every possible application has been covered, and again, that’s fine.

I expect there will be a great many extensions specified in the future.

> So lets start discussing what needs to be addressed here if any.

Thanks,

—John

---

From: Magnus Westerlund <[email-masked]>
Date: Fri, 4 Dec 2020 10:43:36 +0000

Hi John,

Thanks for clarification. Let me try to explain how I interpreted the document.

So the document do talk in the introduction about a few specific tunneling
protocols. It then defines a number of tools. However, what I am missing is the
specification for protocol foo, which combinations that actually are excepted to
work and what sub-tlvs that are to be used. Basically individual requirements on
what needs to be supported for each protocol. 

The second part is really a question if there are need for additional
configuration of these specific tunnel and encapsulation combinatins when it
comes to ECN, IPv6/UDP zero-checksum, and source ports. I think all of these
questions are easier to answer in the context of particular configuration
combinations compared to the unspecified level. 

So what I propose should be done are the following.

For each tunnel protocol that the WG currently want to signal with this create a
subsection. In that subsection include the following.

 - Which encapsulations alternatives that can be supported, and point to the
data plane specifications for these combinations. 

 - Identify all sub-tlvs that are either required or optional to use with this
protocol to correctly configure any of the identified combinations. 

 - If IP is part of the encapsualtion. Determine if outer ECN processing is
mandatory in egress or not. If not mandatory, then I think you likely have a
need for a configuration. And in that case I don't quite understand if the
entity that creates these tunnel configurations will know the egress
capabilities and this can determine if the configuration is supported.

 - For encapsualtions using IPv6/UDP then zero-checksum considerations per that
data plane specification needs to be considered if it can be supported, and if
there is need for a configuration of that option. 

 - For encapsulation using UDP, the data plane specification does it define
source port usage. You appear to indicate that no such need has so far been
raised. 

For the last 3 bullets it might not be necessary to say anything in the
document. However, as the document is currently not well specified on which
combinations are possible, I can't determine if there are a need for these three
bullets or not. I would suspect that current practice might make this clear more
than necessary our combination of specifications saying this specifically. 

My impression for example when it comes to zero-checksum for IPv6/UDP is used
widely with little consideration about the potential issues. However, this work
is done in a WG for inter domain routing. Thus, the general context is inter
domain routing, and that pushes the zero-checkum usage from within a single
domain into multiple ones. Thus, the risk for errors that checksum would detect
is different and also the potential impact of failure to verify is different.
Thus, there might actually exist a configuraiton need for this if one intended
to pay attention to what might be written into the user plane specification.
Here actually we do have an issue with RFC 7510:

So your document states: 

   MPLS-in-UDP [RFC7510] is also supported, but an
   Encapsulation sub-TLV for it is not needed since there are no
   additional parameters to be signaled.

While if you go read Section 3.1 of RFC 7510 it states the following in regards
to zero-checksum:

   The UDP checksum MUST be implemented and MUST be used in accordance
   with [RFC768] and [RFC2460] for MPLS-in-UDP traffic over IPv6 unless
   one or more of the following exceptions applies and the additional
   requirements stated below are complied with.  There are three
   exceptions that allow use of UDP zero-checksum mode for IPv6 with
   MPLS-in-UDP, subject to the additional requirements stated below in
   this section.  The three exceptions are:

   a.  Use of MPLS-in-UDP in networks under single administrative
       control (such as within a single operator's network) where it is
       known (perhaps through knowledge of equipment types and lower-
       layer checks) that packet corruption is exceptionally unlikely
       and where the operator is willing to take the risk of undetected
       packet corruption.

   b.  Use of MPLS-in-UDP in networks under single administrative
       control (such as within a single operator's network) where it is
       judged through observational measurements (perhaps of historic or
       current traffic flows that use a non-zero checksum) that the
       level of packet corruption is tolerably low and where the
       operator is willing to take the risk of undetected packet
       corruption.

   c.  Use of MPLS-in-UDP for traffic delivery for applications that are
       tolerant of misdelivered or corrupted packets (perhaps through
       higher-layer checksum, validation, and retransmission or
       transmission redundancy) where the operator is willing to rely on
       the applications using the tunnel to survive any corrupt packets.

   These exceptions may also be extended to the use of MPLS-in-UDP
   within a set of closely cooperating network administrations (such as
   network operators who have agreed to work together in order to
   jointly provide specific services).  Even when one of the above
   exceptions applies, use of UDP checksums may be appropriate when VPN
   services are provided over MPLS-in-UDP; see Section 6.

   As such, for IPv6, the UDP checksum for MPLS-in-UDP MUST be used as
   specified in [RFC768] and [RFC2460] for tunnels that span multiple
   networks whose network administrations do not cooperate closely, even
   if each non-cooperating network administration independently
   satisfies one or more of the exceptions for UDP zero-checksum mode
   usage with MPLS-in-UDP over IPv6.

So this indicate to me that there is a significant likelyhood that for the
intended usage of these tunnel encapsulations would not qualify for the
exceptions and should use checksums. If that needs signalling or not is the next
question. That comes back to how the configuring node knows about capabilities
and how one can resolve lack of a capability when signalled. 

Cheers

Magnus

On Thu, 2020-12-03 at 21:38 +0000, John Scudder wrote:
> > Please refer to 
> > https://protect2.fireeye.com/v1/url?k=82d10e2c-dd4a3435-82d14eb7-8682aaa22bc0-bdb272a0df47b7f6&q=1&e=99571de8-da16-449c-9dd9-87475a28842b&u=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2F%2Fwww.ietf.org%2Fiesg%2Fstatement%2Fdiscuss-criteria.html__%3B%21%21NEt6yMaO-gk%21R160ShLm3-ilXKD-spxJGlwrvP9vjZX5gkLJJWWbNNKe-2FiwsoZbD19wDsUXw%24
https://protect2.fireeye.com/v1/url?k=9454b6c6-cbcf8cdf-9454f65d-8682aaa22bc0-156a49648ff7e6bf&q=1&e=99571de8-da16-449c-9dd9-87475a28842b&u=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-idr-tunnel-encaps%2F__%3B%21%21NEt6yMaO-gk%21R160ShLm3-ilXKD-spxJGlwrvP9vjZX5gkLJJWWbNNKe-2FiwsoZbD2i6NtMNw%24
> It’s not the goal of the spec to provide an all-encompassing evaluation or
> framework for tunneling as a data-plane technology. It would be a worthy
> effort to do that work, but I don’t think IDR is the right place to do it, nor
> do I think this document is the right document for it. The goal of the spec is
> restricted to signaling; more discussion below.
> 
> It’s also not the goal of the spec to support every conceivable combination in
> which you might construct a tunnel, it’s pragmatically oriented to things
> people actually wanted to ship and deploy. See also my responses to a few of
> the others, where I took the position that if other feature support is desired
> (for example, flow label), it’s up to those wanting that support to specify
> it.
> 
> I guess it depends on what you mean by “necessary”. The spec has been found
> sufficient for at least three implementations to be shipped, so it’s
> demonstrably solving a real-world problem. (See the IDR wiki implementation
> report.) It’s been found sufficient for a number of other specs, largely in
> BESS, to base their functions on. (See the shepherd report.) I’m certain that
> if you compared the functionality that’s been implemented and/or depended on
> by other specs, with the complete matrix of potential functionality, you would
> find the surface has barely been scratched yet, which I think is fine. In that
> sense, I think of this spec as a framework (the attribute itself, and the
> TLV/sub-TLV structure within it, and their generic processing rules) plus a
> number of applications within that framework (the sub-TLVs). In my opinion, it
> would be problematic if the framework were deficient, but there is no
> expectation on the part of the authors or WG that every possible application
> has been covered, and again, that’s fine.
>

---

From: John Scudder <[email-masked]>
Date: Fri, 4 Dec 2020 20:38:39 +0000

Hi Magnus,

For now I’d like to talk just about your zero-checksum point (quoted below but I’ll top-post my comments). It seems to me that the expected deployment model for tunnel-encaps, detailed in Section 11 and further strengthened after my discussion with Roman, exactly corresponds with "single administrative control” or "closely cooperating network administrations”. Here’s the Section 11 text from the current candidate version 21, after the modification:

   The Tunnel Encapsulation attribute is defined as a transitive
   attribute, so that it may be passed along by BGP speakers that do not
   recognize it.  However the Tunnel Encapsulation attribute MUST be
   used only within a well-defined scope, for example, within a set of
   Autonomous Systems that belong to a single administrative entity.

Whether it would be nice to add a “should we use UDP checksums or not” sub-TLV anyway, as an additional bell and whistle, is up for discussion. But I don’t think it’s mandated by RFC 7510.

Thanks,

—John

> On Dec 4, 2020, at 5:43 AM, Magnus Westerlund <[email-masked]> wrote:
>

---

From: Magnus Westerlund <[email-masked]>
Date: Mon, 7 Dec 2020 13:38:33 +0000

On Fri, 2020-12-04 at 20:38 +0000, John Scudder wrote:
> Hi Magnus,
> 
> For now I’d like to talk just about your zero-checksum point (quoted below but
> I’ll top-post my comments). It seems to me that the expected deployment model
> for tunnel-encaps, detailed in Section 11 and further strengthened after my
> discussion with Roman, exactly corresponds with "single administrative
> control” or "closely cooperating network administrations”. Here’s the Section
> 11 text from the current candidate version 21, after the modification:
> 
> Whether it would be nice to add a “should we use UDP checksums or not” sub-TLV 
> anyway, as an additional bell and whistle, is up for discussion. But I don’t
> think it’s mandated by RFC 7510.

The above text clarifies the scope of this document. I think there are ways of
avoiding a TLV for RFC 7510 usages, however, I think that this is in fact
through documenting the assumption in regards to this protocol usage. 

My point is that this document do need per tunnel protocol sections where these
aspect can be considered in this context of this tunnel configuration mechanism.
I am a bit uncertain in my understanding about how this BGP attribute is used.
However, it looks to me that a decapsulator will basically tell those other
entities in the other AS that it cooperates with that you can establish a tunnel
to the decapsulator in relation to a set of routes. However, if that is the
correct understanding then a number of assumptions are made. I think in the
context of RFC7510 you are assuming that zero-checksum is either MAY or MUST be
used for this to work. If it is the later then you also put a requirement on the
receiver of these attributes to not establish a tunnel if it thinks the
requirements in RFC7510 for zero-checksum is not met. 

To conclude I don't think you get away from discussing these aspects in the
document. However, it can be so that there exist no need for defining additional
sub-tlvs for the aspects I brought up. But, I think the document needs to go in
either of two directions here. The first is to be specific on combinations of
tunnel and outer encapsualtion and document the assumptions about capabilities
and requirements on the network deployment to use these zero-checksum as well as
ECN field processing. The other direction is to make this into a framework, but
then you are going to have to discuss these matters in the abstract and likely
need to have signalling indicate what the egress will do in regards. 

From my perspective not discussing zero-checksum and ECN behavior leaves
implementors unaware of relevant implementation requirements. And if a missmatch
occurrs those who sees the downside is likely not the nodes that causes the
issue. For the wrong ECN behaviors this has just casued a change in end-to-end
network flow behaviors as CE marks might be dropped in the egress. For zero-
checksum tunnels usage where the underlying network is corrupting enough packets
these affects those endpoints that receives corrupt packets. 

Cheers

Magnus

---

From: John Scudder <[email-masked]>
Date: Mon, 7 Dec 2020 14:37:43 +0000

One brief clarification pending a more comprehensive reply —

> On Dec 7, 2020, at 8:38 AM, Magnus Westerlund <[email-masked]> wrote:
> 
> to the decapsulator in relation to a set of routes.

You say “in the other AS” but BGP is also used within an AS (“IBGP”). I expect a common deployment scenario will be for tunneling within a single AS. Of course you are right that it also can and will be used in multi-AS scenarios (where the ASes are under a common administration or collaborating, as we’ve already discussed). 

More later. 

—John