Subject: [tsvwg] Prevalence of tunnels in mobile network (was RE: Position Statement on ECT(1))

From: "Tilmans, Olivier (Nokia - BE/Antwerp)" <[email-masked]>
Date: Mon, 27 Apr 2020 11:31:57 +0000

Hi Jake,

 > While typing this I just noticed you mentioned mobile networks universally
 > using tunnels, which is an interesting point.  But I thought many that
 > moved to IPv6 weren't using tunnels for the IPv6 portion?  I guess I'm
 > interested in the scope of the tunnel deployments over shared bottlenecks
 > today, but I'm not sure how much it matters for this question.

3GPP domains are literally a set of tunnels stitching together multiple NFs
and nodes/equipments.

It is true that the data exchanged with the end-users/UEs are (after popping
the 3GPP MAC headers--[SDAP/]RLC/PDCP) IP(v6) packets,
however these are massively encapsulated as soon as they enter the 3GPP domain,
usually with a <substrate>/IP[/UDP]/GTP+GTP-ext/{past-VNFs-encaps}/<user-data>[*].
See [1] for a rough sketch of the typical functions split that are
standardized--the 2nd column is the currently most popular approach to deploy 5G.
Note that the Distributed Unit (DU--a common location for queues to build up as
that is where the PHY effectively constraints what can be sent at any time) can
live in another AS than the Central Unit (CU), e.g., to support roaming, or
lease part of the infrastructure.
That is one of the reasons why the encryption of the user-data (PDCP) is done
quite early in the function chain.
You can find the protocol details from the MAC/interface (e.g., F1) specification
in the 38th Tech. spec. Series [2].

Given the prevalence of mobile networks as access technology[3], either through
smartphones or fixed-wireless offerings, tunnel traversal is a must if you want
more than an anecdotic deployment--and we're not even speaking here of the
challenges of doing flow classification (at scale) to enable any of the
proposed (l)fq/cnq/af schemes despite generally inaccessible user-payload due to
both the encapsulation depth (and short HW pipelines) and the ubiquitous encryption.

As Bob reported, despite RFC6040's age (i.e., 2010), it is still not implemented
properly in Linux (nor freebsd for that matter[4], but is on macOS [5]). When
surveying its tunnel implementation we found that most kernel-based[**] tunneling
protocols were doing RFC3168-style encap/decap, i.e., setting inner ECT as outer
ECT on encap, and only propagating CE to the inner header at decap (thus
stripping ECT(1) is set on the outer header and not the inner header).
User-space tunnels (e.g., OpenVPN) follow a similar trend (i.e., no RFC6040 support,
RFC3168 support at best).
I note that RFC6040 support is advertised by some vendors on newer physical
Appliances (but those claims are complicated to validate as we need ECT(1) to be
inserted in the middle of the tunnel and not at ingress to test this), and
unfortunately does not alleviate concerns for tunnel endpoints in clouds/VNFs (e.g.,
Linux host)

In summary, when ECN survives tunnels, it is still largely using the RFC3168
rules--i.e., ECT(1) set between tunnel endpoints is stripped away; ECT(1) set outside
the tunnel is preserved. This is the state of ECN, which has multiples documents
_specifying_ it; DSCP  traversal is a whole other can of worms (RFC2983 is
_informational_), and is basically to be handled using ad-hoc solutions--if we can
get rid of bleaching at AS boundaries as first deployment hurdle.

I understand the safety concerns with the overloading of CE that L4S provides (and
hear the calls for means to contain the experiment, which are fair and must be
addressed), but at this stage, SCE changes an already challenging deployment story
from upgrading 3-parts (endhosts + bottleneck), to 3 parts + all tunnel
endpoints--and that is solely to get a signal out of the network. It has yet to
address the main challenge of delivering benefits in environment without "L4-flow"
identifiers (others than what RFC3168 provides without needing an extra codepoint).

Given the pressure/hype on low-latency communications in the mobile world and
elsewhere, time is of the essence, and no-one will be happy if each SDO ends up
developing its own flavor of LL-capable transport.

Best,
Olivier

---
[*] For completeness, GTP is replaced by (e)CPRI for the fronthaul (FH)
connection.
[**] Any tunnel using the ip_tunnel facilities (IPIP, GRE, SRv6, ...), and/or the
  INET_ECN_decapsulate() function directly (e.g., geneve); but also those that
  reimplemented their own handler (e.g., wireguard). I may obviously have missed some
  protocols, but this seems to be a fairly large list already. FOU/GUE are notable
  exceptions where no propagation at all happens from outer to inner header.

[1] https://miro.medium.com/max/1400/1*MDG4ArWCpOsKySeavtvfvA.png (or the original
  post https://medium.com/5g-nr/cloud-ran-and-ecpri-fronthaul-in-5g-networks-a1f63d13df67 )
[2] https://www.3gpp.org/DynaReport/38-series.htm
[3] https://www.cisco.com/c/en/us/solutions/executive-perspectives/annual-internet-report/air-highlights.html
[4] https://github.com/freebsd/freebsd/blob/1d6e4247415d264485ee94b59fdbc12e0c566fd0/sys/netinet/ip_ecn.c#L136
[5] https://github.com/apple/darwin-xnu/blob/a449c6a3b8014d9406c2ddbdc81795da24aa7443/bsd/netinet/ip_ecn.c#L117