Subject: Re: [tsvwg] Comment on UDP options draft

From: "C. M. Heard" <[email-masked]>
Date: Sat, 2 Apr 2022 19:44:27 -0700

On Thu, Mar 31, 2022 at 6:14 AM Gorry Fairhurst wrote:

> On 30/03/2022 16:16, Tom Herbert wrote:
> > Sorry, I still don't see a clear answer to my question. Let me try one
> > more time: if a device terminating UDP does not have capability to
> > process protocol trailers, but otherwise has sufficient capabilities
> > to process protocol headers will it be able to support UDP options?
> > Please try to answer with a yes or no.
>
 [ ... ]

> I think the answer is "no".
>
> The design places the protocol control information in the area after the
> UDP Datagram. This was an early design decision, and it places the
> options by appending to an existing datagram, although if a device does
> not support this, the options data will not be available. That's the way
> it has been specified.
>

To expand upon that: an implementation that complies with the current
specification (draft-ietf-tsvwg-udp-options-18) is required to support OCS
and the eight "must-support" options either in the trailer of an
unfragmented or a reassembled UDP datagram or (subject to some caveats ---
cf. https://mailarchive.ietf.org/arch/msg/tsvwg/9ZGGJItKmd6GjtEU9vhGJQCCY6c/)
in the header area of a fragment (including an atomic fragment). A device
terminating UDP that does not have capability to do these things cannot
comply with the specification. But neither can such a device comply with
conventional UDP when the UDP checksum is non-zero -- it can't verify the
UDP checksum if it cannot access the data all the way to the end of the
datagram.

Earlier, on Tue, Mar 29, 2022 at 2:05 AM ,Tom Herbert wrote:

> [W]hat does that mean for most routers in deployment that only have a
> parsing buffer for processing headers and have no capability to process
> protocol trailers. Are they just out of luck for terminating UDP tunnels
> that might have UDP options?
>

As a reminder, RFC 6395 and RFC 6396 were published largely to address
concerns about tunnels in routers that don't have access to all of the data
in a UDP datagram. If that use case is seen as important for UDP options,
we have a way to address it: allow such tunnels to use atomic UDP fragments
with zero UDP checksum, zero OCS, and per-fragment options with no
per-datagram options. Granted, a tunnel endpoint limited in this way does
NOT comply with the requirements that the spec levies on a general-purpose
implementation, but I don't see why it would not work for the stated
purpose. Given that there would be a minimum of 22 bytes in the transport
header in addition to the options, this would not be especially efficient;
if that is pressing a concern, the use case in question might be better
addressed by GENEVE or GUE.

Mike Heard

---

From: Tom Herbert <[email-masked]>
Date: Sun, 3 Apr 2022 08:11:21 -0700

>> not support this, the options data will not be available. That's the way

Hi Mike,

> To expand upon that: an implementation that complies with the current specification

Current *draft* specification that is. AFAICT, consensus has not been
established that processing trailers is required to support UDP
options.

> (draft-ietf-tsvwg-udp-options-18) is required to support OCS and the eight "must-support" options either in the trailer of an unfragmented or a reassembled UDP datagram or (subject to some caveats --- cf. https://mailarchive.ietf.org/arch/msg/tsvwg/9ZGGJItKmd6GjtEU9vhGJQCCY6c/) in the header area of a fragment (including an atomic fragment). A device terminating UDP that does not have capability to do these things cannot comply with the specification. But neither can such a device comply with conventional UDP when the UDP checksum is non-zero -- it can't verify the UDP checksum if it cannot access the data all the way to the end of the datagram.

Computing checksum and processing protocol headers/trailers are
different mechanisms.  As I mentioned in the TSVWG presentation,
checksum computation for ingress and egress is performed by a hardware
mechanism that performs a running one's complement sum over the data.
Processing trailers requires loading specific bytes into a processor
of some sort to parse them. High performance devices have a parsing
buffer for headers (i.e. first N bytes of the packet) and will not
have a means to load and process trailers (other than possibly
deferring to a slow SW path which will kill performance). A device may
be able to compute payload checksums by no process trailers (I am
actually working on such a device right now).

>> [W]hat does that mean for most routers in deployment that only have a parsing buffer for processing headers and have no capability to process protocol trailers. Are they just out of luck for terminating UDP tunnels that might have UDP options?
>
>
> As a reminder, RFC 6395 and RFC 6396 were published largely to address concerns about tunnels in routers that don't have access to all of the data in a UDP datagram. If that use case is seen as important for UDP options, we have a way to address it: allow such tunnels to use atomic UDP fragments with zero UDP checksum, zero OCS, and per-fragment options with no per-datagram options.

Those limitations are unnecessary. UDP checksum is independent of UDP
options so that's already taken care of. IPv6 shows how to perform
fragmentation solely in packet headers-- protocol trailers are NOT
required for fragmentation.

> Granted, a tunnel endpoint limited in this way does NOT comply with the requirements that the spec levies on a general-purpose implementation, but I don't see why it would not work for the stated purpose. Given that there would be a minimum of 22 bytes in the transport header in addition to the options, this would not be especially efficient; if that is pressing a concern, the use case in question might be better addressed by GENEVE or GUE.

RFC6395 and RFC6396 are completely different situation. Those were
published long after RFC2460 only when UDP tunneling became popular
and router vendors wanted to support it. In that case, the specs were
adapted to the realities of implementation of a standard. But in the
case of UDP options, it seems like the proposal is ignoring a known
implementation reality for an entirely new protocol. I'm not sure why
this is so, there's no inherent reason for requiring trailers. For
instance, if I'm configuring a tunnel in a data center I already know
everything about the endpoints including whether they support UDP
options. Why can't I just configure UDP options to always be in
headers using the UDP length ==8 trick and a simple header format that
looks pretty much like TCP or IPV6 HBH options? If fragmentation is
needed, why not just have a fragment option where options before the
first fragment are per packet and options following the fragment
option in the first fragment apply to the reassembled packet? (again
like the way IPv6 does fragmentation).

In any case, if the consensus of the WG is that support for UDP
options with tunnels is not important, then I suggest changing the
name of the protocol to "UDP options for non-tunnels", and adding a
protocol type field in the UDP length==8 header format so that if
someone defines a protocol that could productively use the surplus
space with UDP tunnels it will be distinguishable from UDP options.

Tom

---

From: "[email-masked]" <[email-masked]>
Date: Sun, 3 Apr 2022 08:27:13 -0700

Hi Tom,

> On Apr 3, 2022, at 8:11 AM, Tom Herbert <[email-masked]> wrote:
> 
> ...AFAICT, consensus has not been

This has been WG consensus since the document was adopted by the WG - recall that consensus isn’t unanimous agreement.

> .... But in the
> this is so, there's no inherent reason for requiring trailers.

See Sec 4 of the draft. The *inherent* reason is to interoperate with legacy endpoints.

If we wanted to design a new protocol that legacy endpoints could simply ignore, we’d just use a new transport number. 

> For
> looks pretty much like TCP or IPV6 HBH options?

You can - you are welcome to configure both ends. The fact that the options occur as TLVs doesn’t;’t change the fact that YOU know exactly where they are and in what order. To YOUR endpoints, they are exactly the same as a fixed header.

> If fragmentation is

If YOU want to configure it that way, e.g., because YOU decide you don’t want or need trailing options, YOU are welcome to configure it that way.

But this is NOT a protocol for tunnels only, which is why we are not assuming a fixed field for anything that is not in EVERY packet. It is a single spec that works for both tunnels and endpoints, allowing both un-ignorable options and options that levy endpoints can safely ignore.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Sun, 3 Apr 2022 09:12:42 -0700

> This has been WG consensus since the document was adopted by the WG - recall that consensus isn’t unanimous agreement.
That's not correct. Adoption of a draft by a WG does not establish
consensus on the draft nor any parts of it, nor does it guarantee
publication of the draft and the drafts can not be referred to as
anything other than working draft. WG adoption means that the working
group has agreed to work on the draft. The chairs can correct me if
I'm wrong.

> If YOU want to configure it that way, e.g., because YOU decide you don’t want or need trailing options, YOU are welcome to configure it that way.

Great, then please clearly articulate this in the next draft with the
appropriate normative requirements. Trailers are NOT required for use
with UDP options. And also, I see no reason for any artificial
constraints like not being able to use fragmentation with trailers.

Tom

---

From: "[email-masked]" <[email-masked]>
Date: Sun, 3 Apr 2022 09:34:48 -0700

Tom,

> On Apr 3, 2022, at 9:12 AM, Tom Herbert <[email-masked]> wrote:
> 

I provided a timeline (since), not a reason (because).

>> ...
> appropriate normative requirements.

The;’re already in all the MAYs.

> Trailers are NOT required for use
> with UDP options.

They are for use with legacy endpoints - including legacy UDP tunnel endpoints.

> And also, I see no reason for any artificial

Or the artificial constraint that fragments use only per-fragment options.

Again, already there. These are configuration choices, not protocol constraints.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Sun, 3 Apr 2022 10:01:40 -0700

> Again, already there. These are configuration choices, not protocol constraints.
Joe,

In this same email thread you write "We have options in a *trailer*.
The only time this info is needed is when it is a *header*, which is
only in fragments.", you also wrote "a packet using only SAFE options
always has all the options after the data.". Sounds to me like you
think trailers are required for UDP options. To resolve this, can you
add a section in the draft describing how UDP options work in headers
when nodes don't support trailers and the necessary requirements. If
there are any limitations, those should also be articulated.

Tom

---

From: Gorry Fairhurst <[email-masked]>
Date: Sun, 3 Apr 2022 19:27:34 +0100

On 03/04/2022 19:01, Tom Herbert wrote:
> think trailers are required for UDP options.
Yes, that was exactly my understanding when we started this ID.
> To resolve this, can you
> when nodes don't support trailers and the necessary requirements.

I don't understand why we would add that.

To me, the work on UDP Options has been exclusively devoted to a design 
that use a trailer

after the UDP payload area. The only time this can appear in some

way like a "header" is when that payload area also has zero length, and the

specific case when a  carries a fragment (which is an option with a frag.)

therefore the fragment carried in the trailer of a null UDP datagram.

If nodes do not support this, then as far as I can see - they don't support

UDP Options (in practical cases a receiver will receive only "normal" 
UDP datagrams).

Have I missed something, Joe?

>   If

That is always useful, e.g. a few words explaining this to avoid further 
discussion

would be good.

Gorry

---

From: Joe Touch <[email-masked]>
Date: Sun, 3 Apr 2022 11:33:09 -0700

> always has all the options after the data.".

UDP options are always trailers.  In FRAGs, there is no data, but it still qualifies as a trailer.   

The are no magical properties between the two other than offset - and whether hardware vendors put a limit on that or not, there is no standard describing or endorsing implementing such a limit.

> Sounds to me like you

They are in all cases; they always come after the UDP data field. There is no way to change that in UDP.

When FRAG is used, the user data is hidden in a trailer option rather than the data field. But that is required *only* to support payloads larger than one IP packet OR options that modify the data. Use of UDP options, including for tunnels, is not limited to those conditions.

I.e., If a message fits in one IP packet and has no UNSAFE options, it MUST support options that come after a non null UDP data field.

> there are any limitations, those should also be articulated.

Endpoints(per RFC 1122) that support UDP options support UDP options.  Those that don’t are considered legacy endpoints. We already provide that definition.

Anyone is welcome to implement a subset of the spec that thy think serves their needs, but it would not comply with the spec as a whole.  I see no utility in trueing to do otherwise based on endpoints that have some set of indeterminate capabilities.

Joe

---

From: Joe Touch <[email-masked]>
Date: Sun, 3 Apr 2022 11:36:41 -0700

> On Apr 3, 2022, at 11:28 AM, Gorry Fairhurst <[email-masked]> wrote:
> 
> To me, the work on UDP Options has been exclusively devoted to a design that use a trailer after the UDP payload area. The only time this can appear in some
> UDP Options (in practical cases a receiver will receive only "normal" UDP datagrams).
> 

Nope.

> That is always useful, e.g. a few words explaining this to avoid further discussion
> would be good.

You’ve stated them above.  

In short, any endpoint that cannot implement the requirements of this spec cannot implement the requirements of this spec. It’s just that direct a tautology.

I do not see value in enumerating the ways that can happen, especially when those ways are not the result of compliance with some other spec.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Sun, 3 Apr 2022 11:58:41 -0700

> therefore the fragment carried in the trailer of a null UDP datagram.

Gorry,

Headers are at the beginning of a packet preceding the data, and
trailers are at the end of the packet following the data. I'm trying
to figure out if UDP options require support for that. So I'll try yet
one more time in trying to figure out the basic requirements and pose
the question to not rely on the definition of the term "trailer":

If a device only is capable of processing protocol within the first N
bytes of a packet (256 bytes for example), will it be able to support
and enjoy all of the features of UDP options including SAFE options,
UNSAFE options, fragmentation, per packet fragment options, and per
reassembled packet options? We can assume that both endpoints can be
arbitrarily configured as necessary, so this is really a question of
what the required wire format is.

Tom

---

From: "C. M. Heard" <[email-masked]>
Date: Sun, 3 Apr 2022 12:16:00 -0700

On Sun, Apr 3, 2022 at 8:11 AM Tom Herbert wrote:
> On Sat, Apr 2, 2022 at 7:44 PM C. M. Heard wrote:
> Current *draft* specification that is.

Agreed.

> AFAICT, consensus has not been established that processing
> trailers is required to support UDP options.

There has been no formal consensus call, but as Joe said:

On Sun, Apr 3, 2022 at 8:27 AM Joe Touch wrote:
> The *inherent* reason is to interoperate with legacy endpoints.

Indeed. One of the major promises of UDP options was, and still is,
an incrementally deployable substitute for IP fragmentation (and
IPv6 fragmentation in particular) for DNS and similar request/response
protocols. Eliminating trailer options altogether eliminates that
possibility. The problems with IP fragmentation are not so bad as they
were when the UDP options work began, but they still exist -- see
https://www.potaroo.net/presentations/2021-05-20-v6frag.pdf

If you are not challenging this point, please accept my apologies.

> > (draft-ietf-tsvwg-udp-options-18) is required to support OCS and
> > the eight "must-support" options either in the trailer of an
> > unfragmented or a reassembled UDP datagram or (subject to some
> > caveats) in the header area of a fragment (including an atomic
> > fragment). A device terminating UDP that does not have capability
> > to do these things cannot comply with the specification. But neither
> > can such a device comply with conventional UDP when the UDP checksum
> > is non-zero -- it can't verify the UDP checksum if it cannot access
> > the data all the way to the end of the datagram.
>

Understood. I posted an analysis of how this plays with UDP options at
https://mailarchive.ietf.org/arch/msg/tsvwg/9_1YWp-TApI9mcCuia8U6jfCwTA/
Please comment in that thread if you see anything that is not right.

> be able to compute payload checksums [but not] process trailers (I am

That last point -- "able to compute payload checksums but not process
trailers" -- was not clear to me from the previous messages. I had
assumed that the devices you were referring to were unable to process
trailers because they had **no access at all** to data deep in the
packet. Thank you for the clarification.

As Joe indicates below ...

On Sun, Apr 3, 2022 at 8:27 AM Joe Touch wrote:
> If YOU want to configure it that way, e.g., because YOU decide you
> don’t want or need trailing options, YOU are welcome to configure it
> that way.
>
> But this is NOT a protocol for tunnels only, which is why we are not
> assuming a fixed field for anything that is not in EVERY packet. It
> is a single spec that works for both tunnels and endpoints, allowing
> both un-ignorable options and options that levy endpoints can safely
> ignore.

... the on-the-wire format in the current draft CAN support tunnel
endpoints that are able to compute checksums over the entire packet
but that are not able to process options in trailers. And if I
correctly understand what I have read, the spec REQUIRES a compliant
implementation to provide an upper layer interface that allows the
application to specify that packets be sent with all UDP options
in the header (preceding frag payload data) and none in the trailer
(that is spelled out in Section 13).

If there is no fragmentation and reassembly in the tunnel, the wire
format (omitting IP and UDP headers) would look something like this:

                   +--------+--------+--------+--------+
                   |       OCS       |  NOP   |  NOP   |
                   +--------+--------+--------+--------+
                   | Kind=3 | Len=12 | Frag. Start=XX  |
                   +--------+--------+--------+--------+
                   |           Identification          |
                   +--------+--------+--------+--------+
                   |  Frag. Offset   | Dgram Opt Start |
                   +--------+--------+--------+--------+
                   ~              Options              ~
                   +--------+--------+--------+--------+
                   ~           Payload Data            ~
                   +--------+--------+--------+--------+

The Frag. Start field would point to the beginning of the payload data,
the Frag. Offset field would indicate that the payload data starts
immediately after the UDP header in the (conceptual) reassembled UDP
datagram, and Dgram Opt Start field would indicate the absence of an
options trailer -- that is, no trailing OCS and no trailing options --
by being set equal to the UDP Length of the (conceptual) reassembled UDP
datagram. If alignment of the Identification field is not wanted (it is
not used in atomic fragments), the two leading NOPs could be omitted.

Fragmentation and reassembly in a tunnel would be similar, except that
non-terminal fragments would have Len=10 and two trailing NOPs in place
of Dgram Opt Start. If I correctly understood the IETF 113 slides at
https://datatracker.ietf.org/meeting/113/materials/slides-113-tsvwg-71-checksum-offload-00
and the associated references, that would work well with GSO/GRO. If
that last assertion is incorrect, a detailed rebuttal is requested.
Regarding this suggestion:

> there are any limitations, those should also be articulated.

How much more does the spec really need? It does not seem to be a lot.
Maybe you could suggest specific edits (e.g., to Section 9.4, where
fragmentation and reassembly is described, and Section 13, where the
upper layer interface is specified) where you feel that clarification
of the technical requirements is needed, and then propose specific text
for a section on limited deployment scenarios such as environments that
contain devices that cannot support trailers (and thus that cannot
support all the requirements of the specification).

Another thing that I would urge you to do, if you warm to the idea that
the wire format in the present draft will work in environments that
contain devices that cannot support trailers, is to review the
critique of per-fragment options (and associated responses) at
https://mailarchive.ietf.org/arch/msg/tsvwg/9ZGGJItKmd6GjtEU9vhGJQCCY6c/
and comment as appropriate.

Mike Heard

---

From: Tom Herbert <[email-masked]>
Date: Sun, 3 Apr 2022 12:27:22 -0700

> format (omitting IP and UDP headers) would look something like this:
Mark,

If there's no fragmentation the fragmentation header is pointless. Why
not save 12 bytes and the complexity of processing the the frag header
and just format the packet like this where HLEN is the length of the
options (units of four bytes to preserve alignment of UDP data:

                     |       OCS       |      HLEN       |

If a packet is a fragment, how are per fragment options and per
reassembled packet options distinguished?

Tom

---

From: "[email-masked]" <[email-masked]>
Date: Sun, 3 Apr 2022 12:46:44 -0700

FWIW…

> On Apr 3, 2022, at 12:27 PM, Tom Herbert <[email-masked]> wrote:
> 
> Mark,
> 
> If there's no fragmentation the fragmentation header is pointless.
...
> reassembled packet options distinguished?

That *IS* the point of the fragmentation header.

Joe

---

From: "C. M. Heard" <[email-masked]>
Date: Sun, 3 Apr 2022 13:06:01 -0700

> options (units of four bytes to preserve alignment of UDP data:

What I see here is an argument that the wire format in the draft is not
optimized for the specific use of tunnels that cannot process trailers.
That is not the same as an argument that it won't work.

If options are needed and the additional 12 bytes from the UDP fragment
option make that much of a difference for the use case in question, my
sense is that the correct answer is to use something like GUE or GENEVE,
both of which are already implemented and deployed, rather than
compromise the utility of UDP options for its many other use cases.

> reassembled packet options distinguished?

The per-reassembled datagram options are treated as part of the payload
data of the constituent fragments. Any options that are visible as
separate options in a UDP fragment apply to that fragment alone.

IMO that is clearly spelled out in draft-ietf-tsvwg-udp-options-18.

Mike

---

From: Gorry Fairhurst <[email-masked]>
Date: Sun, 3 Apr 2022 21:18:03 +0100

> the question to not rely on the definition of the term "trailer":

For a UDP Datagram, the payload follows the UDP header, in that sense

it seems fair to me, to call data that follows this payload "a trailer".

> what the required wire format is.

My understanding is: Not really, because UDP Options is a proposal for 
extending a transport.

This is not a tunnel header - as in GRE, or other tunnel headers (of 
which there are several to choose) - it seeks to specify an addition to 
an endpoint-to-endpoint transport (i.e. UDP).

Of course, an endpoint can encapsulate protocols in transports to tunnel 
them. Now there is some text that might help in the specific case of an 
endpoint of UDP that has limited acccess to the data - and that's great 
if that is seen to add value that might be useful for a tunnel.

Gorry

---

From: Tom Herbert <[email-masked]>
Date: Sun, 3 Apr 2022 17:10:38 -0700

Mark,

It may work, but it's hardly necessary.

> compromise the utility of UDP options for its many other use cases.

I'm sorry, but I don't see defining a simpler header format in this
case in any way compromises the utility of the protocol for other use
cases. In fact, if someone were using UDP options just to carry UNSAFE
options without fragmentation, they would incur this unnecessary
twelve byte overhead also.

> separate options in a UDP fragment apply to that fragment alone.

Could you please provide an example header format that shows how UDP
options with fragmentation containing both per fragment options and
per reassembled packet options in the same packet, such that a device
that is limited to processing protocol at the beginning of the packet
can handle this.

Thanks,
Tom

---

From: "C. M. Heard" <[email-masked]>
Date: Sun, 3 Apr 2022 17:45:17 -0700

...
> that is limited to processing protocol at the beginning of the packet

The current draft of the UDP options spec does not support that, but I
didn't claim that it does. If you want to define a format that does that
while also meeting the other stated desiderata (specifically including
interoperation with legacy endpoints that do not support UDP options),
the onus is on you to do so. But please don't just show us a header
format that optimizes your tunnel use case, please also show is how it
works for the other use case as well.

Mike Heard

---

From: Tom Herbert <[email-masked]>
Date: Sun, 3 Apr 2022 18:56:19 -0700

On Sun, Apr 3, 2022 at 5:45 PM C. M. Heard <[email-masked]> wrote:
> didn't claim that it does.

Mark,

Thanks for the clarification. That seems to answer my question (finally).
Devices that can't process protocol trailers, i.e. protocol deep in the
packet, CANNOT support UDP options. Such devices are common in routers, so
it follows that UDP options with tunnels, where routers are often endpoints
of UDP tunnels, are not feasible. There may be other devices as well that
are in this category such as IoT or firewalls. This limitation in use cases
for UDP options needs to be clearly articulated in the draft especially
since UDP tunnels are now considered a major use case of UDP. Whether this
limitation in use case is necessary or acceptable should be a subject of
discussion in the WG. Note that this is NOT just a question of optimizing a
particular use case, it's a question of whether there are feasible means to
support the use case at all.

> If you want to define a format that does that

See https://www.ietf.org/archive/id/draft-herbert-udp-space-hdr-01.txt for
a common header format for UDP surplus space.

For encoding options with fragmentation I suggest a format like:

                   |       OCS       |      HdrLen     |
    ~       Per frag  Options           ~

                     |  Frag. Offset   |
                     ~        Per reasem  Options        ~

The per reassem options are in the first fragment after the fragment
header. In non-first fragments, it wouldn't be allowed to have options
after the fragment option (if there are options after the frag option they
would be ignored or treated as malformed UDP options).

Tom

---

From: "C. M. Heard" <[email-masked]>
Date: Sun, 3 Apr 2022 19:31:49 -0700

On Sun, Apr 3, 2022 at 6:56 PM Tom Herbert wrote:
> Thanks for the clarification. That seems to answer my question
> (finally). Devices that can't process protocol trailers, i.e.
> protocol deep in the packet, CANNOT support UDP options.

If you had said "cannot fully support" I would have stayed silent,
but the way you have phrased this is a mischaracterization that I won't
let pass. I stand by the assertion that the wire format illustrated in
https://mailarchive.ietf.org/arch/msg/tsvwg/83oCCwSLjM5nFbwXRTqrfXvF_zg/
allows such a device to support UDP options in a limited way. Sorry, but
IMO such rhetorical excesses really don't advance the discussion.

With that being said, I will re-read draft-herbert-udp-space-hdr-01 and
provide comments within the next few days.

Mike Heard

---

From: Tom Herbert <[email-masked]>
Date: Sun, 3 Apr 2022 20:06:41 -0700

Mark,

If I mischaracterized anything it's because the requirements in the draft
are unclear. If you want to advance the discussion then articulate the
requirements for this limited support of UDP options in the *draft*.

Tom

---

From: "[email-masked]" <[email-masked]>
Date: Sun, 3 Apr 2022 20:28:16 -0700

Tom,

> On Apr 3, 2022, at 6:56 PM, Tom Herbert <[email-masked]> wrote:
> ...
> See https://www.ietf.org/archive/id/draft-herbert-udp-space-hdr-01.txt <https://www.ietf.org/archive/id/draft-herbert-udp-space-hdr-01.txt> for a common header format for UDP surplus space.

We discussed this 2 years ago. 

> For encoding options with fragmentation I suggest a format like:
You previously argued strongly that you needed all the info for fragmentation up front, in a *fixed location relative to the UDP header*, so that hardware could find it without TLV parsing. Yet you now repeat an example that doesn’t support that requirement.

Note also that the HdrLen field above is not needed; it is redundant with the Frag start pointer.

The format above also limits the total of both per-fragment and per-packet options total less than the size of the min IP packet. The current format only limits that for per-fragment options (which is an inherent limitation).

> The per reassem options are in the first fragment after the fragment header. In non-first fragments, it wouldn't be allowed to have options after the fragment option (if there are options after the frag option they would be ignored or treated as malformed UDP options).

So again, even for non-initial fragments, you’re putting the frag header last, in a variable position, rather than first, where DMA could find it quickly - again, as you requested.

Joe

---

From: "[email-masked]" <[email-masked]>
Date: Sun, 3 Apr 2022 20:33:06 -0700

> On Apr 3, 2022, at 8:06 PM, Tom Herbert <[email-masked]> wrote:
> 
> If you want to advance the discussion then articulate the requirements for this limited support of UDP options in the *draft*.

The doc has no discussion of “limited support”. Custom transport designs for only endpoints that support new protocols can be more directly accomplished using a new transport code point. That is not the purpose of these options.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Sun, 3 Apr 2022 20:40:35 -0700

On Sun, Apr 3, 2022 at 8:33 PM [email-masked] <[email-masked]>
wrote:

> If you want to advance the discussion then articulate the requirements for
> this limited support of UDP options in the *draft*.
>
>
> The doc has no discussion of “limited support”. Custom transport designs
> for only endpoints that support new protocols can be more directly
> accomplished using a new transport code point. That is not the purpose of
> these options.
>

Then document the fact that UDP options will not work with most use cases
of UDP tunnels.

---

From: "[email-masked]" <[email-masked]>
Date: Sun, 3 Apr 2022 20:42:41 -0700

Hi, Mike,

> On Apr 3, 2022, at 7:31 PM, C. M. Heard <[email-masked]> wrote:
> 
> https://mailarchive.ietf.org/arch/msg/tsvwg/83oCCwSLjM5nFbwXRTqrfXvF_zg/ <https://mailarchive.ietf.org/arch/msg/tsvwg/83oCCwSLjM5nFbwXRTqrfXvF_zg/>
> allows such a device to support UDP options in a limited way.
…

Only if the endpoints happen to be configured in a limited way. Such an endpoint would NOT support UDP options.

There is no reason why a UDP tunnel endpoint should constrain itself to *requiring* UNSAFE options, e.g., when its packets don’t need to be either fragmented or per-fragment encrypted (or both).

And if those were not required, then the options MUST be in a trailer - and the location of that trailer CANNOT be constrained by artifacts of current implementation limitations, largely because we need to be able to support IP packets up to 576 for IPv4 and 1280 for IPv6.

—

And, as noted in my other post, attempts to push all the options “up front” to be more “header like” for the fragmentation case only serves to undermine the DMA efficiency of putting the frag option closer to the UDP header.

Joe

---

From: "[email-masked]" <[email-masked]>
Date: Sun, 3 Apr 2022 20:46:04 -0700

—
Dr. Joe Touch, temporal epistemologist
www.strayalpha.com

> On Apr 3, 2022, at 8:40 PM, Tom Herbert <[email-masked]> wrote:
> 
> 
> 
> On Sun, Apr 3, 2022 at 8:33 PM [email-masked] <mailto:[email-masked]> <[email-masked] <mailto:[email-masked]>> wrote:
> 
>> On Apr 3, 2022, at 8:06 PM, Tom Herbert <[email-masked] <mailto:[email-masked]>> wrote:
>> 
> Then document the fact that UDP options will not work with most use cases of UDP tunnels.

Right after you show me the documentation (preferably a standards-track RFC) to cite that defines the limit that routers enforce to make that happen.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Sun, 3 Apr 2022 20:59:49 -0700

On Sun, Apr 3, 2022 at 8:46 PM [email-masked] <[email-masked]>
wrote:

>> If you want to advance the discussion then articulate the requirements
>> for this limited support of UDP options in the *draft*.
>>
>>
> Right after you show me the documentation (preferably a standards-track
> RFC) to cite that defines the limit that routers enforce to make that
> happen.
>
> Joe,

Sorry, it's not my job to change how implementations work just suit how
your protocol works. If you want to try to convince them to start
supporting trailers after forty years of experience with protocol headers
you'll have to do that yourself.

In any case, I see no further need for discussion.

WG chairs: please note my objection with UDP options in thatt does not
support a common real world use case of UDP and that non-support is
undocumented.

Thanks,
Tom

Joe

---

From: Joe Touch <[email-masked]>
Date: Sun, 3 Apr 2022 21:05:32 -0700

I support the documentation of this implementation bug. 

> On Apr 3, 2022, at 9:00 PM, Tom Herbert <[email-masked]> wrote:
> 
> 
> 
> 
>> On Sun, Apr 3, 2022 at 8:46 PM [email-masked] <[email-masked]> wrote:
>> 
>> —
>>> On Sun, Apr 3, 2022 at 8:33 PM [email-masked] <[email-masked]> wrote:
>>>> 
> Sorry, it's not my job to change how implementations work just suit how your protocol works. If you want to try to convince them to start supporting trailers after forty years of experience with protocol headers you'll have to do that yourself.
> 
> WG chairs: please note my objection with UDP options in thatt does not support a common real world use case of UDP and that non-support is undocumented.
> 
> Thanks,
> Tom
> 
>> Joe
>>

---

From: "C. M. Heard" <[email-masked]>
Date: Sun, 3 Apr 2022 21:07:28 -0700

On Sun, Apr 3, 2022 at 8:42 PM Joe Tocu wrote:
> There is no reason why a UDP tunnel endpoint should constrain itself
> to *requiring* UNSAFE options, e.g., when its packets don’t need to
> be either fragmented or per-fragment encrypted (or both).

Begging your pardon, sir, but what became of "let the user decide?"

> And if those were not required, then the options MUST be in a trailer

Where does the specification (-18) say that? And if so, why?

Mike

---

From: Joe Touch <[email-masked]>
Date: Sun, 3 Apr 2022 21:18:28 -0700

> On Apr 3, 2022, at 9:08 PM, C. M. Heard <[email-masked]> wrote:
> 

Users decide what they want.  We are not constraining that decision. 

> Where does the specification (-18) say that? And if so, why?

Frag header is for actually fragmenting or to hide unsafe options. If neither occurs, the default packet structure that allows the valid UDP data to pass needs to be used - to support legacy receivers.

We cannot simultaneously support efficient DMA, buggy router hardware, and legacy receivers at the same time. Of these, the last are the most critical.

Joe

---

From: "C. M. Heard" <[email-masked]>
Date: Sun, 3 Apr 2022 21:36:02 -0700

On Sun, Apr 3, 2022 at 9:18 PM Joe Touch wrote:
>>> And if those were not required, then the options MUST be in a >>>
trailer
> Frag header is for actually fragmenting or to hide unsafe options.
> If neither occurs, the default packet structure that allows the
> valid UDP data to pass needs to be used - to support legacy receivers.

But -18 does not actually say that, either as a MUST or SHOULD.

> We cannot simultaneously support efficient DMA, buggy router
> hardware, and legacy receivers at the same time. Of these, the
> last are the most critical.

Agreed that legacy receivers and efficient implementation of
fragmentation are the key requirements. But if consenting endpoints
wish to do something that you or I think is stupid, who are we to
say NO unless they cause harm to others by their actions.

That being said, I do support limiting the requirements levied on
compliant implementations to things that are demonstrably useful.

In order to avoid rhetorical excesses on my own part, I think I had
better refrain from further discussion on this thread until after some
calm reflection.

Thanks

Mike

---

From: Joe Touch <[email-masked]>
Date: Sun, 3 Apr 2022 21:42:00 -0700

> On Apr 3, 2022, at 9:36 PM, C. M. Heard <[email-masked]> wrote:
> 
> But if consenting endpoints

I explained the harm - to legacy endpoints.

But even if we allow this, we have to allow the other ways too.  Again, we are not constraining the user. That means a compliant implementation has to allow either anyway. 

Joe

---

From: "[email-masked]" <[email-masked]>
Date: Mon, 4 Apr 2022 09:14:19 -0700

Hi, all,

I think there may already be a way forward here that allows users to pick from a variety of ways to support FRAGs that may satisfy the needs of constrained-implementations of router header processing.

The current draft defines the terminal FRAG option as follows:
                   +--------+--------+--------+--------+
                   | Kind=3 | Len=12 |   Frag. Start   |
                   +--------+--------+--------+--------+
                   |           Identification          |
                   +--------+--------+--------+--------+
                   |  Frag. Offset   | Dgram Opt Start |
                   +--------+--------+--------+--------+

                Figure 11   UDP terminal FRAG option format

We can simply allow Dgram Opt Start (DOS) to be anywhere, not just at the end of the user data. 

I.e.: if DOS is larger than FRAGSTART, then the per-packet options are at the end of the packet.
If DOS is smaller than FRAGSTART, then it indicates the end of the per-fragment options and start of per-packet options, all of which precede the frag data.

NOTE: if OCS points where the last IP packet end is, then there are no per-packet options and no per-packet OCS.
If OCS points before FRAGSTART, the same is true - the per-packet options pick up there, but there is no OCS. 

This way:
- FRAG typically occurs at the start of all FRAGs, including atomic fragments
	helping support DMA
- options CAN be configured to all occur before user data
	helping support limited router option processing hardware (pointing out that this was never a requirement and presents a limit to many protocols)

I think we should also add:

	the FRAG option SHOULD be avoided unless:
		the original datagram is split into multiple IP packets (fragmentation actually occurs)
		INCLUSIVE_OR
		UNSAFE options are used
		INCLUSIVE_OR
		users select that configuration to support limited header processing

Would that work for everyone?

Joe

—

---

From: Joe Touch <[email-masked]>
Date: Mon, 4 Apr 2022 18:36:59 -0700

Minor but confusing typo:

> On Apr 4, 2022, at 9:14 AM, [email-masked] wrote:
> 
> NOTE: if OCS

That should say DOS rather than OCS there. 

> points where the last IP packet end is, then there are no per-packet options and no per-packet OCS.
> If OCS

Again, DOS, not OCS. 

> points before FRAGSTART, the same is true - the per-packet options pick up there, but there is no OCS.

The other occurrences of OCS are correct.

---

From: "C. M. Heard" <[email-masked]>
Date: Tue, 5 Apr 2022 05:24:02 -0700

On Mon, Apr 4, 2022 at 9:14 AM Joe Touch  wrote:

> I think there may already be a way forward here that allows users to pick
> from a variety of ways to support FRAGs that may satisfy the needs of
> constrained-implementations of router header processing.
>
> If DOS is smaller than FRAGSTART, then it indicates the end of the
> per-fragment options and start of per-packet options, all of which precede
> the frag data.
>

I don't believe that this works. From the definitions in the draft:

- Frag. Start indicates the location of the beginning of the fragment data,
***measured from the beginning of the UDP header of the fragment***.

- The terminal FRAG option format adds a Datagram Option Start pointer,
***measured from the start of the original UDP datagram  header***,
indicating the end of the reassembled data and the start of the surplus
area after the original UDP datagram.

In other words, DOS is nothing more and nothing less than the UDP Length in
the UDP header of the original unfragmented datagram. As noted in the
description of the fragmentation process:

- Note that all the non-FRAG options in step #1 need not be limited to the
terminal fragment, i.e., the Dgram Opt. Start pointer can indicate the
start of the original surplus area anywhere in the reassembled data.

In other words, fragment boundaries are unrelated to the split between the
user data and the surplus area in the original unfragmented datagram.

This last point is key, and what to me makes the specification very
compelling: the fragmentation and reassembly processes are ***decoupled***
from the  pre-fragmentation and post-reassembly processing. The latter
proceed exactly as in the unfragmented case

NOTE: if [DOS] points where the last IP packet end is, then there are no
> per-packet options and no per-packet OCS.
>

That is correct. In this case, the UDP Length of the reassembled datagram
will point to the end of the datagram, and the surplus area will be absent.

If [DOS] points before FRAGSTART, the same is true - the per-packet options
> pick up there, but there is no OCS.
>

This I do not understand. As far as I can tell, DOS has nothing to do with
FRAGSTART.

If I'm missing something, please let me know what it is.

Thanks,

Mike Heard

---

From: "[email-masked]" <[email-masked]>
Date: Tue, 5 Apr 2022 07:38:49 -0700

Hi, Mike,

> On Apr 5, 2022, at 5:24 AM, C. M. Heard <[email-masked]> wrote:
> 

%% 
To be more clear: we can REDEFINE DOS to be anywhere...

> I.e.: if DOS is larger than FRAGSTART, then the per-packet options are at the end of the packet.

%%
To be more clear again, the NEW definition of DOS would be:

If DOS is larger than the FRAGSTART *of the first fragment (offset=0)* then the per-packet options are at the end of the packet.

> If DOS is smaller than FRAGSTART, then it indicates the end of the per-fragment options and start of per-packet options, all of which precede the frag data.

%%
And again more clearly:

If DOS is s smaller than the  FRAGSTART *of the first fragment (offset=0)*, then it indicates the end of the per-fragment options and start of per-packet options, all of which precede the frag data.

> - Frag. Start indicates the location of the beginning of the fragment data, ***measured from the beginning of the UDP header of the fragment***.

That definition would stay.

> - The terminal FRAG option format adds a Datagram Option Start pointer, ***measured from the start of the original UDP datagram  header***,

That part stays, but the next is now CHANGED as above (see %%):

> indicating the end of the reassembled data and the start of the surplus area after the original UDP datagram.

The changed text would be something like:

Indicating the start of the options, which could occur either after the original UDP datagram or before.

The reassembly text would also change accordingly.

If you prefer, I can issue a -19 to roll these in, but I was looking to see if this is what will satisfy everyone before I did that.

To continue...

> In other words, DOS is nothing more and nothing less than the UDP Length in the UDP header of the original unfragmented datagram. As noted in the description of the fragmentation process:
> 
> - Note that all the non-FRAG options in step #1 need not be limited to the terminal fragment, i.e., the Dgram Opt. Start pointer can indicate the start of the original surplus area anywhere in the reassembled data.
> 
> In other words, fragment boundaries are unrelated to the split between the user data and the surplus area in the original unfragmented datagram.
> 
> This last point is key, and what to me makes the specification very compelling: the fragmentation and reassembly processes are ***decoupled*** from the  pre-fragmentation and post-reassembly processing. The latter proceed exactly as in the unfragmented case

I do find this compelling, but recognize that it also prevents placing all of the options in what are mistakenly called “headers” but would be accessible within 500 bytes or so of the UDP header - provided the sum of the two option areas (per-fragment, per-packet) is 500 bytes or fewer.

> NOTE: if [DOS] points where the last IP packet end is, then there are no per-packet options and no per-packet OCS.
> 
> That is correct. In this case, the UDP Length of the reassembled datagram will point to the end of the datagram, and the surplus area will be absent.
> 
> If [DOS] points before FRAGSTART, the same is true - the per-packet options pick up there, but there is no OCS.
> 
> This I do not understand. As far as I can tell, DOS has nothing to do with FRAGSTART.

I’m allowing DOS to point before the FRAGSTART of the fragment when FRAGOFFSET is zero, which then becomes the demarcation between per-fragment and per-packet options.

This is preferable to using the FRAG option itself as the demarcation (as Tom had suggested) because it allows FRAG to remain up front, enabling DMA without long-chain TLV searches.

Again, *IF WE ALLOW DOS to point before the first FRAGSTART*, then we do give up having the reassembled packet have exactly the same format as if it were not fragmented. I would prefer to not do this, especially to support a router hardware implementation decision that has no basis in RFC requirements, but we can if it’s consensus to do so.

Joe

---

From: "[email-masked]" <[email-masked]>
Date: Tue, 5 Apr 2022 08:33:51 -0700

A few more notes:

I was assuming “offset=0” is the starting fragment offset, though the current text indicates this would be offset=8. We can pick either starting point (i.e., either the start of the original UDP header or the end of that header). So read the note below with that context, i.e., ‘whatever the offset of the first fragment is’…

And throughout the rest, I’m giving a broad quick summary; please try to focus on the basic idea and assume we can work out the rough edges that don’t concur with text in the current draft, unless you see a deal-breaker that I missed. I.e., when this proposal differs with the doc, assume the doc will be pulled in line with this proposal...

Joe

—

> On Apr 5, 2022, at 7:38 AM, [email-masked] wrote:
> 
> Hi, Mike,
> 
> 
>> On Apr 5, 2022, at 5:24 AM, C. M. Heard <[email-masked] <mailto:[email-masked]>> wrote:
>>

---

From: "C. M. Heard" <[email-masked]>
Date: Thu, 7 Apr 2022 09:16:29 -0700

On Tue, Apr 5, 2022 at 8:34 AM Joe Touch wrote:

> A few more notes:
>
> I was assuming “offset=0” is the starting fragment offset, though the
> current text indicates this would be offset=8. We can pick either starting
> point (i.e., either the start of the original UDP header or the end of that
> header). So read the note below with that context, i.e., ‘whatever the
> offset of the first fragment is’…
>
> And throughout the rest, I’m giving a broad quick summary; please try to
> focus on the basic idea and assume we can work out the rough edges that
> don’t concur with text in the current draft, unless you see a deal-breaker
> that I missed. I.e., when this proposal differs with the doc, assume the
> doc will be pulled in line with this proposal...
>

Understood and agreed to. I don't much care whether FRAG.OFFSET = 0 or
FRAG.OFFSET = 8 indicates the first fragment, as long as the final spec
tells a consistent story. For definiteness in the example that I provide
below, I will assume that all offsets are measured with respect to the
appropriate UDP header, so FRAG.OFFSET = 8 indicates the first fragment in
that example.

On Mon, Apr 4, 2022 at 9:14 AM Joe Touch  wrote:
> If DOS is larger than the FRAGSTART *of the first fragment (offset=0)*
> then the per-packet options are at the end of the packet.
>
>
> If DOS is s smaller than the  FRAGSTART *of the first fragment
> (offset=0)*, then it indicates the end of the per-fragment options and
> start of per-packet options, all of which precede the frag data.
>
> [ ... ]

> - Frag. Start indicates the location of the beginning of the fragment
> data, ***measured from the beginning of the UDP header of the fragment***.
>
>

> I’m allowing DOS to point before the FRAGSTART of the fragment when
> FRAGOFFSET is zero, which then becomes the demarcation between per-fragment
> and per-packet options.
>
>
With apologies, I am still having a very hard time figuring out how this is
supposed to work. FRAG.START is an offset ***within a fragment*** and any
data in that fragment that occurs before that point contains per-fragment
options that are not part of the reassembled packet. That applies in
particular to the first fragment (wherein FRAG.OFFSET is eight, if we
reckon such offsets with respect to the start of the UDP header). DOS,
which appears in the final fragment only, is an offset ***within the
reassembled datagram***, as is FRAG.OFFSET. To my small mind, it seems
that FRAG.START
is ***incommensurate*** with DOS. I'm not sure what it means to say that
DOS points before FRAG.START, since they are offsets within different data
structures. Also, when there are multiple fragments, DOS appears in a
***different*** fragment than the FRAG.START with which it is to be
compared.

Moreover: it is not difficult to concoct an example of an atomic fragment,
constructed under the rules in -18, that has DOS numerically smaller than
FRAG.START. Start with a UDP datagram having the usual 8-byte UDP header,
32 bytes of UDP user data (so UDP Length = 40), and some trailing
per-datagram options -- for definiteness, let's say OCS and a TIME option,
for a total of 12 bytes (though this does not really matter). Let's suppose
that we wish to  encrypt this using a per-fragment UENC option; again for
definiteness, I assume that the UENC option uses a 16-byte MAC and so has a
total length of 20 bytes. The on-the-wire packet will then have a UDP
header that is 8 bytes long with its length field set to 8, a two-byte OCS,
two one-byte NOPs for alignment, a terminal FRAG option with a length of 12
bytes, and a UENC option with a length of 20 bytes. Irrespective of the
order of the FRAG and UENC options, we have FRAG.START = 44, FRAG.OFFSET =
8, and DOS = 40.

> This is preferable to using the FRAG option itself as the demarcation (as
> Tom had suggested) because it allows FRAG to remain up front, enabling DMA
> without long-chain TLV searches.
>
> Again, *IF WE ALLOW DOS to point before the first FRAGSTART*, then we do
> give up having the reassembled packet have exactly the same format as if it
> were not fragmented. I would prefer to not do this, especially to support a
> router hardware implementation decision that has no basis in RFC
> requirements, but we can if it’s consensus to do so.
>
>
If I understand the ***intent*** of the proposal to allow DOS to point
before the first FRAG.START, it is to allow a special format for
reassembled packets, in addition to the one already in -18, wherein the
reassembled packet has exactly the same format as if it were not
fragmented. If we give that up in some cases, I would MUCH prefer to end up
with just one one format for reassembled packets. Specifically: I think we
should keep the existing format for unfragmented packets that contain
options in an (optional) trailer, and have another format with all options
at the front for packets that lack conventional UDP data. And if we take
that route, my sense is that we would be better off to adopt something
along the lines of what is proposed in
https://datatracker.ietf.org/doc/html/draft-herbert-udp-space-hdr for those
packets. I can see how to make that work, and in the cases that I have
worked out, it saves space compared to having a FRAG option as in -18 and
still satisfies the desideratum of having the FRAG control information
early to facilitate DMA offload for reassembly. I'll take the time to
provide details if there is WG interest. If not, I am satisfied to proceed
with -18 (modulo polishing of rough edges) with documentation of the
limitations as Mr. Herbert requests.

Mike Heard

---

From: "[email-masked]" <[email-masked]>
Date: Thu, 7 Apr 2022 10:22:31 -0700

Hi, Mike,

Some notes below.

> On Apr 7, 2022, at 9:16 AM, C. M. Heard <[email-masked]> wrote:
> 
> Understood and agreed to. I don't much care whether FRAG.OFFSET = 0 or FRAG.OFFSET = 8 indicates the first fragment, as long as the final spec tells a consistent story. For definiteness in the example that I provide below, I will assume that all offsets are measured with respect to the appropriate UDP header, so FRAG.OFFSET = 8 indicates the first fragment in that example.
> 
> With apologies, I am still having a very hard time figuring out how this is supposed to work. FRAG.START is an offset ***within a fragment*** and any data in that fragment that occurs before that point contains per-fragment options that are not part of the reassembled packet.

Again, let’s wait for -19

> That applies in particular to the first fragment (wherein FRAG.OFFSET is eight, if we reckon such offsets with respect to the start of the UDP header). DOS, which appears in the final fragment only, is an offset ***within the reassembled datagram***, as is FRAG.OFFSET. To my small mind, it seems that FRAG.START is ***incommensurate*** with DOS. I'm not sure what it means to say that DOS points before FRAG.START, since they are offsets within different data structures. Also, when there are multiple fragments, DOS appears in a ***different*** fragment than the FRAG.START with which it is to be compared.

They are intended to be commensurate ONLY In atomic fragments. I never intended for this to be viable with non-atomic fragments, 

> Moreover: it is not difficult to concoct an example of an atomic fragment, constructed under the rules in -18, that has DOSnumerically smaller than FRAG.START. Start with a UDP datagram having the usual 8-byte UDP header, 32 bytes of UDP user data (so UDP Length = 40), and some trailing per-datagram options -- for definiteness, let's say OCS and a TIME option, for a total of 12 bytes (though this does not really matter). Let's suppose that we wish to  encrypt this using a per-fragment UENC option; again for definiteness, I assume that the UENC option uses a 16-byte MAC and so has a total length of 20 bytes. The on-the-wire packet will then have a UDP header that is 8 bytes long with its length field set to 8, a two-byte OCS, two one-byte NOPs for alignment, a terminal FRAG option with a length of 12 bytes, and a UENC option with a length of 20 bytes. Irrespective of the order of the FRAG and UENC options, we have FRAG.START = 44, FRAG.OFFSET = 8, and DOS = 40.

It wouldn’t have the NOPs (we’re requiring only 2-byte alignment for OCS), but let’s ignore that because two extra NOPs are allowed.

I had been assuming that DOS was relative to the final packet in the atomic fragment case, not the original. But only in that case.

Here:
	 8 UDP header
	 2 OCS
	 2 NOP (not needed, but left to be comparable to your example)
	12 terminal FRAG with FRAG.START=44, FRAG.OFFSET=8, and DOS=76
	20 UENC
	32 user data
	2 OCS (actually, we would not have this in terminal fragments anymore, but let’s leave it for the example)
	10 TIME

> If I understand the ***intent*** of the proposal to allow DOS to point before the first FRAG.START, it is to allow a special format for reassembled packets,

For ATOMIC “reassembled” packets only.

> in addition to the one already in -18, wherein the reassembled packet has exactly the same format as if it were not fragmented. If we give that up in some cases, I would MUCH prefer to end up with just one one format for reassembled packets.

As would I, but the problem is that I do NOT think we should limit options in multi-packet fragments to the size of a single fragment, either initial or terminal. That means that in multifragment packets, DOS should be absolute, i.e., counted from the initial packet. But that would mean DOS cannot be used as a boundary outside a single packet except in the atomic packet case.

> Specifically: I think we should keep the existing format for unfragmented packets that contain options in an (optional) trailer, and have another format with all options at the front for packets that lack conventional UDP data.

I don’t understand what this means.

The only packets that lack conventional UDP data are ones that either have no options (which we don’t affect) or ones that do. But there are two kinds of packets that could have options and data:

	1) packets with options that fit in the same packet as the data (per-fragment options)
	2) packets with options *as data*, i.e., the long trailer options that might not fit in a single UDP packet

> And if we take that route, my sense is that we would be better off to adopt something along the lines of what is proposed in https://datatracker.ietf.org/doc/html/draft-herbert-udp-space-hdr <https://datatracker.ietf.org/doc/html/draft-herbert-udp-space-hdr> for those packets. I can see how to make that work, and in the cases that I have worked out, it saves space compared to having a FRAG option as in -18 and still satisfies the desideratum of having the FRAG control information early to facilitate DMA offload for reassembly. I'll take the time to provide details if there is WG interest.

I cannot see how to make that work for case #2 above, which is a key feature of the current design. We do NOT limit options to fitting in a single packet, either just as trailers or in sum with per-fragment options.

> If not, I am satisfied to proceed with -18 (modulo polishing of rough edges) with documentation of the limitations as Mr. Herbert requests.
> 
> Mike Heard
> 
> 
> 
>

---

From: Joe Touch <[email-masked]>
Date: Thu, 7 Apr 2022 11:45:26 -0700

Ps - it might be a LOT simpler to say that limited routers should use only per fragment options. We can then leave things as they are and avoid the complexity of varying format interpretations. 

Joe

---

From: "C. M. Heard" <[email-masked]>
Date: Thu, 7 Apr 2022 12:47:49 -0700

On Thu, Apr 7, 2022 at 10:22 AM Joe Touch wrote:

> On Apr 7, 2022, at 9:16 AM, C. M. Heard wrote:
> With apologies, I am still having a very hard time figuring out how this
> is supposed to work. FRAG.START is an offset ***within a fragment*** and
> any data in that fragment that occurs before that point contains
> per-fragment options that are not part of the reassembled packet.
>
>

I would ask that we not pollute -19 with this stuff UNLESS there is a CLEAR
consensus to do so.

> They are intended to be commensurate ONLY In atomic fragments. I never
> intended for this to be viable with non-atomic fragments,
>

OK.

> constructed under the rules in -18, that has DOSnumerically smaller than
> It wouldn’t have the NOPs (we’re requiring only 2-byte alignment for OCS),
> but let’s ignore that because two extra NOPs are allowed.
>
> I had been assuming that DOS was relative to the final packet in the
> atomic fragment case, not the original. But only in that case.
>
> Here:
>  8 UDP header
>  2 OCS
DOS-76 makes NO SENSE with respect to the REASSEMBLED PACKET, as that
points PAST THE END of the reassembled packet (which is at offset 52)  The
20 bytes of UENC -- as I specified -- is a PER-FRAGMENT option, intended to
protect both the fragment data, including any post-reassembly options
embedded therein.

> reassembled packets,

OK, I didn't understand that.

> in addition to the one already in -18, wherein the reassembled packet has
> exactly the same format as if it were not fragmented. If we give that up in
> some cases, I would MUCH prefer to end up with just one one format for
> reassembled packets.
>
>
> As would I, but the problem is that I do NOT think we should limit options
> in multi-packet fragments to the size of a single fragment, either initial
> or terminal. That means that in multifragment packets, DOS should be
> absolute, i.e., counted from the initial packet. But that would mean DOS
> cannot be used as a boundary outside a single packet except in the atomic
> packet case.
>
> Specifically: I think we should keep the existing format for unfragmented
> packets that contain options in an (optional) trailer, and have another
> format with all options at the front for packets that lack conventional UDP
> data.
>
>

I hope that "the existing format for unfragmented packets that contain
options in an (optional) trailer" is clear enough.

The other case was anything involving a UDP header with UDP Length = 8.
That's what I mean by lacking conventional UDP data.

> The only packets that lack conventional UDP data are ones that either have
> no options (which we don’t affect) or ones that do. But there are two kinds
> of packets that could have options and data:
>
> And if we take that route, my sense is that we would be better off to
> adopt something along the lines of what is proposed in
> https://datatracker.ietf.org/doc/html/draft-herbert-udp-space-hdr for
> those packets. I can see how to make that work, and in the cases that I
> have worked out, it saves space compared to having a FRAG option as in -18
> and still satisfies the desideratum of having the FRAG control information
> provide details if there is WG interest.
>
>
> I cannot see how to make that work for case #2 above, which is a key
> feature of the current design. We do NOT limit options to fitting in a
> single packet, either just as trailers or in sum with per-fragment options.
>

Absolutely, it could not.

On Thu, Apr 7, 2022 at 11:45 AM Joe Touch wrote:

> Ps - it might be a LOT simpler to say that limited routers should use only
> per fragment options. We can then leave things as they are and avoid the
> complexity of varying format interpretations.

I would MUCH prefer to see that than to have varying interpretations in -19
(to which I would, in fact, strongly object).

With the current specification, routers unable to process trailers can
still process per-fragment options, a point that I have made before, and
they could even use fragmentation and reassembly in tunnels. But
per-datagram options would be beyond their reach. So far there has been no
evidence presented that this would not satisfy real-world use cases. Tom
Herbert, please provide a counterexample if there is one.

Thanks,

Mike Heard

---

From: Tom Herbert <[email-masked]>
Date: Thu, 7 Apr 2022 13:06:41 -0700

On Thu, Apr 7, 2022, 12:48 PM C. M. Heard <[email-masked]> wrote:

> I would ask that we not pollute -19 with this stuff UNLESS there is a
> CLEAR consensus to do so.
>
>
>> Moreover: it is not difficult to concoct an example of an atomic
>> fragment, constructed under the rules in -18, that has DOSnumerically
>> smaller than FRAG.START. Start with a UDP datagram having the usual 8-byte
>> UDP header, 32 bytes of UDP user data (so UDP Length = 40), and some
>> trailing per-datagram options -- for definiteness, let's say OCS and a TIME
>> option, for a total of 12 bytes (though this does not really matter). Let's
>> suppose that we wish to  encrypt this using a per-fragment UENC option;
>> again for definiteness, I assume that the UENC option uses a 16-byte MAC
>> and so has a total length of 20 bytes. The on-the-wire packet will then
>> have a UDP header that is 8 bytes long with its length field set to 8, a
>> two-byte OCS, two one-byte NOPs for alignment, a terminal FRAG option with
>> a length of 12 bytes, and a UENC option with a length of 20 bytes.
>> Irrespective of the order of the FRAG and UENC options, we have FRAG.START
>> = 44, FRAG.OFFSET = 8, and DOS = 40.
>>
>>
>> It wouldn’t have the NOPs (we’re requiring only 2-byte alignment for
>> OCS), but let’s ignore that because two extra NOPs are allowed.
>>
>> As would I, but the problem is that I do NOT think we should limit
>> options in multi-packet fragments to the size of a single fragment, either
>> initial or terminal. That means that in multifragment packets, DOS should
>> be absolute, i.e., counted from the initial packet. But that would mean DOS
>> The only packets that lack conventional UDP data are ones that either
>> have no options (which we don’t affect) or ones that do. But there are two
>> kinds of packets that could have options and data:
>>
>> Ps - it might be a LOT simpler to say that limited routers should use
>> only per fragment options. We can then leave things as they are and avoid
>> the complexity of varying format interpretations.
>
>
> I would MUCH prefer to see that than to have varying interpretations in
> -19 (to which I would, in fact, strongly object).
>

Mike,

IPv6 allows both per fragment options and per reassembled packet options.
This is enabled by Destination options before the Fragment header, and
Destination options after the Fragment headers respectively. All of these
are in headers of packets.

Tom

---

From: Joe Touch <[email-masked]>
Date: Thu, 7 Apr 2022 14:16:45 -0700

> On Apr 7, 2022, at 12:48 PM, C. M. Heard <[email-masked]> wrote:
> 
> I would MUCH prefer to see that than to have varying interpretations in -19 (to which I would, in fact, strongly object)

Additionally, this approach allows UDP fragmentation as BITW, which could be useful in traversing paths with f do ie stream MTU limits. 

The only downside is limiting options to per frag for constrained implementation routers, which seems like a reasonable compromise to me. 

Joe

---

From: Tom Herbert <[email-masked]>
Date: Sun, 10 Apr 2022 08:14:08 -0700

On Thu, Apr 7, 2022 at 2:17 PM Joe Touch <[email-masked]> wrote:

> -19 (to which I would, in fact, strongly object)
>
>
> Additionally, this approach allows UDP fragmentation as BITW, which could
> be useful in traversing paths with f do ie stream MTU limits.
>
> The only downside is limiting options to per frag for constrained
> implementation routers, which seems like a reasonable compromise to me.
>
>
Joe,

That would be constraining users' use of UDP options which is contrary to
the oft stated goal of UDP options not to constrain users and leave it to
their choice. We don't not know the scope of this problem, it's quite
conceivable that other devices may also have limitations that would prevent
processing of trailers (I imagine this might affect low end IoT devices).
But even if this somehow adequately addresses the problem with routers, it
doesn't address the loss of common accelerations in hosts like GRO, TSO,
header-data split, etc. which I have previously raised as an issue.

The only benefit of this "compromise" I see is that UDP options could have
more than MTU size length. When I asked for a real world use case of this
there was no answer. Frankly, other than would-be attackers, there is
simply no need for this. As I stated previously, IPv6 has already solved
the problem of options related to fragmentation by ensuring that options
are in headers. Not only that, IPv6 also requires that all options and the
whole header chain up to the upper layer header. From RFC8200:

Extension headers, if any, and the Upper-Layer header.  These headers
must be in the first fragment.

Note: This restricts the size of the headers through the Upper-Layer
header to the

MTU of the path to the packet's destinations(s).

So, IMO, the best answer to the problem is to follow IPv6's lead and
create a format that allows all UDP options to reside in exclusively
headers of packets. For fragmentation, this can be accomplished by
placing per fragment options before the fragment option, and per
reassembled options after the fragment options. Also, again following
IPv6's lead, if the packet is not a fragment then don't force an
unnecessary fragment option into the packet, a simple length of UDP
options field after the OCS is sufficient and save twelve bytes of
header overhead.

Tom

Joe

---

From: "C. M. Heard" <[email-masked]>
Date: Sun, 10 Apr 2022 20:34:58 -0700

On Sun, Apr 10, 2022 at 8:14 AM Tom Herbert wrote:

> On Thu, Apr 7, 2022 at 2:17 PM Joe Touch wrote:
>
>> [ ...]

> header-data split, etc. which I have previously raised as an issue.

Tom, as far as I can tell, GRO, TSO, and header-data split are not ruled
out by the FRAG formats proposed in -18 if there are only per-frag options.
If they are, it would be helpful if you would explain why.

> more than MTU size length.
>

And that the reassembled datagram has exactly the same format as in the
non-fragmented case (a compelling feature of -18, IMO).

> When I asked for a real world use case of this there was no answer.
> Frankly, other than would-be attackers, there is simply no need for this.
> As I stated previously, IPv6 has already solved the problem of options
> related to fragmentation by ensuring that options are in headers. Not only
> that, IPv6 also requires that all options and the whole header chain up to
> the upper layer header. From RFC8200:
>
> Extension headers, if any, and the Upper-Layer header.  These headers must
> be in the first fragment. Note: This restricts the size of the headers
> through the Upper-Layer header to the MTU of the path to the packet's
> destinations(s).
>

If we are going to be honest with our rhetoric, we need to remember that
the motivation for that restriction was all about allowing intermediate
systems to see the transport header in the first fragment (which is not an
issue for UDP options). It was NOT about limiting the length of the
extension headers and transport header to a length that could be processed
by devices that could see into only the first 128. 256, 384, or 512 bytes
of the packet. The latter was discussed by the IPv6 WG and explicitly
rejected.

> So, IMO, the best answer to the problem is to follow IPv6's lead and create a format that allows all UDP options to reside in exclusively headers of packets. For fragmentation, this can be accomplished by placing per fragment options before the fragment option, and per reassembled options after the fragment options. Also, again following IPv6's lead, if the packet is not a fragment then don't force an unnecessary fragment option into the packet, a simple length of UDP options field after the OCS is sufficient and save twelve bytes of header overhead.
>
>
IMO this is a bad idea because it forces devices to parse through all the
per-fragment option TLVs to find out if a FRAG option is present. As I
understand it, the motivation behind the proposal in -18 to put the FRAG
header as close as possible to the UDP header is to minimize the amount of
TLV processing that something like an NPU-based NIC would need to do in
order to use DMA to support the reassembly process.

That being said, it is possible to devise a format somewhat akin to Section
2.2 of https://www.ietf.org/archive/id/draft-herbert-udp-space-hdr-01.txt
could perhaps fulfill all of the following desiderata:

- supports per-datagram and per-fragment options
- per-datagram options appear first in the reassembled datagram
- per-datagram options may be in first frag but are not limited to that
- no frag control fields needed for unfragmented datagrams
- supports header/data split and efficient DMA

Here is a strawman proposal. If no one supports it, I am perfectly happy to
proceed with -18 as is, modulo remaining edits for internal consistency.

The basic idea is to replace the FRAG option and its variants with a
variable-length header that appears when, and only when, UDP Length = 8; we
would continue to use the trailer format when UDP Length >= 9. The general
format would be:

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\
   |        Source Port            |      Destination Port         | |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ UDP
   |       UDP Length = 8          |        UDP Checksum           | |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+/
   |    Bit-Encoded Type/Length    |            OCS                |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   ~                      Type Specific Data                       ~
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The format for non-fragments would look something like this:

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\
   |        Source Port            |      Destination Port         | |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ UDP
   |       UDP Length = 8          |        UDP Checksum           | |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+/
   |  Datagram User Data Offset  |0|            OCS                |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   ~      UDP Options (without OCS) followed by dgram user data    ~
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The format for initial fragments would look something like this:

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\
   |        Source Port            |      Destination Port         | |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ UDP
   |       UDP Length = 8          |        UDP Checksum           | |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+/
   | Frag Data Start |0 0|1 1 1 1 1|            OCS                |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                        Identification                         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Datagram User Data Offset  |0|                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+     Per-Fragment              ~
   |                            Options                            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   ~                         Fragment Data                         ~
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The format for intermediate fragments would look something like this:

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\
   |        Source Port            |      Destination Port         | |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ UDP
   |       UDP Length = 8          |        UDP Checksum           | |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+/
   | Frag Data Start |0 1|1 1 1 1 1|            OCS                |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                        Identification                         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Fragment Data Offset    |                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+     Per-Fragment              ~
   |                            Options                            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   ~                         Fragment Data                         ~
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The format for final fragments would look something like this:

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\
   |        Source Port            |      Destination Port         | |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ UDP
   |       UDP Length = 8          |        UDP Checksum           | |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+/
   | Frag Data Start |1 1|1 1 1 1 1|            OCS                |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                        Identification                         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Fragment Data Offset    |                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+     Per-Fragment              ~
   |                            Options                            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   ~                         Fragment Data                         ~
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Notes:

1. I envision the conceptual reassembled datagram without its UDP header,
just options (without OCS) followed by user data.
2. The non-fragmented case would be the same, just as if it consisted of an
atomic datagram with the per-frag stuff removed.
3. Datagram User Data Offset would be measured in 16-bit words from the
start of the conceptual reassembled datagram.
4. Fragment Data Offset would be measured in bytes from the start of
the conceptual
reassembled datagram. It is zero in initial fragments.
5. Frag. Data Start would be measured in 32-bit words from the start of the
fragment's UDP header.
6. Bit-Encoded Type/Length fields of the following form would be reserved
for future standardization:

    0                   1
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |X X X X X X X X X X X|0 1 1 1 1|
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

I'll admit that I am not thrilled about making reassembled datagrams (or
non-fragments with unsafe options) have a different format than in the
non-fragmented case with only safe options (UDP Length >= 9). I am also not
thrilled about extending the debate -- I think we need to get basic
decisions made and get the specification shipped. But I think this strawman
proposal would work, and I offer it up to see if it will help address
significant concerns of the WG.

Mike Heard

---

From: "[email-masked]" <[email-masked]>
Date: Sun, 10 Apr 2022 21:31:26 -0700

On Apr 10, 2022, at 8:34 PM, C. M. Heard <[email-masked]> wrote:
> 
> I'll admit that I am not thrilled about making reassembled datagrams (or non-fragments with unsafe options) have a different format than in the non-fragmented case with only safe options (UDP Length >= 9).

+1

You’ve just undone the ability to use fragmentation as BITW, which is a feature of the current design. Notably, it allows tunnels to use fragmentation while endpoints use packet options without either knowing about each other at all, i.e., that tunnels could operate independently of endpoints.

I won’t bother with most of Tom's other arguments that led to this proposal; they are based on the claim that trailers aren’t used (Teredo is a counterexample), that IOT devices could have hardware packet processing limitations (where those optimizations appear in big-iron routers, not lite-iron IOT), or the claim that because IPv6 uses headers (a TLV chain of headers) UDP should use fixed headers (where we aren’t really using headers at all).

The byte savings shown are for a few simple reasons, but they come at the cost of additional special-case code at both endpoints:

	a) you save 2 bytes of kind/length by assuming the kind (i.e., that FRAG always follows OCS)
		but you also prevent the ability to encrypt or authenticate before FRAG is processed (options are processed in order)
		i.e., even though we allow FRAG to come early, we don’t necessarily want it in a fixed place for that reason

	b) but the length byte you saved needs to be encoded elsewhere (that’s basically the encoding of the flags and such)

	c) you save 4 bytes by omitting the ID for atomic fragments 
		(note: IPv6 allows them, IPv4 allows them too)
		this can be useful in endpoints for things like duplicate packet filtering (as long as they’re not repeated over known periods)
		so again, you’ve removed a feature

	d) you limit the length of options - more than just being in the initial packet, now you’re also doing the following:
		- in atomic ‘fragments’ you’re limited to 32K of options; granted, that’s plenty
		- in initial fragments, you’re limiting the sum of per-packet and per-fragment options to 2KB for no good reason

If we also consider the kinds of things we know middle boxes are likely to end up trying to do, we want the existing information that has been omitted (to save space), e.g., the atomic fragment allows downstream BITW UDP fragmentation to occur without needing to worry that new ID fields might collide with others inserted at the source.

So sure, we can have variants that save 2 or 6 bytes using the existing format where the shorter versions are used exclusively either for initial fragments (omitting the FRAGOFFSET value) or atomic (omitting both FRAGOFFSET and ID, though that interferes with the BITW trick above).

I don’t see the value in these sort of complications, as noted above.

Joe

---

From: "C. M. Heard" <[email-masked]>
Date: Sun, 10 Apr 2022 21:37:21 -0700

On Sun, Apr 10, 2022 at 9:31 PM Joe Tuuch wrote:

> You’ve just undone the ability to use fragmentation as BITW, which is a
> feature of the current design. Notably, it allows tunnels to use
> fragmentation while endpoints use packet options without either knowing
> about each other at all, i.e., that tunnels could operate independently of
> endpoints.
>

I do not understand that claim. Seems to me that the strawman proposal
would work just as well as -18 for BITW fragmentation, especially if
per-datagram options were not used.

Mike

---

From: "[email-masked]" <[email-masked]>
Date: Sun, 10 Apr 2022 21:45:11 -0700

—

> On Apr 10, 2022, at 9:37 PM, C. M. Heard <[email-masked]> wrote:
> 
> I do not understand that claim. Seems to me that the strawman proposal would work just as well as -18 for BITW fragmentation, especially if per-datagram options were not used.

BITW is based on the idea that one part of the protocol uses per-datagram options and another performs fragmentation, whether those are two parts of a single codebase (implemented separately) or implemented in different devices (whether on the same endpoint or one downstream of the other).

If you have to assume non per-datagram options, you can’t be BITW anymore. You can be “BITW but only for endpoints that don’t use UDP options”. That takes away the ability to be a “B” (the whole point of BITW is that it can be inserted or deleted without such conditions).

Joe

---

From: "[email-masked]" <[email-masked]>
Date: Sun, 10 Apr 2022 21:48:06 -0700

See below…

> On Apr 10, 2022, at 8:14 AM, Tom Herbert <[email-masked]> wrote:
> 
> On Thu, Apr 7, 2022 at 2:17 PM Joe Touch <[email-masked] <mailto:[email-masked]>> wrote:
> ...
> That would be constraining users' use of UDP options which is contrary to the oft stated goal of UDP options not to constrain users and leave it to their choice.

It constrains implementations on devices that are broken, which seems appropriate to me. By “broken”, I mean that they already do not support the full variety of existing protocol standards. I do not agree that we should make things more complex for everyone simply to accommodate devices that have gone off the standards path. 

Joe

---

From: Tom Herbert <[email-masked]>
Date: Mon, 11 Apr 2022 06:57:34 -0700

On Sun, Apr 10, 2022 at 9:48 PM [email-masked] <[email-masked]>
wrote:

> their choice.
> It constrains implementations on devices that are broken, which seems
> appropriate to me. By “broken”, I mean that they already do not support the
> full variety of existing protocol standards. I do not agree that we should
> make things more complex for everyone simply to accommodate devices that
> have gone off the standards path.
>

Joe,

Regardless of whether these devices are "broken" or not, they are
prevalent. Without analysis or interoperability testing of a running
implementation we really don't know the scope or the effects of trying to
deploy UDP options in the real world. But either way, these limitations on
use cases of UDP options should be documented.

Tom

---

From: "[email-masked]" <[email-masked]>
Date: Mon, 11 Apr 2022 07:34:34 -0700

—

> On Apr 11, 2022, at 6:57 AM, Tom Herbert <[email-masked]> wrote:
> 
> Joe,
> 
> Regardless of whether these devices are "broken" or not,

Absent a pointer to a spec that requires such limits as they have, they are.

> they are prevalent.

You yourself say that you don’t know how prevalent, though. And if they are prevalent, how does that affect the prevalence (or not) of Teredo extensions, that use trailers? We don’t know.

> Without analysis or interoperability testing of a running implementation we really don't know the scope or the effects of trying to deploy UDP options in the real world.

That’s partly true, but also true that so many devices have taken liberties with standards. Given how little information we can gather about such devices, it’s not clear we will ever know.

> But either way, these limitations on use cases of UDP options should be documented.

I’ve already noted that will happen in -19, including the suggestion that options to those endpoints accept those limitations.

IMO, that’s vastly better than redesigning the approach for those devices at the expense of additional code, processing, and validation of implementations for other devices.

---

From: "C. M. Heard" <[email-masked]>
Date: Mon, 11 Apr 2022 09:14:59 -0700

On Sun, Apr 10, 2022 at 9:45 PM Joe Touch wrote:

> BITW is based on the idea that one part of the protocol uses per-datagram
> options and another performs fragmentation, whether those are two parts of
> a single codebase (implemented separately) or implemented in different
> devices (whether on the same endpoint or one downstream of the other).
>

Thank you, I now understand what you mean. Such a BITW (or BITS - bump in
the stack) implementation of UDP fragmentation could be extended to handle
per-datagram UDP encryption (or any other relevant per-fragment options),
should that be wanted. And as far as I can tell, it could take full
advantage of checksum offload, GSO, and GRO (see
https://datatracker.ietf.org/meeting/113/materials/slides-113-tsvwg-71-checksum-offload-00
and
the analysis in
https://mailarchive.ietf.org/arch/msg/tsvwg/9_1YWp-TApI9mcCuia8U6jfCwTA/).

If you have to assume [no] per-datagram options, you can’t be BITW anymore.
> You can be “BITW but only for endpoints that don’t use UDP options”. That
> takes away the ability to be a “B” (the whole point of BITW is that it can
> be inserted or deleted without such conditions).
>

The scenario I had in mind was different: it was using UDP fragmentation in
a tunnel with one or both ends terminated in a router with the ability to
inspect packets only to a limited depth from the beginning. This is the
scenario that Tom Herbert put forth. Certainly such a tunnel could have a
use for UDP fragmentation if the path MTU is less than the size of the
packets it is supposed to carry, and it could conceivably benefit from
running DPLPMTUD, which would bring REQ and RES and possibly MDS and MRDS
into play. In this scenario all of those options would work fine as
per-fragment options or as options in stand-alone probe packets devoid of
fragment data; per-datagram options don't appear to be necessary. So far,
there has been no contrary evidence presented, despite my asking for some
(see
https://mailarchive.ietf.org/arch/msg/tsvwg/PQ779ildsmc7zzzxg0hMJGokfFM/).

This discussion has been helpful to me. In particular, tt confirms to me
that using the trailer format for per-datagram options in both unfragmented
and reassembled datagrams is not only compelling from a conceptual point of
view, the modularity of the design has practical advantages (the BITW/BITS
example described above being an example). I agree that we should not give
up those advantages to accommodate limited devices.

Mike Heard

---

From: Christian Huitema <[email-masked]>
Date: Tue, 12 Apr 2022 08:19:11 -0700

> Absent a pointer to a spec that requires such limits as they have, they are.

I am with Tom on that one. There is a lot of attention nowadays on 
preventing exfiltration of data from secure networks. Seems like any 
firewall worried about exfiltration may very well drop any packet in 
which UDP payload size and IP packet size don't match. That's a very 
plausible defense, and that would not be the first time such defenses 
get deployed, regardless of what the IETF says. It would be very 
interesting to measure whether such behavior is prevalent.

By the way, this is one of the reasons why I would really like to see a 
thorough development of the security considerations. If firewall 
managers do not understand the risks, their natural reaction will be to 
drop the packets.

> You yourself say that you don’t know how prevalent, though. And if they are prevalent, how does that affect the prevalence (or not) of Teredo extensions, that use trailers? We don’t know.

Joe, you should stop mentioning Teredo. We have established that Teredo 
packets just use UDP to carry a Teredo payload with no trailers. The 
Teredo packets themselves are the assembly of IPv6 packets and Teredo 
extensions, but that usage is not the same as the proposed Teredo 
options. So, no, when it comes to measuring the prevalence of packet 
filtering, Teredo is not a precedent.

> That’s partly true, but also true that so many devices have taken liberties with standards. Given how little information we can gather about such devices, it’s not clear we will ever know.
That's pretty much a fact of life in the Internet nowadays, see RFC9170 
for a discussion. It is entirely possible that UDP has ossified already. 
If we do not measure, we will not know.
> IMO, that’s vastly better than redesigning the approach for those devices at the expense of additional code, processing, and validation of implementations for other devices.

The draft is proposing a radical modification of a 40 years old 
protocol. The bar is going to be pretty high, both in terms of 
deployability and security.

-- Christian Huitema

---

From: Joe Touch <[email-masked]>
Date: Tue, 12 Apr 2022 08:44:05 -0700

> On Apr 12, 2022, at 8:20 AM, Christian Huitema <[email-masked]> wrote:
> 
> Joe, you should stop mentioning Teredo. We have established that Teredo packets just use UDP to carry a Teredo payload with no trailers. The Teredo packets themselves are the assembly of IPv6 packets and Teredo extensions, but that usage is not the same as the proposed Teredo options. So, no, when it comes to measuring the prevalence of packet filtering, Teredo is not a precedent.

I’m speaking of Teredos trailers, not UDP ones. 

Joe

---

From: Joe Touch <[email-masked]>
Date: Tue, 12 Apr 2022 09:00:14 -0700

> The draft is proposing a radical modification of a 40 years old protocol. The bar is going to be pretty high, both in terms of deployability and security.

The draft isn’t requiring anyone to use or implement it. Like UDP itself, it is unreliable; users need to check if it works and if it doesn’t, they are in the best position to know what to do anyway.

Some of the issues you raised in the name of security undermine the security we can add (transport authentication and encryption) and don’t really provide protection.  E.g., Letting through UDP packets just because they don’t have surplus space you think could be used for covert attacks (or pruning that space) just gives users a false sense that the same covert channel cannot occur in ip options or in user data that aren’t validated at the same checkpoint.

Joe

---

From: "C. M. Heard" <[email-masked]>
Date: Tue, 12 Apr 2022 09:17:40 -0700

On Tue, Apr 12, 2022 at 8:44 AM Joe Touch wrote:
> On Apr 12, 2022, at 8:20 AM, Christian Huitema wrote:
> > Joe, you should stop mentioning Teredo. We have established that
> > Teredo packets just use UDP to carry a Teredo payload with no
> > trailers. The Teredo packets themselves are the assembly of IPv6
> > packets and Teredo extensions, but that usage is not the same as
> > the proposed Teredo options. So, no, when it comes to measuring
> > the prevalence of packet filtering, Teredo is not a precedent.
>

Right. Just so that we do not talk past each other in the discussion,
it may be useful to note that there are two very distinct concerns
being raised against trailers:

1.) There is a concern that some devices likely to be used to terminate
tunnels cannot access the trailers to process them. This, as I
understand it, is Tom Herbert's concern. It applies equally to Teredo
trailer options and UDP trailer options. So in this context, it's fair
game to cite Teredo as a precedent for the use of trailer options.

2.) There is a concern that packets with UDP trailers -- i.e., where
the IP Payload Length exceeds the UDP Length -- will be filtered in
intermediate systems. If I understand correctly, this is the concern
(or at least one of the concerns) that Christian Huitema has raised.
The Aberdeen group has actually investigated this matter, and Tom Jones
presented the results at a MAPRG meeting several years ago. I'll search
my archives and provide appropriate pointers to that work in a bit.

Mike Heard

---

From: Joe Touch <[email-masked]>
Date: Tue, 12 Apr 2022 09:34:54 -0700

> On Apr 12, 2022, at 9:18 AM, C. M. Heard <[email-masked]> wrote:
> 

Right but:

- stripping UDP trailers provides no unique protection vs stripping IP options. Ie, if you think this is an issue, neither is reliable.

- UDP options are the only proposed way that allows fragments to pass through firewalls without flowID state (which is another covert channel, BTW). This *increases* traversal of devices, which should be balanced with concerns of it being stripped. 

- UDP options CAN always be stripped; we’re designing with the assumption that this is what legacy endpoints would do anyway. So users need to verify capabilities they rely upon. This is better than IP option use, which doesn’t tend to force users to make that assumption and check. 

The better way to think of this draft - and, for that matter, the TCP header extensions in TCPM, is “if you want this, here’s a way it can possibly work”. Nobody is being required to implement either extension. That means they’re both a lot more legacy friendly than most other changes to these essential protocols.  

Joe

---

From: Christian Huitema <[email-masked]>
Date: Tue, 12 Apr 2022 09:46:30 -0700

> game to cite Teredo as a precedent for the use of trailer options.

Teredo trailer options are processed by devices specifically upgraded to 
provide Teredo services, such as Teredo endpoints, relays and servers. 
Indeed these devices do support Teredo, that's what they are designed to 
do. That does not teach you much about the ability of generic tunnel 
servers to process UDP trailers. I don't think that the draft should 
mention Teredo at all.

> my archives and provide appropriate pointers to that work in a bit.

Thanks. Waiting to see the data.

-- Christian Huitema

---

From: Joe Touch <[email-masked]>
Date: Tue, 12 Apr 2022 10:36:40 -0700

> On Apr 12, 2022, at 9:47 AM, Christian Huitema <[email-masked]> wrote:
> 
> Teredo trailer options are processed by devices specifically upgraded to provide Teredo services, such as Teredo endpoints, relays and servers. Indeed these devices do support Teredo, that's what they are designed to do.

Everything said about Teredo her applies to UDP options. 

> That does not teach you much about the ability of generic tunnel servers to process UDP trailers.

 The issue that has been on this thread for a while is routers with hardware constraints that would impede their ability to be upgraded to support trailer options.  

> I don't think that the draft should mention Teredo at all.

Teredo trailer option support implies potential for UDP trailer option support at routers for tunnels. 

Ie, you wanted us to address potential for network device traversal, and Teredo is relevant to that issue. 

Joe

---

From: "C. M. Heard" <[email-masked]>
Date: Tue, 12 Apr 2022 12:20:15 -0700

...
> > There is a concern that packets with UDP trailers -- i.e., where
> > The Aberdeen group has actually investigated this matter, and Tom
> > Jones presented the results at a MAPRG meeting several years ago.
> > I'll search my archives and provide appropriate pointers to that
> > work in a bit.
>

The 2018 MAPRG presentation authored by Gorry Fairhurst, Tom Jones, and
Raffaele Zullo is available at:

https://datatracker.ietf.org/meeting/103/materials/slides-103-maprg-a-tale-of-two-checksums-tom-jones

There is a more recent published paper by Zullo, Jones, and Fairhurst
resulting from this work; it is available at:

https://tma.ifip.org/2020/wp-content/uploads/sites/9/2020/06/tma2020-camera-paper70.pdf

The principal finding was that UDP options, as originally envisioned,
were blocked quite often by middleboxes re-computing the UDP checksum
as if the options trailer was part of the UDP packet -- in other words,
using the IP Payload Length instead of the UDP Length both to control
the extent of the UDP checksum calculation and as the the length value
used in the pseudo-header. This was the motivation for the way OCS is
specified. It compensates for this middlebox pathology, and in addition,
it happens to make commonly used forms of checksum offload work as
intended in the presence of UDP options -- see
https://mailarchive.ietf.org/arch/msg/tsvwg/9_1YWp-TApI9mcCuia8U6jfCwTA/.

However, the work also considered many other kinds of middlebox
pathologies, including refusal to pass anything that does not have
IP Payload Length equal to UDP Length, passing the surplus area only
if it contained nothing but zeroes, and a couple of other checksum
re-calculation anomalies.

I just became aware of the paper and have yet to digest it, but back
in November 2018 I put several questions to Tom Jones regarding the
interpretation of the figures in the MAPRG presentation. He has
graciously permitted me to forward his response to those questions,
in addition to providing the link to the published paper.

---------- Forwarded message ---------
From: Tom Jones <[email-masked]>
Date: Mon, Nov 26, 2018 at 5:19 AM
Subject: Re: Questions about "A Tale of Two Checksums"
To: C. M. Heard <[email-masked]>
Cc: <[email-masked]>, <[email-masked]>

Hi Mike,

Thanks for your questions and feedback on the TSVWG mailing list.

> A few questions about the IETF 103 maprg presentation "A Tale of Two
> Checksums" came up when reviewing the slides at https://datatracker.ietf
> .org/meeting/103/materials/slides-103-maprg-a-tale-of-two-checksums-tom-
> jones. I'll put the questions here, but If there us a writeup available
> that will answer these questions I'd be happy to have a pointer to it.

We are writing a paper on these measurements and will share it with the
list when it is published.

> On page 10:
>
>    1. Does "Works" mean UDP Length Checksum, UDP length Pseudoheader,
>    surplus area passed transparently?

Yes, UDP datagrams with Options work end to end on that path. (this
includes the correct checksum and when no checksum validation is done on
the path)

>    2. Does "Full Payload Checksum" mean Full Payload Checksum, IP length
>    Pseudoheader, surplus area passed transparently?

Figure 1 in the CCO draft will help.

https://tools.ietf.org/html/draft-fairhurst-udp-options-cco-00

The Full Payload Checksum is calculated over the number of bytes as
described by the IP Payload length. That is, it is calculated over the
UDP Datagram and any surplus space (the option area).

IP Length Pseudoheader here means that the UDP Pseudeoheader constructed
for the checksum uses the length from the IP Payload length and not the
one in the UDP header.

>    3. It seems that "Full Payload Checksum, UDP length Pseudoheader" would
>    get typically through if the options space is populated by zeroes
(since
>    the checksum computation would yield the same value as UDP Length
>    Checksum, UDP length Pseudoheader). How is "Only passes 0s as options
>    space" distinguished from this case?

If zero padded works and 3rd CS fails, it is a problem with the options
space. If 3rd CS passes, zero padded will always work.

If zero padded fails, 3rd CS passes, it falls into the 'other' category.
This is a very small number of cases.

>
> On pages 8+12 and 17-18:
>
>    1. Is the correct reading of these charts that blue = what passes
>    without CCO, blue+red = what passes with CCO?
>

Yes

> On pages 15-16 and 19-20:
>
>    1. In general, do the various colours slices on the graphs indicate the
>    fractions of packets _with options_ (IP payload length <> UDP Length)
>    passed for the various checksum options listed in the key on the
>    following page?

Slides 15-16 and 22 are the same slide, the key was moved to make it
easier to discuss when presenting.

It is the same case for 19-20 and 21.

>    2. Does "No CS" mean "no packets with options passed" (indicating a
path
>    that require IP payload length == UDP Length)?

Yes

>    3. Is "4th CS" the same as "UDP Length Checksum, IP length
Pseudoheader"
>    on p. 10?

Yes

>    4. Is "3rd CS" the same "Full Payload Checksum, UDP length
Pseudoheader"
>    on page 10? If so, is there an additional proviso that the option
space is
>    non-zero?

Yes

>    5.  Does "Zero-Padded Options only" mean the same thing as "Only passes
>    0s as options space" on p. 10? If so, were any checksum combinations
>    other than "Correct UDP CS" (i.e., UDP Length Checksum, UDP length
>    Pseudoheader) included in this test case?

Yes, no other combinations were included.

>    6. Does the colour red indicate paths for which "Full Payload Checksum,
>    IP length Pseudoheader" was the only one that worked for packets
without
>    CCO?

Yes

>    7. Does the colour yellow indicate paths for which both "Correct UDP
CS"
>    and "Full IP Payload CS" worked in the absence of CCO? That this
>    combination was so prevalent seems to indicate that many middleboxes
do two
>    checksum computations!

Yes. The two computations can also happen at different layers, in that case
we get an OR between them.

> Thanks for the work. It was very enlightening, if somewhat discouraging,
> given that there are drop rates in the 5% - 28% range even with CCO.
>
> Mike Heard

- Tom and Raf
------------------------------------------------------------------------

A quick scan of the published paper suggests that many of the questions
I asked -- as well as others -- are resolved there. I shall be reading
it carefully in the next few days. That being said, there is one special
case that deserves to be singled out that, as far as I have seen, is
mot explicitly addressed: what happens when there is no conventional
UDP data at all, i.e., when UDP Length = 8? This case figures
prominently in the current specification, since it is used for
fragmentation, options that cannot be safely ignored, and packets that
carry options with no user data, such as DPLPMTUD probe packets. If
tests were conducted for that case, I at least would find the results
to be of great interest.

Mike Heard