Subject: Re: [tcpm] [EXTERNAL] Re: Seeking WG opinions on ACKing ACKs with good cause

From: Bob Briscoe <[email-masked]>
Date: Sat, 10 Jul 2021 22:13:44 +0100

Yoshi, Richard, Mirja, tcpm list,

We came to a good consensus in late March on the conditions to place 
around ACKing ACKs in order to keep congestion feedback flowing, but to 
ensure ping-pong ACKs are rapidly damped as well as avoiding false 
DupACK detection. I have written that up in a rev of the draft, which I 
will post separately so as not to distract from the question I have here...
...While writing up, I realized we had glossed over a possibly more 
important point, when we proposed to change the number in the basic rule 
from 2 to 3, as follows:

       An AccECN Data Receiver MUST immediately send an ACK once 'n'
       CE marks have arrived since the previous ACK, where 'n' SHOULD
       be 3 and MUST be in the range 3 to 6 inclusive.

I'd like to suggest an alternative:

       An AccECN Data Receiver MUST immediately send an ACK once 'n'
       CE marks have arrived since the previous ACK. If there is new
       data to acknowledge, 'n' SHOULD be 2.  If there is no new data
       to acknowledge, 'n' SHOULD be 3 and MUST be no less than 3.
       In either case, 'n' MUST be no greater than 6.

Rationale: The data ACKs case shouldn't be compromised by the ACKs of 
ACKs case.

When we were originally only thinking of the data ACKs case (before we 
realized this rule might trigger ACKs of ACKs), we recommended n=2  
because it ensures congestion information is kept fresh, and during runs 
of congestion it generates more ACKs, which might make it more likely 
that at least one ACK survives some types of coalescing before the ACE 
field wraps. Also 2 is the default for the equivalent repetition of 
DCTCP feedback.

Then, we noticed the ACKs of ACKs case, and we wanted to damp any ACK 
ping-pong, so we recommended 3. Without really discussing the pros and 
cons, we extended 3 to all cases (both data ACKs and ACKs of ACKs). I 
suspect the main reason was that it is just simpler to have a single 
rule. But these two cases are likely to be controlled from different 
parts of the code anyway.

I've also realized that, in the data ACKs case, it was wrong to set a 
lower bound on 'n', let alone a /mandatory/ lower bound. There might be 
scenarios where it makes sense to trigger an ACK every CE mark; we have 
no reason to stop implementers doing that if they want. A lower bound 
could even be perversely read to mean that you're not allowed to 
immediately ACK data until you've received 'n' CE marks.

Thoughts?

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Christian Huitema <[email-masked]>
Date: Sat, 10 Jul 2021 22:38:21 -0700

On 7/10/2021 2:13 PM, Bob Briscoe wrote:
> around ACKing ACKs in order to keep congestion feedback flowing, but 
> to ensure ping-pong ACKs are rapidly damped as well as avoiding false 
> DupACK detection. I have written that up in a rev of the draft, which 
> I will post separately so as not to distract from the question I have 
> here...
> important point, when we proposed to change the number in the basic 
> rule from 2 to 3, as follows:
>
> because it ensures congestion information is kept fresh, and during 
> runs of congestion it generates more ACKs, which might make it more 
> likely that at least one ACK survives some types of coalescing before 
> the ACE field wraps. Also 2 is the default for the equivalent 
> repetition of DCTCP feedback.
>
> lower bound on 'n', let alone a /mandatory/ lower bound. There might 
> be scenarios where it makes sense to trigger an ACK every CE mark; we 
> have no reason to stop implementers doing that if they want. A lower 
> bound could even be perversely read to mean that you're not allowed to 

I am looking at this from the QUIC point of view. In general, frequent 
ACKs provide better control, but there are two points of tension: 
managing low bandwidth return paths; and, reducing the impact of ACK on 
performance.

There are a number of scenarios in which the return path used by ACKs 
has a much lower bandwidth than the data path. If ACKs are two frequent, 
the return path can become congested. ACKing every two or 3 packets is 
only sustainable if the return path bandwidth is not too small compared 
to the data path -- 1/10th is generally sustainable, but lower that 
1/20th and the ACKs have to be spaced. The alternative to spacing would 
be increased RTT due to congestion on the return path, and random losses 
of ACKs. Of course, even with ACK spacing, there are limits to 
asymmetry: control becomes loose if the transport does not receive 
multiple ACKs per RTT.

ACK frequency has been found a key factor in the performance of QUIC 
implementations. For example, implementations may achieve 5Gbps with 
spaced ACK, but the performance drops to a few 100 Mbps if we insist on 
an ACK for 2 packets. High performance requires sending packets in 
batches, using API like UDP GSO, and performance seems best when the 
frequency of ACKs more or less matches the frequency of batches.

-- Christian Huitema

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 11 Jul 2021 12:43:11 +0100

Christian,

On 11/07/2021 06:38, Christian Huitema wrote:
>> When we were originally only thinking of the data ACKs case (before 
>> we realized this rule might trigger ACKs of ACKs), we recommended 
>> n=2  because it ensures congestion information is kept fresh, and 
>> during runs of congestion it generates more ACKs, which might make it 
>> more likely that at least one ACK survives some types of coalescing 
>> before the ACE field wraps. Also 2 is the default for the equivalent 
>> ping-pong, so we recommended 3. Without really discussing the pros 
>> and cons, we extended 3 to all cases (both data ACKs and ACKs of 
>> ACKs). I suspect the main reason was that it is just simpler to have 
>> a single rule. But these two cases are likely to be controlled from 
>> different parts of the code anyway.
>>
>> bound could even be perversely read to mean that you're not allowed 
>> to immediately ACK data until you've received 'n' CE marks.
>>
>> Thoughts? 
>
> [CH] I am looking at this from the QUIC point of view. In general, 
> frequent ACKs provide better control, but there are two points of 
> tension: managing low bandwidth return paths; and, reducing the impact 
> of ACK on performance.
>
> has a much lower bandwidth than the data path. If ACKs are two 
> frequent, the return path can become congested. ACKing every two or 3 
> packets is only sustainable if the return path bandwidth is not too 
> small compared to the data path -- 1/10th is generally sustainable, 
> but lower that 1/20th and the ACKs have to be spaced. The alternative 
> to spacing would be increased RTT due to congestion on the return 
> path, and random losses of ACKs. Of course, even with ACK spacing, 
> there are limits to asymmetry: control becomes loose if the transport 
> does not receive multiple ACKs per RTT.
>
> spaced ACK, but the performance drops to a few 100 Mbps if we insist 
> on an ACK for 2 packets. High performance requires sending packets in 

[BB] I think a brief summary of AccECN might be in order here - it seems 
like you're starting from scratch, but we're aware of all the above.

The implementation will have its own ACK ratio - that's out of scope of 
AccECN, except to set this max of 6 CEs, which is to mitigate wrap of 
the 3-bit counter of CE-marks (which in the worst case of 100% marking 
in the data direction could then induce 1 ACK per 6 packets). This 
shouldn't limit forward performance, because it only increases the 
reverse ACK rate if there is heavy congestion in the forward path, when 
the Data Sender should be reducing the forward rate anyway.

AccECN can also use the AccECN TCP Option that provides a much larger 
counter (24-bit), but AccECN has to work reasonably well with just the 
3-bit counter in the main TCP header in case the option can't traverse 
the path. However, the Data Receiver doesn't know whether the option is 
reaching the other end, so it has to assume it might not be.

My original question asks what the recommended value of 'n' should be 
for how much the counter of CE-marked data packets should be allowed to 
increment between ACKs. We don't have to recommend anything, but it 
helps implementers if we do. AccECN is TCP only. So when we originally 
recommended 2, we figured that TCP's default ACK ratio is 2. So, even if 
ECN marking in the forward direction was 100%, a default increment of 2 
CE marked packets ('n') wouldn't cause any more frequent ACKs than the 
default.

Nonetheless, if the implementer has chosen a longer ACK ratio (or 
perhaps allowed the ratio to be controlled by the sender e.g. 
draft-gomez-tcpm-ack-rate-request) they don't have to use the 
recommended n=2; the proposed wording allows a range.

Don't worry, we certainly wouldn't "insist on an ACK for 2 packets".

Also, FYI, AccECN provides feedback about ECN marking on the ACK stream, 
back to the Data Receiver who is generating the ACKs. It then has a hook 
on which it can hang feedback control of TCP's ACK stream without 
waiting for anyone else to deploy anything (it's up to the implementer 
or the IETF to add this if they want). However, the scope of the AccECN 
spec is deliberately wire protocol only, so it doesn't say what to do 
with this feedback.

Cheers

Bob

---

From: Christian Huitema <[email-masked]>
Date: Sun, 11 Jul 2021 08:54:37 -0700

> On Jul 11, 2021, at 4:42 AM, Bob Briscoe <[email-masked]> wrote:
> 
> Christian,
> 
>>> We came to a good consensus in late March on the conditions to place around ACKing ACKs in order to keep congestion feedback flowing, but to ensure ping-pong ACKs are rapidly damped as well as avoiding false DupACK detection. I have written that up in a rev of the draft, which I will post separately so as not to distract from the question I have here...
>>> ...While writing up, I realized we had glossed over a possibly more important point, when we proposed to change the number in the basic rule from 2 to 3, as follows:
>>> 
>>> Rationale: The data ACKs case shouldn't be compromised by the ACKs of ACKs case.
>>> 
>>> When we were originally only thinking of the data ACKs case (before we realized this rule might trigger ACKs of ACKs), we recommended n=2  because it ensures congestion information is kept fresh, and during runs of congestion it generates more ACKs, which might make it more likely that at least one ACK survives some types of coalescing before the ACE field wraps. Also 2 is the default for the equivalent repetition of DCTCP feedback.
>>> 
>>> Then, we noticed the ACKs of ACKs case, and we wanted to damp any ACK ping-pong, so we recommended 3. Without really discussing the pros and cons, we extended 3 to all cases (both data ACKs and ACKs of ACKs). I suspect the main reason was that it is just simpler to have a single rule. But these two cases are likely to be controlled from different parts of the code anyway.
>>> 
>>> I've also realized that, in the data ACKs case, it was wrong to set a lower bound on 'n', let alone a /mandatory/ lower bound. There might be scenarios where it makes sense to trigger an ACK every CE mark; we have no reason to stop implementers doing that if they want. A lower bound could even be perversely read to mean that you're not allowed to immediately ACK data until you've received 'n' CE marks.
>>> 
>>> Thoughts? 
>> 
>> [CH] I am looking at this from the QUIC point of view. In general, frequent ACKs provide better control, but there are two points of tension: managing low bandwidth return paths; and, reducing the impact of ACK on performance.
>> 
>> There are a number of scenarios in which the return path used by ACKs has a much lower bandwidth than the data path. If ACKs are two frequent, the return path can become congested. ACKing every two or 3 packets is only sustainable if the return path bandwidth is not too small compared to the data path -- 1/10th is generally sustainable, but lower that 1/20th and the ACKs have to be spaced. The alternative to spacing would be increased RTT due to congestion on the return path, and random losses of ACKs. Of course, even with ACK spacing, there are limits to asymmetry: control becomes loose if the transport does not receive multiple ACKs per RTT.
>> 
>> ACK frequency has been found a key factor in the performance of QUIC implementations. For example, implementations may achieve 5Gbps with spaced ACK, but the performance drops to a few 100 Mbps if we insist on an ACK for 2 packets. High performance requires sending packets in batches, using API like UDP GSO, and performance seems best when the frequency of ACKs more or less matches the frequency of batches.
> 
> [BB] I think a brief summary of AccECN might be in order here - it seems like you're starting from scratch, but we're aware of all the above.
> 
> The implementation will have its own ACK ratio - that's out of scope of AccECN, except to set this max of 6 CEs, which is to mitigate wrap of the 3-bit counter of CE-marks (which in the worst case of 100% marking in the data direction could then induce 1 ACK per 6 packets). This shouldn't limit forward performance, because it only increases the reverse ACK rate if there is heavy congestion in the forward path, when the Data Sender should be reducing the forward rate anyway.

1 ACK per 6 packets would cause performance issues on high speed links. Common setup is "4 to 8 ACK per RTT", which means intervals much larger than 6 packets. 

> 
> AccECN can also use the AccECN TCP Option that provides a much larger counter (24-bit), but AccECN has to work reasonably well with just the 3-bit counter in the main TCP header in case the option can't traverse the path. However, the Data Receiver doesn't know whether the option is reaching the other end, so it has to assume it might not be.

In QUIC, the ECN counters are 62 bits, so I guess the guidance can be much looser.

> 
> My original question asks what the recommended value of 'n' should be for how much the counter of CE-marked data packets should be allowed to increment between ACKs. We don't have to recommend anything, but it helps implementers if we do. AccECN is TCP only. So when we originally recommended 2, we figured that TCP's default ACK ratio is 2. So, even if ECN marking in the forward direction was 100%, a default increment of 2 CE marked packets ('n') wouldn't cause any more frequent ACKs than the default.

The default ACK ratio in TCP is one thing, the practical ratio is another. On many paths, middleboxes perform ACK thinning, pruning ACKs to improve performance. A three bit counter is not very robust against such thinning. To be robust, the counter should be wide enough to accumulate an RTT worth of ECN marks. 

> 
> Nonetheless, if the implementer has chosen a longer ACK ratio (or perhaps allowed the ratio to be controlled by the sender e.g. draft-gomez-tcpm-ack-rate-request) they don't have to use the recommended n=2; the proposed wording allows a range.
> 
> Also, FYI, AccECN provides feedback about ECN marking on the ACK stream, back to the Data Receiver who is generating the ACKs. It then has a hook on which it can hang feedback control of TCP's ACK stream without waiting for anyone else to deploy anything (it's up to the implementer or the IETF to add this if they want). However, the scope of the AccECN spec is deliberately wire protocol only, so it doesn't say what to do with this feedback.

Yes, using ECN on the ACK stream has potential. There has been very little research on congestion control for ACK. I wish there would be more. 

-- Christian Huitema

---

From: Neal Cardwell <[email-masked]>
Date: Sun, 11 Jul 2021 14:28:28 -0400

> > On Jul 11, 2021, at 4:42 AM, Bob Briscoe <[email-masked]> wrote:
...

> AccECN, except to set this max of 6 CEs, which is to mitigate wrap of the
> 3-bit counter of CE-marks (which in the worst case of 100% marking in the
> data direction could then induce 1 ACK per 6 packets). This shouldn't limit
> forward performance, because it only increases the reverse ACK rate if
> there is heavy congestion in the forward path, when the Data Sender should
> be reducing the forward rate anyway.
>
> 1 ACK per 6 packets would cause performance issues on high speed links.
> Common setup is "4 to 8 ACK per RTT", which means intervals much larger
> than 6 packets.

I share this concern about the AccECN requirement of one ACK per 6
CE-marked data segments potentially causing performance issues with
high-speed links. Today, with most high-speed last-hop link technologies
(wifi, cellular, Ethernet) TCP receivers often receive (from lower layers
of the hardware and software networking stack) large aggregates  (often up
to 64KBytes or around 44 packets), and generate a single ACK for that
aggregate. AFAICT changing this scenario to require 1 packet per 6
CE-marked data segments means that during CE-marked scenarios receiving
such an aggregate would require ceil(44/6) = 8 ACKs, increasing the number
of ACKs by up to 8x.

I have two main concerns in that scenario:

(1) CPU load: This seems like it would impose much higher CPU load on the
data receiver, at just the moment when the receiver may already be under
stress due to receiving data near the maximum rate of the link.

(2) Congestion and ACK loss: This requirement for generating up to 8 ACKs
per aggregate seems like it is likely to produce a tight burst of 8 ACKs,
which is going to increase congestion and increase the odds of losing at
least one ACK, which is going to cause accuracy problems for "Accurate"
ECN. :-)

I understand that this proposed "ACK per 6 CE-marked data segments" rule is
necessary to avoid issues with the 3-bit ACE field wrapping. So IMHO this
is one of the good arguments against including the ACE field in the AccECN
design.

The other main concerns I have with the ACE field are:

o Complexity

o Redundancy with respect to the AccECN counter options

o Potential problems with middleboxes

o Known problems with NICs and drivers (based on Ilpo's nice talk, "Accurate
ECN Linux Implementation: Experiences and Challenges", at the April 2020
TCPM interim meeting)

I realize that if AccECN does not have the ACE field feature, then AccECN
and TCP L4S will not be usable on paths with middleboxes that strip the AccECN
counter options. But IMHO living without the ACE field is preferable. IMHO
it's acceptable to say that L4S can only be used with (a) QUIC, or (b) TCP
connections where no middleboxes are stripping AccECN options.

cheers,
neal

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 12 Jul 2021 13:30:17 +0100

Neal,

On 11/07/2021 19:28, Neal Cardwell wrote:
>
>
> On Sun, Jul 11, 2021 at 11:55 AM Christian Huitema 
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>
>
>     > On Jul 11, 2021, at 4:42 AM, Bob Briscoe <[email-masked]
>     <mailto:[email-masked]>> wrote:
>
> ...
>
>     > The implementation will have its own ACK ratio - that's out of
>     scope of AccECN, except to set this max of 6 CEs, which is to
>     mitigate wrap of the 3-bit counter of CE-marks (which in the worst
>     case of 100% marking in the data direction could then induce 1 ACK
>     per 6 packets). This shouldn't limit forward performance, because
>     it only increases the reverse ACK rate if there is heavy
>     congestion in the forward path, when the Data Sender should be
>     reducing the forward rate anyway.
>
>     1 ACK per 6 packets would cause performance issues on high speed
>     links. Common setup is "4 to 8 ACK per RTT", which means intervals
>     much larger than 6 packets.
>
>
> high-speed links. Today, with most high-speed last-hop link 
> technologies (wifi, cellular, Ethernet) TCP receivers often receive 
> (from lower layers of the hardware and software networking stack) 
> large aggregates  (often up to 64KBytes or around 44 packets), and 
> generate a single ACK for that aggregate. AFAICT changing this 
> scenario to require 1 packet per 6 CE-marked data segments means that 
> during CE-marked scenarios receiving such an aggregate would require 
> ceil(44/6) = 8 ACKs, increasing the number of ACKs by up to 8x.
>
> (1) CPU load: This seems like it would impose much higher CPU load on 
> the data receiver, at just the moment when the receiver may already be 
> under stress due to receiving data near the maximum rate of the link.
>
> (2) Congestion and ACK loss: This requirement for generating up to 8 
> ACKs per aggregate seems like it is likely to produce a tight burst of 
> 8 ACKs, which is going to increase congestion and increase the odds of 
> losing at least one ACK, which is going to cause accuracy problems for 
> "Accurate" ECN. :-)

[BB] I hadn't intended this wording to apply in response to a large 
burst. I'd want to alter the wording to allow just one ACK in this case.

>
> I understand that this proposed "ACK per 6 CE-marked data segments" 
> rule is necessary to avoid issues with the 3-bit ACE field wrapping. 
> So IMHO this is one of the good arguments against including the ACE 
> field in the AccECN design.

[BB] I think the subtext here is a preference for the DCTCP style of 
feedback vs ACE (when there is no AccECN TCP Option). I don't think this 
particular issue is any different between the two.

With DCTCP feedback, when a large data burst like this arrives, if there 
are transitions to and from CE marking within the burst, does Linux 
DCTCP generate an ACK for each transition, and for every n repetitions 
of CE not ECT, like the RFC says it should? Or does the implementation 
just pop out one ACK at the end?

Ilpo & I have been doing experiments with high levels of ACK coalescing 
causing the ACE field seen by the Data Sender to wrap multiple times 
under high congestion. For example, 1 ACK per ~4ms is one common scheme, 
which would result in about 1 ACK for 34 data packets at just 100Mb/s.

> The other main concerns I have with the ACE field are:

[BB] I understand there could be a tradeoff between complexity and 
accuracy. But I think the complexity of ACE and DCTCP feedback are 
pretty much the same but, with even low levels of ACK coalescing, the 
accuracy of ACE is superior. Nonetheless, I understand that deploying 
something different to DCTCP when you've already got DCTCP could involve 
deployment complexity.

> o Redundancy with respect to the AccECN counter options

[BB] The redundancy is intended. For cases where the TCP Option doesn't 
traverse the path, or where TCP option space is limited.

> o Potential problems with middleboxes

[BB] Well, we've tested the 3 header bits over millions of paths without 
problems. But yes, there are billions more to test.

>
> o Known problems with NICs and drivers (based on Ilpo's nice talk, 
> "Accurate ECN Linux Implementation: Experiences and Challenges", at 
> the April 2020 TCPM interim meeting)

[BB] I understood that talk as concluding there weren't really problems. 
What specific problem do you have in mind?

> I realize that if AccECN does not have the ACE field feature, then 
> AccECN and TCP L4S will not be usable on paths with middleboxes that 
> strip the AccECN counter options. But IMHO living without the ACE 
> field is preferable. IMHO it's acceptable to say that L4S can only be 
> used with (a) QUIC, or (b) TCP connections where no middleboxes are 
> stripping AccECN options.

[BB] When you say 'living without the ACE field', alongside the AccECN 
TCP Option, would you leave classic ECN feedback? Or put DCTCP feedback 
in its place?

I don't think there's any case for using classic ECN feedback within 
AccECN. I see the competition as between ACE and DCTCP feedback, each 
optionally with the AccECN TCP Option.

If DCTCP feedback were a stop-gap until we could get good traversal of 
the TCP Option, it might have some merit, but I think DCTCP isn't going 
to work well with the level of ACK coalescing in the Internet. However, 
this is very difficult to judge unless we do large scale A-B experiments 
on the real Internet (and potentially within DCs). I'll think further 
about enabling some way to do that, possibly within AccECN's negotiation 
framework. But today, I have to prioritize for the draft deadline.

Cheers

Bob

---

From: "Ilpo Järvinen" <[email-masked]>
Date: Mon, 12 Jul 2021 17:31:41 +0300 (EEST)

Hi Neal,

I'm trying to understand your scenarios and I fail to see consistency here 
and there (which makes me wonder if I got you right at all).

On Sun, 11 Jul 2021, Neal Cardwell wrote:
>       mitigate wrap of the 3-bit counter of CE-marks (which in the
>       worst case of 100% marking in the data direction could then
>       induce 1 ACK per 6 packets). This shouldn't limit forward
>       performance, because it only increases the reverse ACK rate if
>       there is heavy congestion in the forward path, when the Data
>       Sender should be reducing the forward rate anyway.
> 
>       links. Common setup is "4 to 8 ACK per RTT", which means
>       intervals much larger than 6 packets.
> 
> 
> I share this concern about the AccECN requirement of one ACK per 6 CE-marked
> data segments potentially causing performance issues with high-speed links.
> Today, with most high-speed last-hop link technologies (wifi, cellular,
> Ethernet) TCP receivers often receive (from lower layers of the hardware and
> software networking stack) large aggregates  (often up to 64KBytes or around
> 44 packets), and generate a single ACK for that aggregate. AFAICT changing
> this scenario to require 1 packet per 6 CE-marked data segments means that

Those would be back-to-back ACKs in response to a 44 seg blob? At that 
point with warm caches & context, essentially just duplicating the same 
ACK with a few fields varied a little? Is "much higher" perhaps an 
overstatement?

I'd also tend to think link and CPU are not bottleneck at the same time 
(but I admit I might have dealt with too idealized environments to 
experience that in practice).

This next doesn't relate directly to your concern about CPU load but only
to effect of extra ACK on link performance. I remember discussing with 
Yuchung (perhaps you were there at that time too) some years back in some 
IETF about making more ACKs appear on the wire using some kind of ACK 
sending offloading (back then, main motivation for me was to increase SACK 
reduncancy). Yuchung told me that it would be really bad for things like 
Wifi. I now wonder if it really relates more to the cases where ACKs are 
roughly evenly spread out over the RTT preventing efficient use of the 
wireless channel or does that apply also to cases where they're very 
unevenly spread (back-to-back)?

And for the record, sending x ACKs was not the way I interpreted the
particular draft text. I took it just a rule when at latest, the ACK
should be generated. I just took it as "Since we got 44 CE marks, we
MUST send ACK now". (It seems to have be slightly differently worded
in -09 where my interpretation originates from).

> least one ACK, which is going to cause accuracy problems for "Accurate" ECN.
> :-)

So instead of 44 CE marks, 44 or 36 are detected (depending on how the 
sender treats safe/unsafe on that ACK after the loss)?

To me more challenging are the cases where ACE delta is zero but a wrap 
has occurred because zero usually means there's no CE event going on 
at all.

> necessary to avoid issues with the 3-bit ACE field wrapping. So IMHO this is
> one of the good arguments against including the ACE field in the AccECN
> design. 
> 

I hope you don't really mean ACE field itself is complex? Most of the 
complexity related to the header bits is due to the negotiation.

> o Redundancy with respect to the AccECN counter options

I ended up using ACE field to improve byte counter estimation between 
AccECN option arrivals when AccECN option is not put into every packet.
It makes the corrections to sender's estimation of CE (delivered_ce) 
smoother as the option eventually arrives. It is of course not necessary
if AccECN option is always present.

> ECN Linux Implementation: Experiences and Challenges", at the April 2020

Are you saying that the offloading engine don't have a toggle for the
CWR clearing logic? (I didn't claim to know).

I've since then thought more about the offloading cases.
For sending, the headers remain the same so the feedback doesn't matter.
On receiving, I first looked into GRO expecting some impact, however, it 
turns out that advancing ACK sequence triggers a flush. As no cumulative 
ACKs get combined, ACE field changes did not trigger additional flushing 
in practice.

What I don't know, is if HW has ACK receiver offloading which combines ACK 
with different ACK sequence numbers (in the end, this works same as ACK 
coalescing). That might be the only case where clearly more ACKs would 
be passed upwards to processing when ACE field changes for a continuous 
run of CEs. In randomly spread CEs case, however, DCTCP feedback produces 
edges that is roughly similar in number so it's hard to see how ACE field 
would result in unbearable number of ACKs to process compared with DCTCP 
feedback.

> and TCP L4S will not be usable on paths with middleboxes that strip the
> AccECN counter options. But IMHO living without the ACE field is preferable.
> IMHO it's acceptable to say that L4S can only be used with (a) QUIC, or (b)
> TCP connections where no middleboxes are stripping AccECN options.

This is just trading off "potential problems with middleboxes" with 
other potential problems with middleboxes.

I'm not saying here AccECN with option isn't good but it seems that you 
say you have concern x on ACE and then accept AccECN option as the 
solution but it has rather similar challenges related to x.

-- 
 i.

---

From: Neal Cardwell <[email-masked]>
Date: Mon, 12 Jul 2021 10:35:38 -0400

On Mon, Jul 12, 2021 at 8:30 AM Bob Briscoe <[email-masked]> wrote:

> [BB] I hadn't intended this wording to apply in response to a large burst.
> I'd want to alter the wording to allow just one ACK in this case.
>

In the case where there is just one ACK for a big CE-marked burst of >= 8
segments, does that mean the result is:

(a) the ACE field loses accuracy, or

(b) the TCP connection interpreting the ACE field should presume the ACE
field probably wrapped and estimate the number of CE-marked segments using
the number of SACKed/ACKed segments, rather than the ACE field?

> I understand that this proposed "ACK per 6 CE-marked data segments" rule
> is necessary to avoid issues with the 3-bit ACE field wrapping. So IMHO
> this is one of the good arguments against including the ACE field in the
> AccECN design.
>
>

Yes, when the AccECN TCP Option is not present, I'd prefer DCTCP-style
feedback rather than the ACE field.

> are transitions to and from CE marking within the burst, does Linux DCTCP
> generate an ACK for each transition, and for every n repetitions of CE not
> ECT, like the RFC says it should? Or does the implementation just pop out
> one ACK at the end?
>

Yes, with Linux DCTCP (or BBRv2), when there is a big burst, there is an
ACK for each CE<->non-CE transition. They also handle the case where there
is a CE<->non-CE transition while a delayed ACK is pending. This requires
cooperation from a few different layers.

(1) the NIC hardware LRO aggregation is supposed to not aggregate across
CE<->non-CE transitions. I believe the de facto spec on NIC aggregation
with respect to IP fields is the following, which mandates that:

https://docs.microsoft.com/en-us/windows-hardware/drivers/network/updating-the-ip-headers-for-coalesced-segments

(2) the Linux software GRO aggregation does not aggregate across
CE<->non-CE transitions; see inet_gro_receive() checking the  ToS bytes:
(iph->tos ^ iph2->tos)

(3) the DCTCP and BBRv2 code use dctcp_ece_ack_update() to handle the case
where there is a CE<->non-CE transition while a delayed ACK is pending

> causing the ACE field seen by the Data Sender to wrap multiple times under
> high congestion. For example, 1 ACK per ~4ms is one common scheme, which
> would result in about 1 ACK for 34 data packets at just 100Mb/s.
>

Great. Yes, certainly that level of aggregation is important to test.

> accuracy. But I think the complexity of ACE and DCTCP feedback are pretty
> much the same but, with even low levels of ACK coalescing, the accuracy of
> ACE is superior. Nonetheless, I understand that deploying something
> different to DCTCP when you've already got DCTCP could involve deployment
> complexity.
>

IMHO the ACE field is more complex, due to:

o context-dependent interpretation of the header bits used by ACE

o logic to handle wraps in the ACE field and estimate what the real
increment to the ACE field was in the case of possible ACK loss

o interactions with drivers or hardware that already have assumptions about
the header bits used for ACE (e.g. that the CWR bit should be cleared for
the first N-1 segments in a TSO burst and only set on the last segment of
the TSO burst)

> traverse the path, or where TCP option space is limited.

I realize the redundancy is intentional, but the redundant code does impose
a maintenance cost.

> What specific problem do you have in mind?

Mainly I'm concerned about:

o interactions with drivers or hardware that already have assumptions about
the header bits used for ACE (e.g. that the CWR bit should be cleared for
the first N-1 segments in a TSO burst and only set on the last segment of
the TSO burst)

It sounds like since the ACE field redefines the semantics of some
long-standing TCP header fields, there will be a long trial-and-error
period of finding the drivers and NICs that are incompatible with the ACE
field, and working around those.

> [BB] When you say 'living without the ACE field', alongside the AccECN TCP
> Option, would you leave classic ECN feedback? Or put DCTCP feedback in its
> place?
>

When 'living without the ACE field', I mean either using the AccECN TCP
Option or DCTCP-style feedback.

> optionally with the AccECN TCP Option.

Agreed. I agree there's no case for using classic ECN feedback within
AccECN.

> If DCTCP feedback were a stop-gap until we could get good traversal of the
> TCP Option, it might have some merit, but I think DCTCP isn't going to work
> well with the level of ACK coalescing in the Internet. However, this is
> very difficult to judge unless we do large scale A-B experiments on the
> real Internet (and potentially within DCs). I'll think further about
> enabling some way to do that, possibly within AccECN's negotiation

I agree it's unclear whether DCTCP-style feedback will work well over the
public Internet, due to ACK loss and the variety of aggregation mechanisms
in play. That is why in my previous email in this thread I was mainly
presuming that if the AccECN TCP Option is not forwarded by the path then
the flow might have to just disable AccECN and L4S.

cheers,
neal

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 12 Jul 2021 21:47:28 +0100

Neal,

On 12/07/2021 15:35, Neal Cardwell wrote:
>
>
> On Mon, Jul 12, 2021 at 8:30 AM Bob Briscoe <[email-masked] 
>>         > On Jul 11, 2021, at 4:42 AM, Bob Briscoe
>>         <[email-masked] <mailto:[email-masked]>> wrote:
>>
>>     ...
>>
>>         > The implementation will have its own ACK ratio - that's out
>>         of scope of AccECN, except to set this max of 6 CEs, which is
>>         to mitigate wrap of the 3-bit counter of CE-marks (which in
>>         the worst case of 100% marking in the data direction could
>>         then induce 1 ACK per 6 packets). This shouldn't limit
>>         forward performance, because it only increases the reverse
>>         ACK rate if there is heavy congestion in the forward path,
>>         when the Data Sender should be reducing the forward rate anyway.
>>
>>         1 ACK per 6 packets would cause performance issues on high
>>         speed links. Common setup is "4 to 8 ACK per RTT", which
>>         means intervals much larger than 6 packets.
>>
>>
>>     I share this concern about the AccECN requirement of one ACK per
>>     6 CE-marked data segments potentially causing performance issues
>>     with high-speed links. Today, with most high-speed last-hop link
>>     technologies (wifi, cellular, Ethernet) TCP receivers often
>>     receive (from lower layers of the hardware and software
>>     networking stack) large aggregates  (often up to 64KBytes or
>>     around 44 packets), and generate a single ACK for that aggregate.
>>     AFAICT changing this scenario to require 1 packet per 6 CE-marked
>>     data segments means that during CE-marked scenarios receiving
>>     such an aggregate would require ceil(44/6) = 8 ACKs, increasing
>>     the number of ACKs by up to 8x.
>>
>>     (1) CPU load: This seems like it would impose much higher CPU
>>     load on the data receiver, at just the moment when the receiver
>>     may already be under stress due to receiving data near the
>>     maximum rate of the link.
>>
>>     (2) Congestion and ACK loss: This requirement for generating up
>>     to 8 ACKs per aggregate seems like it is likely to produce a
>>     tight burst of 8 ACKs, which is going to increase congestion and
>>     increase the odds of losing at least one ACK, which is going to
>>     cause accuracy problems for "Accurate" ECN. :-)
>
>     [BB] I hadn't intended this wording to apply in response to a
>     large burst. I'd want to alter the wording to allow just one ACK
>     in this case.
>
>
> In the case where there is just one ACK for a big CE-marked burst of 
> >= 8 segments, does that mean the result is:
>
> (b) the TCP connection interpreting the ACE field should presume the 
> ACE field probably wrapped and estimate the number of CE-marked 
> segments using the number of SACKed/ACKed segments, rather than the 
> ACE field?
>

[BB] Answers:
a) Yes, (obviously) less accurate than with the Option
b) Sort-of.
Just presuming the field wrapped as many times as possible gives v poor 
performance.
Ilpo and I have been developing heuristics to deal with cases where the 
number of SACKed/ACKed segments is >=8.
I'm planning to report the results in the meeting.

>>     I understand that this proposed "ACK per 6 CE-marked data
>>     segments" rule is necessary to avoid issues with the 3-bit ACE
>>     field wrapping. So IMHO this is one of the good arguments against
>>     including the ACE field in the AccECN design.
>
>     [BB] I think the subtext here is a preference for the DCTCP style
>     of feedback vs ACE (when there is no AccECN TCP Option). I don't
>     think this particular issue is any different between the two.
>
>
>     With DCTCP feedback, when a large data burst like this arrives, if
>     there are transitions to and from CE marking within the burst,
>     does Linux DCTCP generate an ACK for each transition, and for
>     every n repetitions of CE not ECT, like the RFC says it should? Or
>     does the implementation just pop out one ACK at the end?
>
>
> Yes, with Linux DCTCP (or BBRv2), when there is a big burst, there is 
> an ACK for each CE<->non-CE transition. They also handle the case 
> where there is a CE<->non-CE transition while a delayed ACK is 
> pending. This requires cooperation from a few different layers.
>
> (1) the NIC hardware LRO aggregation is supposed to not aggregate 
> across CE<->non-CE transitions. I believe the de facto spec on NIC 
> aggregation with respect to IP fields is the following, which mandates 
> that:
> <https://docs.microsoft.com/en-us/windows-hardware/drivers/network/updating-the-ip-headers-for-coalesced-segments>
>
> CE<->non-CE transitions; see inet_gro_receive() checking the  ToS 
> bytes: (iph->tos ^ iph2->tos)
>
> (3) the DCTCP and BBRv2 code use dctcp_ece_ack_update() to handle the 
> case where there is a CE<->non-CE transition while a delayed ACK is 
> pending

[BB] Understood. So doesn't my point stand that the existing 
hardware/software combination already often/typically emits numerous 
ACKs in response to arrival of a large data burst? Indeed, designing the 
ACE field as a counter was deliberately intended to reduce that.

Did you see the text we included in the current rev (-14 in Feb'21) that 
explains this:
https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-accurate-ecn-14#section-3.3.4
It also explains how the new approach can be handled while hardware is 
not available. Essentially, assuming you start with a subset of users 
negotiating AccECN, those sessions would switch receive offload to 
software-only, until hardware became available. But I'd appreciate your 
thoughts on the whole section.

I've updated my local copy of the 'rules' in our proposed new rev of 
draft-15, to replace 'immediately send' and explain better what it 
means. Here it is:

3.2.2.5.1.  Data Receiver Safety Procedures

    The following rules define when a Data Receiver in AccECN mode emits
    an ACK:

    Change-Triggered ACKs:  An AccECN Data Receiver SHOULD emit an ACK
       whenever a data packet marked CE arrives after the previous packet
       was not CE.

       Even though this rule is stated as a "SHOULD", it is important for
       a transition to trigger an ACK if at all possible, The only valid
       exception to this rule is given below these bullets.

       For the avoidance of doubt, this rule is deliberately worded to
       apply solely when _data_ packets arrive, but the comparison with
       the previous packet includes any packet, not just data packets.

    Increment-Triggered ACKs:  An AccECN Data Receiver MUST emit an ACK
       if 'n' CE marks have arrived since the previous ACK.  If there is
       new data to acknowledge, 'n' SHOULD be 2.  If there is no new data
       to acknowledge, 'n' SHOULD be 3 and MUST be no less than 3.  In
       either case, 'n' MUST be no greater than 6.

    The above rules for when to send an ACK are designed to be
    complemented by those in Section 3.2.3.3, which concern whether the
    AccECN TCP Option ought to be included on ACKs.

    If the arrivals of a number of data packets are all processed as one
    event, e.g. using large receive offload (LRO) or generic receive
    offload (GRO), both the above rules SHOULD be interpreted as
    requiring multiple ACKs to be emitted back-to-back (for each
    transition and for each repetition by 'n' CE marks).  If this is
    problematic for high performance, either rule can be interpreted as
    requiring just a single ACK at the end of the whole receive event.

    Even if a number of data packets do not arrive as one event, the
    'Change-Triggered ACKs' rule could sometimes cause the ACK rate to be
    problematic for high performance (although high performance protocols
    such as DCTCP already successfully use change-triggered ACKs).  The
    rationale for change-triggered ACKs is so that the Data Sender can
    rely on them to detect queue growth as soon as possible, particularly
    at the start of a flow.  The approach can lead to some additional
    ACKs but it feeds back the timing and the order in which ECN marks
    are received with minimal additional complexity.  If CE marks are
    infrequent, or there are multiple marks in a row, the additional load
    will be low.  Other marking patterns could increase the load
    significantly.  One possible compromise would be for the receiver to
    heuristically detect whether the sender is in slow-start, then to
    implement change-triggered ACKs while the sender is in slow-start,
    and offload otherwise.

    With ECN-capable pure ACKs [I-D.ietf-tcpm-generalized-ecn], the
    'Increment-Triggered ACKs' rule could cause ECN-marked pure ACKs to
    trigger further ACKs. [...snipped to avoid muddling this thread
                  with the "ACKing ACKs without good cause" thread]

>
>     Ilpo & I have been doing experiments with high levels of ACK
>     coalescing causing the ACE field seen by the Data Sender to wrap
>     multiple times under high congestion. For example, 1 ACK per ~4ms
>     is one common scheme, which would result in about 1 ACK for 34
>     data packets at just 100Mb/s.
>
>
>     pretty much the same but, with even low levels of ACK coalescing,
>     the accuracy of ACE is superior. Nonetheless, I understand that
>     deploying something different to DCTCP when you've already got
>     DCTCP could involve deployment complexity.
>
>

[BB] That would I think be the same for the DCTCP feedback scheme (it's 
only not the case now in a DC where you can hard-code rather than 
negotiate the scheme).

> increment to the ACE field was in the case of possible ACK loss

[BB] That's slightly true ;)

I would say it like this:
* With DCTCP feedback, a simple line of code can fill in the missing 
ACKs: it assumes the marking continued as per the last signal. This is 
very inaccurate under high levels of ACK coalescing and there is no 
obvious way to make it more accurate.
* With ACE feedback a simple line of code can also fill in the missing 
ACKs. However, in contrast to DCTCP, you can add code to make ACE more 
accurate if you are prepared to add the extra complexity (but you don't 
have to).

>
> o interactions with drivers or hardware that already have assumptions 
> about the header bits used for ACE (e.g. that the CWR bit should be 
> cleared for the first N-1 segments in a TSO burst and only set on the 
> last segment of the TSO burst)

[BB]

>     [BB] The redundancy is intended. For cases where the TCP Option
>     doesn't traverse the path, or where TCP option space is limited.
>
>
> I realize the redundancy is intentional, but the redundant code 
> does impose a maintenance cost.
>
>>
>     [BB] Well, we've tested the 3 header bits over millions of paths
>     without problems. But yes, there are billions more to test.
>
>>
>>     o Known problems with NICs and drivers (based on Ilpo's nice
>>     talk, "Accurate ECN Linux Implementation: Experiences and
>>     Challenges", at the April 2020 TCPM interim meeting)
>
>     [BB] I understood that talk as concluding there weren't really
>     problems. What specific problem do you have in mind?
>
>
> o interactions with drivers or hardware that already have assumptions 
> about the header bits used for ACE (e.g. that the CWR bit should be 
> cleared for the first N-1 segments in a TSO burst and only set on the 
> last segment of the TSO burst)
>
> period of finding the drivers and NICs that are incompatible with the 
> ACE field, and working around those.

[BB] Yes. You can't make an omelette without breaking eggs. Meaning, to 
get significantly lower delay over the public Internet, we can't expect 
it all just to fall into place with minimal effort. Given we have worked 
out a way to deploy pretty accurate ECN feedback without the option, it 
would be a shame to decide not to, just because there /might/ be 
problems. There always /might/ be problems, indeed there nearly always are.

If DCTCP feedback were used over the public Internet, it would have to 
be included some form of negotiation framework, probably like AccECN, 
and that would probably also trip over some hardware incompatibilities. 
Why make half-hearted changes and still have potential deployment problems?

I really think we need to question the prevailing preference for 
inconsequential changes, when we end up with so much grief making /any/ 
change.

>>     I realize that if AccECN does not have the ACE field feature,
>>     then AccECN and TCP L4S will not be usable on paths with
>>     middleboxes that strip the AccECN counter options. But IMHO
>>     living without the ACE field is preferable. IMHO it's acceptable
>>     to say that L4S can only be used with (a) QUIC, or (b) TCP
>     [BB] When you say 'living without the ACE field', alongside the
>     AccECN TCP Option, would you leave classic ECN feedback? Or put
>     DCTCP feedback in its place?
>
>
> When 'living without the ACE field', I mean either using the AccECN 
> TCP Option or DCTCP-style feedback.
>
>     I don't think there's any case for using classic ECN feedback
>     within AccECN. I see the competition as between ACE and DCTCP
>     feedback, each optionally with the AccECN TCP Option.
>
>
>     If DCTCP feedback were a stop-gap until we could get good
>     traversal of the TCP Option, it might have some merit, but I think
>     DCTCP isn't going to work well with the level of ACK coalescing in
>     the Internet. However, this is very difficult to judge unless we
>     do large scale A-B experiments on the real Internet (and
>     potentially within DCs). I'll think further about enabling some
>     way to do that, possibly within AccECN's negotiation framework.
>     But today, I have to prioritize for the draft deadline.
>
>
> I agree it's unclear whether DCTCP-style feedback will work well over 
> the public Internet, due to ACK loss and the variety of aggregation 
> mechanisms in play. That is why in my previous email in this thread I 
> was mainly presuming that if the AccECN TCP Option is not forwarded by 
> the path then the flow might have to just disable AccECN and L4S.

[BB] If you're saying a third possibility would be to have no ECN 
feedback in the header, only via an option, that seems perverse, given 
the ECN header bits would then be idle. And, if you consumed the option 
space for SACK, timestamps, (even MPTCP) etc. you might have no ECN 
feedback at all.

The ability to have wider early deployment is important for the economic 
motivations of everyone involved, so not to be dismissed lightly.

I'll think whether we can find some way to try out both ACE and DCTCP 
feedback within a realistic protocol so that all the deployment niggles 
and performance can be tested and compared. But if you also agree that 
DCTCP probably won't work well over the Internet, it might be a 
non-starter anyway (but worth checking if not too onerous).

Bob