Subject: Re: [tcpm] Linux doesn’t implement RFC3465

From: Vidhi Goel <[email-masked]>
Date: Wed, 28 Jul 2021 14:12:56 -0700

Resurrecting the 3465 thread.

In the TCPM meeting at IETF 111, we discussed about this issue of L=2 which is a MUST in RFC 3465. This is a very strict requirement and stacks like Linux already doesn’t follow it.
> The basic principle in the Linux code is that the sender should use the ACKs to learn about the capacity of the path (both in volumetric and rate dimensions), and should not ignore that information. This allows the sender to quickly grow and achieve high throughput, even in the presence of stretch ACKs, which are pervasive, due to TSO/GSO, GRO/LRO, etc.
> 
> Considering bursts is important, but that can be tackled as an orthogonal issue. Bursts are avoided in the Linux TCP ecosystem by the combination of TSO autosizing, pacing, TSQ, and fair queueing.

As Neal described, the congestion controller should use the information in stretch ACKs to increase its congestion window so that it correctly adjusts the cwnd based on available link capacity.
Burstiness is an orthogonal issue which can be solved by pacing.

QUIC loss recovery (RFC 9002) also follows this approach.

Mark,
As a lot of transport and congestion control drafts reference RFC 3465, do you think we should update this RFC to reflect the current deployment? This would also be useful for someone who is just starting with a new implementation.

Thanks,
Vidhi

> On Nov 27, 2019, at 5:14 AM, Mark Allman <[email-masked]> wrote:
> 
> 
>> +Mark Allman
> 
> Just to clear it up, I *was* at BBN long ago when the ABC document
> was written.  It's a cool place to work.  I recommend it.  But, I
> now hang out at ICSI.
> 
>> I believe that ABC was written to solve the problem with ACK
>> counting by counting the number of bytes acknowledged for
>> misbehaving receivers. Limiting the increase to 2*MSS was a good
>> solution to avoid bursts at the time.
> 
> The main motivation behind ABC was to counteract delayed ACKs.  The
> common approach at the time was to just bump cwnd by one MSS every
> time an ACK rolled in.  So, if an ACK covered two segments because
> the receiver was delaying the ACKs then the growth rate during slow
> start was 1.5x per RTT instead of the 2x that was really
> envisioned.  During congestion avoidance the growth was 1 MSS every
> 2 RTTs instead of the envisioned every RTT.
> 
> A secondary motivation was to counteract these ACK division attacks
> that Savage taught us about.  I.e., we could ACK an MSS-sized packet
> one byte at a time and the sender would then increase the cwnd by
> MSS*MSS bytes in the prevalent ACK counting scheme (i.e., cwnd would
> get bumped by MSS bytes for every ACK).
> 
> The limit has two roots ...
> 
> (1) The limit is important in slow starts that follow an RTO.  As
>    the RFC discusses, in this case we might retransmit a single
>    packet and this will cause the receiver's window to slide a
>    great deal.  Therefore, an ACK may indicate that a ton of data
>    has left the network, but that isn't really the case.  So, we
>    don't want to increase the cwnd based on all the new bytes
>    ACKed.
> 
>    I have since mostly decided that this use of L is crude.
>    Probably there is a more elegant way to do it by using the
>    scoreboard and the SACK information to get a better
>    understanding of what left the network and when.  That said, in
>    this case L is simple and probably about right most of the
>    time.
> 
> (2) I think there was some general conservativeness to bursts and
>    using L everywhere quelled some of the worry.  Here L=2 was used
>    to exactly offset delayed ACKs.
> 
>> I agree that increasing the congestion window and controlling the
>> burst rate are orthogonal issues.
> 
> Yes.  In fact, we did subsequent research on mitigating bursts
> because we never really thought of ABC as somehow the way to control
> bursts (research papers available, but never went into RFCs).
> 
> And, in a world that leverages stretch ACKs as a routine I think
> Linux's approach of not using an L may well be correct.  Documenting
> that and the reasoning behind it in modern networks seems useful to
> me.
> 
> allman
>

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 28 Jul 2021 16:20:14 -0700

Thank you Vidhi and Mark for supporting an ABC update.

To give this a start, my recommended update is:

1) if the sender uses some form of pacing to send data packets, L is
RECOMMENDED to be the effective window worth of packets. Pacing here refers
to spread packet transmission following a rate based on the congestion
window and round trip. Additionally if SACK is supported, the same L
applies in slow start after RTO

2) otherwise L is RECOMMENDED to be 8

Thoughts?

On Wed, Jul 28, 2021 at 2:13 PM Vidhi Goel <vidhi_goel=
[email-masked]> wrote:

> In the TCPM meeting at IETF 111, we discussed about this issue of L=2
> which is a MUST in RFC 3465. This is a very strict requirement and stacks
> like Linux already doesn’t follow it.
>
> The basic principle in the Linux code is that the sender should use the
> ACKs to learn about the capacity of the path (both in volumetric and rate
> dimensions), and should not ignore that information. This allows the sender
> to quickly grow and achieve high throughput, even in the presence of
> stretch ACKs, which are pervasive, due to TSO/GSO, GRO/LRO, etc.
>
> Considering bursts is important, but that can be tackled as an orthogonal
> issue. Bursts are avoided in the Linux TCP ecosystem by the combination of
> TSO autosizing, pacing, TSQ, and fair queueing.
>
>
> As Neal described, the congestion controller should use the information in
> stretch ACKs to increase its congestion window so that it correctly adjusts
> the *cwnd* based on available link capacity.
> *Mark*,
> As a lot of transport and congestion control drafts reference RFC 3465, do
> you think we should update this RFC to reflect the current deployment? This
> would also be useful for someone who is just starting with a new
> implementation.
>
> Thanks,
> Vidhi
>
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

---

From: Neal Cardwell <[email-masked]>
Date: Wed, 28 Jul 2021 20:41:27 -0400

I would vote that the guidance should be something like:

  On an ACK, don't increase the cwnd by more than the amount of data ACKed.

This implies:

 o slow-start is limited to doubling each round trip

 o the Savage ACK division attacks are squashed

I don't think limiting cwnd growth to some constant L less than cwnd is the
best road to go down, since:

(a) At high speeds (at least >10Gbps) TCP has needed to operate in
TSO/GRO/LRO/GRO aggregates of 64KBytes for a decade or more, and is now
moving on to larger aggregates to go beyond 100Gbps (see Eric Dumazet's
nice talk at the Linux Netdev conference a few weeks ago). Having a
constant L is not scalable or future-proof. I would imagine any constant L
lower than 64KBytes will be ignored, in practice, with good reason. A
constant L bigger than 64KBytes will be out of date soon, if it's not
already.

(b) If we're worried about bursts (which we should be), then we should
recommend pacing. And RFC 7661 already does that:
  https://datatracker.ietf.org/doc/html/rfc7661#section-4.4.2

  "A TCP sender in the non-validated phase SHOULD control the maximum
   burst size, e.g., using a rate-based pacing algorithm in which a
   sender paces out the cwnd over its estimate of the RTT, or some
   other method, to prevent many segments being transmitted
   contiguously at line-rate."

  So a sender that follows RFC 7661 already has a solution for bursts:
pacing.

(b) If an implementation ignores RFC 7661 and does not do pacing, then
limiting cwnd *growth* to L per ACK does not prevent bursts. A sender that
ignores RFC 7661 will be frequently restarting from idle to blast a full
cwnd at line rate. Those restart-from-idle cwnd bursts are already
super-common in some of the most common types of TCP traffic: web, RPC,
streaming video. Having a particular L limit for cwnd growth will not save
us from those cwnd-magnitude bursts.

Just my two cents on that,
neal

On Wed, Jul 28, 2021 at 7:20 PM Yuchung Cheng <[email-masked]> wrote:

>> As Neal described, the congestion controller should use the information
>> in stretch ACKs to increase its congestion window so that it correctly
>> adjusts the *cwnd* based on available link capacity.
>> As a lot of transport and congestion control drafts reference RFC 3465,
>> do you think we should update this RFC to reflect the current deployment?
>> This would also be useful for someone who is just starting with a new
>> implementation.
>>
>> Thanks,
>> Vidhi
>>

---

From: "Mark Allman" <[email-masked]>
Date: Thu, 29 Jul 2021 08:41:24 -0400

I just inhaled notes from Vidhi, Yuchung and Neal.  And, I am trying
to page the recesses of my brain back in ... :-)

I have a few thoughts ...

  - I sort of agree with Yuchung & Neal in spirit, if not
    particulars.

  - During the initial slow start phase I think it'd be fine to say
    either:

    (a) If you use some sort of burst mitigator/pacing, an
        implementation can increase cwnd by the number of bytes
        ACKed on each ACK (i.e., no L).

    (b) If there is no burst mitigation then we have to figure out
        if L is still useful for this purpose and whether we want to
        retain it.  Seems like perhaps L=2 is sensible here.  L was
        never meant to be some general burst mitigator.  However,
        ABC clearly *can* aggravate bursting and so perhaps it makes
        sense to have it also try to limit the impact of the
        aggravation (in the absence of some general mechanism).

  - During slow starts that follow RTOs there is a general problem
    that just because the window slides by X bytes doesn't say
    anything about the *network*, as that sliding can happen because
    much of the data was likely queued for the application on the
    receiver.  So, e.g., you can RTO and send one packet and get an
    ACK back that slides the window 10 packets.  That doesn't mean
    10 packets left.  It means one packet left the network and nine
    packets are eligible to be sent to the application.  So, it is
    not OK to set the cwnd to 1+10 = 11 packets in response to this
    ACK.  Here L should exist and be 1.

  - It is possible that someone could work out a SACK-based scheme
    to try to figure out how much of the ACK is data leaving the
    network and how much of it was data sitting in the buffer at the
    receiver.  And, if we could do that then we could say "L=1
    unless you use a fancy algorithm".  My guess, however, is this
    is likely a fool's errand for two reasons.  First, on an RTO the
    scoreboard is supposed to be cleared and while SACK information
    will slowly get built back up we lose a bunch of state that we
    could perhaps use to differentiate between "left network" and
    "left buffer".  Second, receivers are supposed to ACK out of
    order segments and those that fill in holes immediately.  If
    they do that, it means that every ACK received actually does
    convey one segment leaving the network and therefore L=1 is
    correct and even an accurate SACK-based algorithm wouldn't let
    one increase cwnd more than L=1 would.  (One caveat here is I am
    not sure how this thinking would play with the offload stuff.)

allman

---

From: Neal Cardwell <[email-masked]>
Date: Thu, 29 Jul 2021 09:38:39 -0400

>         aggravation (in the absence of some general mechanism).

Even if recommending a static L value, IMHO L=2 is a bit conservative.
Given that RFC 6928 and the experience with IW10 seems to show that the
vast majority of paths on the public Internet work fine with bursts of 10
packets, this seems to suggest that it is safe to allow an ACK for 5
packets in slow start to increase cwnd by 5 packets and allow the release
of 10 packets, which I guess implies an L of at least 5.

>     ACK.  Here L should exist and be 1.

AFAICT this argument only applies to non-SACK connections. For connections
with SACK (the vast majority of connections over the public Internet and in
datacenters), it is quite feasible to determine how many packets really
left the network (and Linux TCP does this; see below).

>     unless you use a fancy algorithm".

The SACK-based scheme to try to figure out how much of the ACK is data
leaving the network has been implemented in Linux for at least 20 years or
so ( see the 2002 paper "Congestion Control in Linux TCP" by Pasi Sarolahti
and Alexey Kuznetsov:
https://www.cs.helsinki.fi/research/iwtcp/papers/linuxtcp.pdf )

> My guess, however, is this
>     "left buffer".

Clearing the scoreboard on RTO is indeed a "SHOULD" from RFC 2018:

   After a retransmit timeout the data sender SHOULD turn off all of the
   SACKed bits, since the timeout might indicate that the data receiver
   has reneged.  The data sender MUST retransmit the segment at the left
   edge of the window after a retransmit timeout, whether or not the
   SACKed bit is on for that segment.

But our experience with Linux TCP is that clearing the scoreboard on RTO is
unnecessary, and causes unnecessary performance problems and redundant
retransmissions.  While it is true that "the timeout might indicate that
the data receiver has reneged", it is vastly more common for timeouts to be
due to simple packet loss. And the sender can specifically detect reneging
if it finds it times out and hits the specific scenario implied in the RFC
(the segment at the left edge of the window is SACKed); and if this
specific indication of reneging happens, *then* it makes sense for the
sender to clear the SACK scoreboard. This is the approach Linux TCP has
used for a decade or more, and it works very well in practice.

>   Second, receivers are supposed to ACK out of

Yes, offload mechanisms are so pervasive in practice, that this kind of
reasoning no longer holds in common cases. In common cases, there are
hardware (LRO) or software (GRO) mechanisms that are doing aggregation of
consecutive byte ranges at a layer that is too low to know whether the byte
ranges are out of order or fill a hole, so by the time that out-of-order or
hole-filling ranges reach TCP they may be up to 64KBytes in length. So
although TCP will try to ACK an out-of-order or hole-filling sequence range
"immediately" from the TCP layer's perspective, it may often be up to
64Kbytes or roughly 44 segments in length.

best regards,
neal

---

From: "Mark Allman" <[email-masked]>
Date: Thu, 29 Jul 2021 10:06:31 -0400

> Even if recommending a static L value, IMHO L=2 is a bit

Well, perhaps.  L=2 was designed to exactly counteract delayed ACKs.
So, it isn't exactly a new magic number.  We could wave our hands
and say "5 seems OK" or "10 seems OK" or whatever.  And, I am sure
we could come up with something that folks felt was fine.  However,
my feeling is that if we want to worry about bursts then let's worry
about bursts in some generic way.  And, if you have some way to deal
with bursts then L isn't needed.  And, if you don't have a way to
deal with bursts then a conservative L seems fine.  But, perhaps
putting the effort into a generic mechanism instead of cooking yet
another magic number we need to periodically refresh is probably a
better way to spend effort.

>>   - During slow starts that follow RTOs there is a general
>>     problem that just because the window slides by X bytes
>>     doesn't say anything about the *network*, as that sliding can
>>     happen because much of the data was likely queued for the
>>     application on the receiver.  So, e.g., you can RTO and send
>>     one packet and get an ACK back that slides the window 10
>>     packets.  That doesn't mean 10 packets left.  It means one
>>     packet left the network and nine packets are eligible to be
>>     sent to the application.  So, it is not OK to set the cwnd to
>>     1+10 = 11 packets in response to this ACK.  Here L should
>>     exist and be 1.
>
> AFAICT this argument only applies to non-SACK connections. For
> connections with SACK (the vast majority of connections over the
> public Internet and in datacenters), it is quite feasible to
> determine how many packets really left the network (and Linux TCP
> does this; see below).

If you have an accurate way to figure out how many of the ACKed
bytes left the network and how many were just buffered at the
receiver then I see no problem with increasing based on byte count
as you do in the initial slow start.

(I don't remember what the paper you cite says, but my guess is it's
often the case that L=1 is a reasonable substitute for something
complicated here.  But, perhaps I am running the simulation in my
head wrong ... it has been a while, admittedly!)

> Yes, offload mechanisms are so pervasive in practice,

I am trying to build a mental model here.  How pervasive would you
guess these are?  And, where in the network?  I have assumed that
they are for sure pervasive in data centers and server farms, but
not for the vast majority of Internet-connected devices.

allman

---

From: Neal Cardwell <[email-masked]>
Date: Thu, 29 Jul 2021 11:19:16 -0400

> better way to spend effort.

Yes, I very much agree that "putting the effort into a generic mechanism
instead of cooking yet another magic number we need to periodically refresh
is probably a better way to spend effort."

>From my impression looking at public Internet traces, aggregation
mechanisms that cause TCP ACKs for more than 2 segments are very common. I
suspect that's because the majority of public Internet traffic these days
has a bottleneck that is either wifi, cellular, or DOCSIS, and all of these
have a shared medium with a large latency overhead for L2 MAC control of
gets to speak next. So a lot of batching happens, both in big batches of
data that arrive at the client in the same L2 medium time slot, and big
batches of ACKs that accumulate while the client waits (often several
milliseconds, sometimes even tens of milliseconds) for its chance to send a
big stretch ACK or batch of ACKs.

This brings up a related point: even if there is some ABC-style per-ACK L
limit on cwnd increases, the time structure of most public Internet ACK
streams is massively bursty because of these aggregation mechanisms
inherent in L2 behavior on most public Internet bottlenecks (wifi,
cellular, DOCSIS). So even if there is a limit L that limits the per-ACK
behavior to be smooth, if there is no pacing of data segments then the data
transmit time structure will still be bursty because the ACK arrivals these
days are very bursty.

best regards,
neal

---

From: Vidhi Goel <[email-masked]>
Date: Thu, 29 Jul 2021 13:19:23 -0700

> Yes, I very much agree that "putting the effort into a generic mechanism instead of cooking yet another magic number we need to periodically refresh is probably a better way to spend effort.” 

I agree that defining such a number doesn’t fully solve the problem but it gives some recommendation for implementations that don’t do pacing. So, defining a somewhat less restrictive value for L (5 or 10) would be a last resort for implementations that don’t pace.

Thanks,
Vidhi

> On Jul 29, 2021, at 8:19 AM, Neal Cardwell <[email-masked]> wrote:
> 
> 
> 
> On Thu, Jul 29, 2021 at 10:06 AM Mark Allman <[email-masked] <mailto:[email-masked]>> wrote:
> 
> Yes, I very much agree that "putting the effort into a generic mechanism instead of cooking yet another magic number we need to periodically refresh is probably a better way to spend effort." 
> 
> From my impression looking at public Internet traces, aggregation mechanisms that cause TCP ACKs for more than 2 segments are very common. I suspect that's because the majority of public Internet traffic these days has a bottleneck that is either wifi, cellular, or DOCSIS, and all of these have a shared medium with a large latency overhead for L2 MAC control of gets to speak next. So a lot of batching happens, both in big batches of data that arrive at the client in the same L2 medium time slot, and big batches of ACKs that accumulate while the client waits (often several milliseconds, sometimes even tens of milliseconds) for its chance to send a big stretch ACK or batch of ACKs.
> 
> This brings up a related point: even if there is some ABC-style per-ACK L limit on cwnd increases, the time structure of most public Internet ACK streams is massively bursty because of these aggregation mechanisms inherent in L2 behavior on most public Internet bottlenecks (wifi, cellular, DOCSIS). So even if there is a limit L that limits the per-ACK behavior to be smooth, if there is no pacing of data segments then the data transmit time structure will still be bursty because the ACK arrivals these days are very bursty. 
> 
> best regards,
> neal

---

From: Yuchung Cheng <[email-masked]>
Date: Thu, 29 Jul 2021 13:47:42 -0700

On Thu, Jul 29, 2021 at 1:19 PM Vidhi Goel <vidhi_goel=
[email-masked]> wrote:

> is probably a better way to spend effort.”
>
>
> I agree that defining such a number doesn’t fully solve the problem but it
> gives some recommendation for implementations that don’t do pacing. So,
> defining a somewhat less restrictive value for L (5 or 10) would be a last
> resort for implementations that don’t pace.
>
How about putting a number 10, and also put all the rationales to follow to
decide a higher or lower value. It's never one-size for all.

Also I believe it's time to move ABC into the standards track, in the era
of (bigger and bigger) stretch ACKs.

---

From: Vidhi Goel <[email-masked]>
Date: Thu, 29 Jul 2021 18:03:10 -0700

> How about putting a number 10, and also put all the rationales to follow to decide a higher or lower value. It's never one-size for all.

That sounds great. Something on the lines of,

 “This document RECOMMENDS using mechanisms like Pacing to control how many bytes are sent to the network at a point of time. But if it is not possible to implement pacing, an implementation MAY implicitly pace their traffic by applying a limit L to the increase in congestion window per ACK during slow start. In modern stacks, acknowledgments are aggregated for various reason, CPU optimization, reducing network load etc. Hence it is common for a sender to receive an aggregated ACK that acknowledges more than 2 segments. For example, a stack that implements GRO could aggregate packets up to 64Kbytes or ~44 segments before passing on to the TCP layer and this would result in a single ACK to be generated by the TCP stack. Given that an initial window of 10 packets in current deployments has been working fine, the draft makes a recommendation to set L=10 during slow start. This would mean that with every ACK, we are probing for a new capacity by sending 10 packets in addition to the previously discovered capacity. Implementations MAY choose to set a lower limit if they believe an increase of 10 is too aggressive."

Does this sound like what we would like to say?

-
Vidhi

> On Jul 29, 2021, at 1:47 PM, Yuchung Cheng <ycheng=[email-masked]> wrote:
> 
> 
> 
> On Thu, Jul 29, 2021 at 1:19 PM Vidhi Goel <vidhi_goel=[email-masked] <mailto:[email-masked]>> wrote:
> How about putting a number 10, and also put all the rationales to follow to decide a higher or lower value. It's never one-size for all.
> 
> Also I believe it's time to move ABC into the standards track, in the era of (bigger and bigger) stretch ACKs.
> 
> 
> Thanks,
> Vidhi
> 
> 
> 
>> On Jul 29, 2021, at 8:19 AM, Neal Cardwell <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> 
>> 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/tcpm <https://www.ietf.org/mailman/listinfo/tcpm>

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 30 Jul 2021 11:04:32 -0700

On Thu, Jul 29, 2021 at 6:03 PM Vidhi Goel <vidhi_goel=
[email-masked]> wrote:

>> I agree that defining such a number doesn’t fully solve the problem but
>> it gives some recommendation for implementations that don’t do pacing. So,
> How about putting a number 10, and also put all the rationales to follow
> to decide a higher or lower value. It's never one-size for all.
>
>
>  “This document RECOMMENDS using mechanisms like Pacing to control how
> many bytes are sent to the network at a point of time. But if it is not
> possible to implement pacing, an implementation MAY implicitly pace their
> traffic by applying a limit L to the increase in congestion window per ACK
> during slow start. In modern stacks, acknowledgments are aggregated for
> various reason, CPU optimization, reducing network load etc. Hence it is
> common for a sender to receive an aggregated ACK that acknowledges more
> than 2 segments. For example, a stack that implements GRO could aggregate
> packets up to 64Kbytes or ~44 segments before passing on to the TCP layer
> and this would result in a single ACK to be generated by the TCP stack.
> Given that an initial window of 10 packets in current deployments has
> been working fine, the draft makes a recommendation to set L=10 during slow
> start. This would mean that with every ACK, we are probing for a new
> capacity by sending 10 packets in addition to the previously discovered
> capacity. Implementations MAY choose to set a lower limit if they believe
> an increase of 10 is too aggressive."
>
Thanks for taking a shot. I would put more description on Pacing to ensure
better implementation. How about:
"Pacing here refers to spread packet transmission following a rate based on
the congestion window and round trip." with a citation of
https://datatracker.ietf.org/doc/html/rfc7661#section-4.4.2

I would also refer to IW RFC 6928 in case it gets increased / updated a few
years later.
Hmm maybe we should also move RFC6928 to the standard track :-)

> -
> Vidhi
>
> On Jul 29, 2021, at 1:47 PM, Yuchung Cheng <
> ycheng=[email-masked]> wrote:
>
>
>
>> I agree that defining such a number doesn’t fully solve the problem but
>> it gives some recommendation for implementations that don’t do pacing. So,
> How about putting a number 10, and also put all the rationales to follow
> to decide a higher or lower value. It's never one-size for all.
>

---

From: Vidhi Goel <[email-masked]>
Date: Fri, 30 Jul 2021 13:25:51 -0700

Hi Mark, Yuchung, Neal,

Given that we have some ideas / suggestion text on improving 3465, how do you think we should proceed?
Mark, do you want to start a GitHub repo or such with some of the changes already suggested so far and others can review and / or contribute to the bis draft?

Thanks,
Vidhi

> On Jul 30, 2021, at 11:04 AM, Yuchung Cheng <ycheng=[email-masked]> wrote:
> 
> 
> 
> On Thu, Jul 29, 2021 at 6:03 PM Vidhi Goel <vidhi_goel=[email-masked] <mailto:[email-masked]>> wrote:
> Thanks for taking a shot. I would put more description on Pacing to ensure better implementation. How about:
> "Pacing here refers to spread packet transmission following a rate based on the congestion window and round trip." with a citation of https://datatracker.ietf.org/doc/html/rfc7661#section-4.4.2 <https://datatracker.ietf.org/doc/html/rfc7661#section-4.4.2>
> 
> 
> 
> I would also refer to IW RFC 6928 in case it gets increased / updated a few years later.
>> On Jul 29, 2021, at 1:47 PM, Yuchung Cheng <ycheng=[email-masked] <mailto:ycheng=[email-masked]>> wrote:
>> 
>> 
>>

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 30 Jul 2021 16:04:52 -0700

github repo sounds good to me.

On Fri, Jul 30, 2021 at 1:26 PM Vidhi Goel <vidhi_goel=
[email-masked]> wrote:

> Mark, do you want to start a GitHub repo or such with some of the changes
> already suggested so far and others can review and / or contribute to the
> bis draft?
>
> Thanks,
> Vidhi
>
> On Jul 30, 2021, at 11:04 AM, Yuchung Cheng <
> "Pacing here refers to spread packet transmission following a rate based
> on the congestion window and round trip." with a citation of
>>> This brings up a related point: even if there is some ABC-style per-ACK
>>> L limit on cwnd increases, the time structure of most public Internet ACK

---

From: Neal Cardwell <[email-masked]>
Date: Fri, 30 Jul 2021 19:07:19 -0400

A github repo sounds good to me as well.

thanks,
neal

On Fri, Jul 30, 2021 at 7:05 PM Yuchung Cheng <[email-masked]> wrote:
> On Fri, Jul 30, 2021 at 1:26 PM Vidhi Goel <vidhi_goel=[email-masked]> wrote:
>>
>> Hi Mark, Yuchung, Neal,
>>
>> On Thu, Jul 29, 2021 at 6:03 PM Vidhi Goel <vidhi_goel=[email-masked]> wrote:
>>>>>
>> "Pacing here refers to spread packet transmission following a rate based on the congestion window and round trip." with a citation of https://datatracker.ietf.org/doc/html/rfc7661#section-4.4.2
>>
>>
>>
>>> On Thu, Jul 29, 2021 at 1:19 PM Vidhi Goel <vidhi_goel=[email-masked]> wrote:
>>>>>