Subject: [tcpm] I-D Action: draft-ietf-tcpm-hystartplusplus-02.txt

From: [email-masked]
Date: Mon, 12 Jul 2021 08:46:04 -0700

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the TCP Maintenance and Minor Extensions WG of the IETF.

        Title           : HyStart++: Modified Slow Start for TCP
        Authors         : Praveen Balasubramanian
                          Yi Huang
                          Matt Olson
	Filename        : draft-ietf-tcpm-hystartplusplus-02.txt
	Pages           : 8
	Date            : 2021-07-12

Abstract:
   This doument describes HyStart++, a simple modification to the slow
   start phase of TCP congestion control algorithms.  Traditional slow
   start can cause overshotting of the ideal send rate and cause large
   packet loss within a round-trip time which results in poor
   performance.  HyStart++ is composed of the delay increase variant of
   HyStart to prevent overshooting of the ideal sending rate, while also
   mitigating poor performance which can result from false positives.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-tcpm-hystartplusplus/

There is also an htmlized version available at:
https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-hystartplusplus-02

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-hystartplusplus-02

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 22 Jul 2021 16:42:18 +0100

Praveen and co-authors.

Thank you for hystart++.
This one's my technical review. I've sent my editorial review in a 
separate posting.

I actually reviewed an earlier draft (balasubrmanian-02), but never got 
round to writing it up for you. I've just reviewed the latest (ietf-02) 
draft, and incorporated my comments about the older revision where still 
relevant.

Cheers

Bob

    ===Technical Comments==

4.2.  Algorithm Details

------------------------------------------------------------------------

  Round can be approximated using sequence numbers as follows:
    ...
    At the start of each round during normal slow start and CSS:

It might be worth stating that using sequence numbers is the /best/ way 
to measure rounds, not just one way. You want every round to contain at 
least one of the ping-ponged segments that were grandfathered by the 
very first segment of the IW (because that will be the segment most 
likely to have the min RTT). For instance, if you measure rounds in 
time, you might underestimate the duration of a round. Then, in some 
rounds, you would risk not seeing one of this ping-pong series at all.

------------------------------------------------------------------------

"where currRTT is the RTT sampled from the incoming ACK"
Need to make it clear this RTT is based on the latest ACK'd packet, not 
the earliest (in contrast to the RTT used for RTO).

------------------------------------------------------------------------

The somewhat opaque clamping of RTTThresh below needs to be justified:

       For rounds where cwnd is at or higher than LOW_CWND and
       N_RTT_SAMPLE RTT samples have been obtained, check if delay
       increase triggers slow start exit
       ...
             RttThresh = clamp(MIN_RTT_THRESH, lastRoundMinRTT / 8, MAX_RTT_THRESH)
             if (currentRoundMinRTT >= (lastRoundMinRTT + RttThresh))
                cssBaselineMinRtt = currentRoundMinRTT
                exit slow start and enter CSS

To aid comprehension for the benefit of the list, I'll separate the 
clamp line into two steps and substitute the default constants:
             RttThresh = clamp(32ms, lastRoundMinRTT, 128ms)
             RttThresh /= 8

To me, it makes good sense without the clamping. In english, that would 
be just:
     "if queuing due to slow start increases the min RTT by more than 
12.5% from one round to the next, exit slow start"

However, taking the default values of the constants, the clamping 
expression adds the following conditions:

    #1) If the base RTT is lower than 32ms, the queue has to be greater
    than 4ms to exit slow start.
    #2) If the base RTT is higher than 128ms, the queue only has to
    exceed 16ms to exit slow start.

I.e. at both extremes, the threshold becomes absolute, rather than a 
fraction of the base RTT.

Possible rationales for the lower condition when the base RTT is small (#1):

    a) an absolute Qdelay threshold is needed because small delay
    measurements are too noisy.
    b) a relative amount of Qdelay would become smaller than humans can
    perceive, so an absolute floor is sufficient.

If (a), this needs to be stated, because different min delay measurement 
techniques can achieve different noise immunity.
I don't agree with (b), because unnecessary delay over a sequence of 
rounds can accumulate to become perceivable.

I can't think of a good rationale for the higher condition (#2). I'm 
sure it was the result of experiments, but it seems to be likely that as 
base RTT increases beyond 128ms, Hystart++ would become increasingly 
sensitive and exit slow start earlier and earlier - unnecessarily (?).

Whatever, the rationale for clamping RttThresh needs to be stated, so 
admins know how to set the parameters for other scenarios (for instance, 
in DCs excessive loss would occur with the given defaults).

------------------------------------------------------------------------

5.  Deployments and Performance Evaluations

It would be useful to see testbed results separated by whether the 
bottleneck before the flow arrives is empty, full, or part-full of traffic.

Although Hystart is default enabled in Linux, it is invariably disabled. 
So, it's misleading to just say Hystart is default enabled, which 
implies it's widely used, when people clearly find it has problems 
(which motivates Hystart++). I found this out through an informal survey 
I did at the Mar'18 IETF in London by asking round the implementers of 
the most prevalent stacks (I would name names if I could find the note I 
later sent to someone or to some list, but I can't find it - sry).

------------------------------------------------------------------------

Regards

Bob

On 12/07/2021 16:46, [email-masked] wrote:
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 22 Jul 2021 16:43:05 +0100

Praveen and co-authors.

Thank you for hystart++.
I've sent a separate posting with technical review comments.

Cheers

Bob

    ==Editorial comments==

      Abstract

s/doument/document/
s/overshotting/overshooting

      1. Intro

s/pefrormance/performance/

"...lead to overshooting the ideal sending rate and cause significant 
packet loss"
Also queuing delay? Is reducing queuing delay (for other flows) also a 
goal of Hystart++?

Last para of intro seems to be about Hystart, and it doesn't make it 
clear how Hystart relates to Hystart++:
* "HyStart++ adds delay increase..."
    Adds to what?
* "After the HyStart delay algorithm finds an exit point, a Conservative 
Slow Start (CSS) phase is used"
   But what about Hystart++? does it do CSS too?

Final sentence is not related to the rest of the para about Hystart, so 
should be a separate para.

      3. Definitions

It seems odd to define some well-known variables (like cwnd), but not 
others like SND.UNA, SND.NXT, etc.

cwnd: Would be useful to define the units assumed in the pseudocode 
later (bytes), because some implementations use segments.
It's confusing for cwnd to use bytes but LOW_CWND to use segments.

      4.2 Algorithm Details

     "The choice of value of L is up to the implementation."
I suggest:
     "The choice of value of L is discussed in RFC 3465."
Reason: RFC3465 doesn't say it's completely up to the implementation - 
it gives recommendations and advice.

"normal slow start"
I appreciate you were asked to use this word, but... I suggest it's best 
to define this phrase because, for instance, the normal slow start for 
Cubic (Hystart) is different from the normal slow start for Reno. It's 
also a problem finding a good word that means the pre-existing thing. 
'Reno slow start' would be more precise than 'normal slow start' (if you 
mean the one in Reno). Because Cubic is more normal than Reno these 
days. And saying 'standard' will soon be ambiguous, because Cubic is on 
its way onto the standards track.

     "If congestion is observed anytime during slow start or CSS, enter 
congestion avoidance."
The word 'congestion' is ambiguous, because it could include delay 
increase. How about:
     "If loss or ECN-marking is observed anytime during slow start or 
CSS, enter congestion avoidance."

    As of the time of writing, HyStart++ has been default enabled for all
    TCP connections in Windows for two years.  The original Hystart has
    been default-enabled for all TCP connections in Linux TCP for a
    decade.

I suggest you give the actual deployment times, so these two sentences 
don't keep having to be updated as the draft ages.
Also, in Linux, Hystart is only implemented for Cubic (the default CC 
module), so Hystart is not default if a non-default CC module is in use. 
How about:

    In Linux, the original Hystart has been default enabled for
    TCP connections using the default congestion control module (Cubic)
    for a decade.

Regards

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 24 Jul 2021 11:13:13 +0100

Praveen & co-authors,

Sry, omitted one more comment about the abstract. Added inline.

Bob

> I've sent a separate posting with technical review comments.

s/prevent/mitigate/
Otherwise over-claiming.

> * "After the HyStart delay algorithm finds an exit point, a 
> Conservative Slow Start (CSS) phase is used"
> Final sentence is not related to the rest of the para about Hystart, 
> so should be a separate para.
>
>
>       3. Definitions
>
> I appreciate you were asked to use this word, but... I suggest it's 
> best to define this phrase because, for instance, the normal slow 
> start for Cubic (Hystart) is different from the normal slow start for 
> Reno. It's also a problem finding a good word that means the 
> pre-existing thing. 'Reno slow start' would be more precise than 
> 'normal slow start' (if you mean the one in Reno). Because Cubic is 
> more normal than Reno these days. And saying 'standard' will soon be 
> ambiguous, because Cubic is on its way onto the standards track.
>
>
> module), so Hystart is not default if a non-default CC module is in 
> use. How about:
>
> Bob Briscoehttp://bobbriscoe.net/

---

From: Praveen Balasubramanian <[email-masked]>
Date: Tue, 27 Jul 2021 00:33:54 +0000

Thanks Bob for the review. We published draft-03 addressing some of your comments.

Inline prefixed by >>>>

From: Bob Briscoe <[email-masked]>
Sent: Thursday, July 22, 2021 8:42 AM
To: Praveen Balasubramanian <[email-masked]>
Cc: [email-masked]; [email-masked]
Subject: [EXTERNAL] Full review of: draft-ietf-tcpm-hystartplusplus-02 (technical)

Praveen and co-authors.

Thank you for hystart++.
This one's my technical review. I've sent my editorial review in a separate posting.

I actually reviewed an earlier draft (balasubrmanian-02), but never got round to writing it up for you. I've just reviewed the latest (ietf-02) draft, and incorporated my comments about the older revision where still relevant.

Cheers

Bob

===Technical Comments==

4.2.  Algorithm Details
________________________________

   ...

   At the start of each round during normal slow start and CSS:
It might be worth stating that using sequence numbers is the /best/ way to measure rounds, not just one way. You want every round to contain at least one of the ping-ponged segments that were grandfathered by the very first segment of the IW (because that will be the segment most likely to have the min RTT). For instance, if you measure rounds in time, you might underestimate the duration of a round. Then, in some rounds, you would risk not seeing one of this ping-pong series at all.

>>>> Added a recommendation for using the sequence number approach.
________________________________

"where currRTT is the RTT sampled from the incoming ACK"
Need to make it clear this RTT is based on the latest ACK'd packet, not the earliest (in contrast to the RTT used for RTO).

>>>> Clarified.
________________________________

      ...

To aid comprehension for the benefit of the list, I'll separate the clamp line into two steps and substitute the default constants:
            RttThresh = clamp(32ms, lastRoundMinRTT, 128ms)
            RttThresh /= 8

To me, it makes good sense without the clamping. In english, that would be just:
    "if queuing due to slow start increases the min RTT by more than 12.5% from one round to the next, exit slow start"

However, taking the default values of the constants, the clamping expression adds the following conditions:
#1) If the base RTT is lower than 32ms, the queue has to be greater than 4ms to exit slow start.
#2) If the base RTT is higher than 128ms, the queue only has to exceed 16ms to exit slow start.
I.e. at both extremes, the threshold becomes absolute, rather than a fraction of the base RTT.

Possible rationales for the lower condition when the base RTT is small (#1):
a) an absolute Qdelay threshold is needed because small delay measurements are too noisy.
b) a relative amount of Qdelay would become smaller than humans can perceive, so an absolute floor is sufficient.
If (a), this needs to be stated, because different min delay measurement techniques can achieve different noise immunity.
I don't agree with (b), because unnecessary delay over a sequence of rounds can accumulate to become perceivable.

I can't think of a good rationale for the higher condition (#2). I'm sure it was the result of experiments, but it seems to be likely that as base RTT increases beyond 128ms, Hystart++ would become increasingly sensitive and exit slow start earlier and earlier - unnecessarily (?).

Whatever, the rationale for clamping RttThresh needs to be stated, so admins know how to set the parameters for other scenarios (for instance, in DCs excessive loss would occur with the given defaults).

>>>> This is worth discussing in the WG meeting tomorrow time permitting. These are a result of experiments. These are fixed clamps because queue build up at the bottleneck has no direct correlation with the base RTT. I agree though that stating the justification is useful and we will look into this for the next revision. We are open to ideas to make these thresholds dynamic.
________________________________

It would be useful to see testbed results separated by whether the bottleneck before the flow arrives is empty, full, or part-full of traffic.

>>>> This is a good suggestion and can be implemented by staggered start of multiple flows. We will incorporate these tests.

Although Hystart is default enabled in Linux, it is invariably disabled. So, it's misleading to just say Hystart is default enabled, which implies it's widely used, when people clearly find it has problems (which motivates Hystart++). I found this out through an informal survey I did at the Mar'18 IETF in London by asking round the implementers of the most prevalent stacks (I would name names if I could find the note I later sent to someone or to some list, but I can't find it - sry).

>>>> I'd like some citations on this versus anecdata if possible before I add that caveat to the text. Do large deployments disable this? I haven't come across this suggestion in any Linux tuning guides to date.

________________________________

Regards

Bob

On 12/07/2021 16:46, [email-masked]<mailto:[email-masked]> wrote:

                          Yi Huang

                          Matt Olson

  Pages           : 8

  Date            : 2021-07-12

Abstract:

https://datatracker.ietf.org/doc/draft-ietf-tcpm-hystartplusplus/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-tcpm-hystartplusplus%2F&data=04%7C01%7Cpravb%40microsoft.com%7Cea743d922a3b42d4af8f08d94d274b02%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637625654367629970%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=wnozkJrwBmG2cbtSB39sg0vYr8cP8%2BBqRnn7bNh6lbM%3D&reserved=0>

https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-hystartplusplus-02<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fhtml%2Fdraft-ietf-tcpm-hystartplusplus-02&data=04%7C01%7Cpravb%40microsoft.com%7Cea743d922a3b42d4af8f08d94d274b02%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637625654367639937%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=SOKqs1Y%2Bi5cNPnn4rv1jlP7rRR%2Fy1iFs9eFlMtxd2ic%3D&reserved=0>

https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-hystartplusplus-02<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Frfcdiff%3Furl2%3Ddraft-ietf-tcpm-hystartplusplus-02&data=04%7C01%7Cpravb%40microsoft.com%7Cea743d922a3b42d4af8f08d94d274b02%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637625654367639937%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=GlGu0C5KV8%2BdYbVeoJzOszS3YmDCnqf5L5GNBEK9eWY%3D&reserved=0>

ftp://ftp.ietf.org/internet-drafts/<https://nam06.safelinks.protection.outlook.com/?url=ftp%3A%2F%2Fftp.ietf.org%2Finternet-drafts%2F&data=04%7C01%7Cpravb%40microsoft.com%7Cea743d922a3b42d4af8f08d94d274b02%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637625654367649886%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=ENURbF%2FuD14Os8t2xQ9oNQVCDITagh%2FAfznG30S%2BjTw%3D&reserved=0>

tcpm mailing list

[email-masked]<mailto:[email-masked]>

https://www.ietf.org/mailman/listinfo/tcpm<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Ftcpm&data=04%7C01%7Cpravb%40microsoft.com%7Cea743d922a3b42d4af8f08d94d274b02%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637625654367649886%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=pXpAocP3fib4kNDsuwgDZAqljB0TUWWJQB4RDWSkT3U%3D&reserved=0>

--

Bob Briscoe                               http://bobbriscoe.net/<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbobbriscoe.net%2F&data=04%7C01%7Cpravb%40microsoft.com%7Cea743d922a3b42d4af8f08d94d274b02%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637625654367649886%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=rRojz5MR9vnVizaQBeqy%2FLdUifSnt%2FjBAD1TBXuukLA%3D&reserved=0>

---

From: Praveen Balasubramanian <[email-masked]>
Date: Tue, 27 Jul 2021 00:50:05 +0000

Inline prefixed by >>>>

From: Bob Briscoe <mailto:[email-masked]> 
Sent: Thursday, July 22, 2021 8:42 AM
To: Praveen Balasubramanian <mailto:[email-masked]>
Cc: mailto:[email-masked]; mailto:[email-masked]
Subject: [EXTERNAL] Full review of: draft-ietf-tcpm-hystartplusplus-02 (technical)

Praveen and co-authors.

Thank you for hystart++. 

Cheers

Bob
===Technical Comments==

4.2.  Algorithm Details
________________________________________

>>>> Added a recommendation for using the sequence number approach. 
________________________________________

>>>> Clarified.
________________________________________

>>>> This is worth discussing in the WG meeting tomorrow time permitting. These are a result of experiments. These are fixed clamps because queue build up at the bottleneck has no direct correlation…
________________________________________

>>>> I'd like some citations on this versus anecdata if possible before I add that caveat to the text. Do large deployments disable this? I haven't come across this suggestion in any Linux tuning gui…

________________________________________

Regards

Bob
On 12/07/2021 16:46, mailto:[email-masked] wrote:

Abstract:

The IETF datatracker status page for this draft is:
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-tcpm-hystartplusplus%2F&data=04%7C01%7Cpravb%40microsoft.com%7Cea743d922a3b42d4af8f08d94d274b02%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637625654367629970%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=wnozkJrwBmG2cbtSB39sg0vYr8cP8%2BBqRnn7bNh6lbM%3D&reserved=0

There is also an htmlized version available at:
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fhtml%2Fdraft-ietf-tcpm-hystartplusplus-02&data=04%7C01%7Cpravb%40microsoft.com%7Cea743d922a3b42d4af8f08d94d274b02%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637625654367639937%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=SOKqs1Y%2Bi5cNPnn4rv1jlP7rRR%2Fy1iFs9eFlMtxd2ic%3D&reserved=0

A diff from the previous version is available at:
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Frfcdiff%3Furl2%3Ddraft-ietf-tcpm-hystartplusplus-02&data=04%7C01%7Cpravb%40microsoft.com%7Cea743d922a3b42d4af8f08d94d274b02%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637625654367639937%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=GlGu0C5KV8%2BdYbVeoJzOszS3YmDCnqf5L5GNBEK9eWY%3D&reserved=0

Internet-Drafts are also available by anonymous FTP at:
https://nam06.safelinks.protection.outlook.com/?url=ftp%3A%2F%2Fftp.ietf.org%2Finternet-drafts%2F&data=04%7C01%7Cpravb%40microsoft.com%7Cea743d922a3b42d4af8f08d94d274b02%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637625654367649886%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=ENURbF%2FuD14Os8t2xQ9oNQVCDITagh%2FAfznG30S%2BjTw%3D&reserved=0

_______________________________________________
tcpm mailing list
mailto:[email-masked]
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Ftcpm&data=04%7C01%7Cpravb%40microsoft.com%7Cea743d922a3b42d4af8f08d94d274b02%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637625654367649886%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=pXpAocP3fib4kNDsuwgDZAqljB0TUWWJQB4RDWSkT3U%3D&reserved=0

-- 
________________________________________________________________
Bob Briscoe                               https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbobbriscoe.net%2F&data=04%7C01%7Cpravb%40microsoft.com%7Cea743d922a3b42d4af8f08d94d274b02%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637625654367649886%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=rRojz5MR9vnVizaQBeqy%2FLdUifSnt%2FjBAD1TBXuukLA%3D&reserved=0

---

From: Praveen Balasubramanian <[email-masked]>
Date: Tue, 27 Jul 2021 01:24:36 +0000

I am having issues replying with inline replies to your email so I am copy pasting.

>> s/doument/document/
Fixed all typos except this one. Will fix in next revision.

>> Is reducing queuing delay (for other flows) also a goal of Hystart++?
It's not an explicit goal. Since this can be used with a loss based congestion controller, we cannot make a blanket statement about reducing queueing delay.

>> Last para of intro seems to be about Hystart, and it doesn't make it clear how Hystart relates to Hystart++:
Rewrote most of this text to make it clear.

>> It seems odd to define some well-known variables (like cwnd), but not others like SND.UNA, SND.NXT, etc.
RFC 5681 doesn't define them either but uses them.

>> cwnd: Would be useful to define the units assumed in the pseudocode later (bytes),
These are verbatim from RFC 5681. We can add a clarification if it helps in the pseudocode section.

>> I suggest:     "The choice of value of L is discussed in RFC 3465."
Fixed to convey this.

>>"normal slow start"
Replaced by standard slow start and a reference to RFC 5681.

>> The word 'congestion' is ambiguous, because it could include delay increase. How about:"If loss or ECN-marking is observed anytime during slow start or CSS, enter congestion avoidance."
Fixed.

>> I suggest you give the actual deployment times, so these two sentences don't keep having to be updated as the draft ages
We have made some edits. Will make this section up to date as we get more data on draft-02.

>> Also, in Linux, Hystart is only implemented for Cubic
Clarified this and added a reference to RFC 8312.

From: Bob Briscoe <[email-masked]>
Sent: Thursday, July 22, 2021 8:43 AM
To: Praveen Balasubramanian <[email-masked]>
Cc: [email-masked]; [email-masked]
Subject: [EXTERNAL] Full review of: draft-ietf-tcpm-hystartplusplus-02 (editorial)

Praveen and co-authors.

Thank you for hystart++.

Cheers

Bob
==Editorial comments==

Abstract
s/doument/document/
>>>> Fixed

1. Intro

"...lead to overshooting the ideal sending rate and cause significant packet loss"
Also queuing delay? Is reducing queuing delay (for other flows) also a goal of Hystart++?

Last para of intro seems to be about Hystart, and it doesn't make it clear how Hystart relates to Hystart++:
* "HyStart++ adds delay increase..."
   Adds to what?
* "After the HyStart delay algorithm finds an exit point, a Conservative Slow Start (CSS) phase is used"
  But what about Hystart++? does it do CSS too?

Final sentence is not related to the rest of the para about Hystart, so should be a separate para.
3. Definitions
It seems odd to define some well-known variables (like cwnd), but not others like SND.UNA, SND.NXT, etc.

cwnd: Would be useful to define the units assumed in the pseudocode later (bytes), because some implementations use segments.
It's confusing for cwnd to use bytes but LOW_CWND to use segments.

4.2 Algorithm Details

    "The choice of value of L is up to the implementation."
I suggest:
    "The choice of value of L is discussed in RFC 3465."
Reason: RFC3465 doesn't say it's completely up to the implementation - it gives recommendations and advice.

"normal slow start"
I appreciate you were asked to use this word, but... I suggest it's best to define this phrase because, for instance, the normal slow start for Cubic (Hystart) is different from the normal slow start for Reno. It's also a problem finding a good word that means the pre-existing thing. 'Reno slow start' would be more precise than 'normal slow start' (if you mean the one in Reno). Because Cubic is more normal than Reno these days. And saying 'standard' will soon be ambiguous, because Cubic is on its way onto the standards track.

    "If congestion is observed anytime during slow start or CSS, enter congestion avoidance."
The word 'congestion' is ambiguous, because it could include delay increase. How about:
    "If loss or ECN-marking is observed anytime during slow start or CSS, enter congestion avoidance."

   decade.

I suggest you give the actual deployment times, so these two sentences don't keep having to be updated as the draft ages.
Also, in Linux, Hystart is only implemented for Cubic (the default CC module), so Hystart is not default if a non-default CC module is in use. How about:
In Linux, the original Hystart has been default enabled for
TCP connections using the default congestion control module (Cubic)
for a decade.

Regards

Bob

                          Yi Huang

                          Matt Olson

  Pages           : 8

  Date            : 2021-07-12

Abstract:

https://datatracker.ietf.org/doc/draft-ietf-tcpm-hystartplusplus/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-tcpm-hystartplusplus%2F&data=04%7C01%7Cpravb%40microsoft.com%7C7d87872da4bc44b8b16508d94d276713%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637625654471940975%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=6%2B5UCWDRoooESeKwuY236LzMZeetnuWyVsnau4GLnWI%3D&reserved=0>

https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-hystartplusplus-02<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fhtml%2Fdraft-ietf-tcpm-hystartplusplus-02&data=04%7C01%7Cpravb%40microsoft.com%7C7d87872da4bc44b8b16508d94d276713%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637625654471950867%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=XiUnldnrVGSab7DqQLQ6qRXZCVYtubDQ5d1Zxzvi7Ug%3D&reserved=0>

https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-hystartplusplus-02<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Frfcdiff%3Furl2%3Ddraft-ietf-tcpm-hystartplusplus-02&data=04%7C01%7Cpravb%40microsoft.com%7C7d87872da4bc44b8b16508d94d276713%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637625654471960830%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=AJiW4Hj1Z2eFeiwqmYFpk2jt1N352O2LxbE3j8q6an8%3D&reserved=0>

ftp://ftp.ietf.org/internet-drafts/<https://nam06.safelinks.protection.outlook.com/?url=ftp%3A%2F%2Fftp.ietf.org%2Finternet-drafts%2F&data=04%7C01%7Cpravb%40microsoft.com%7C7d87872da4bc44b8b16508d94d276713%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637625654471960830%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=XywaX58KX%2BVkDTHOVEQDHKbR0NqozqtvVeLk5l4S0YI%3D&reserved=0>

tcpm mailing list

https://www.ietf.org/mailman/listinfo/tcpm<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Ftcpm&data=04%7C01%7Cpravb%40microsoft.com%7C7d87872da4bc44b8b16508d94d276713%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637625654471970796%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=Tsk%2FmLAKRkI2U3N95ccB%2BZrILbXy1FH2ZuymO3mwM38%3D&reserved=0>

--

Bob Briscoe                               http://bobbriscoe.net/<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbobbriscoe.net%2F&data=04%7C01%7Cpravb%40microsoft.com%7C7d87872da4bc44b8b16508d94d276713%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637625654471980743%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=p4uRO6hnVy7DahbDd4fz9%2Bd83pBTi4I9%2B8uWe9OeZUI%3D&reserved=0>

---

From: Praveen Balasubramanian <[email-masked]>
Date: Tue, 27 Jul 2021 01:29:59 +0000

>> It might be worth stating that using sequence numbers is the /best/ way to measure rounds

>> Need to make it clear this RTT is based on the latest ACK'd packet

Clarified.

>> Although Hystart is default enabled in Linux, it is invariably disabled

Praveen and co-authors.

Thank you for hystart++.

Cheers

Bob

===Technical Comments==

4.2.  Algorithm Details

   ...

      ...

Regards

Bob

                          Yi Huang

                          Matt Olson

  Pages           : 8

  Date            : 2021-07-12

Abstract:

tcpm mailing list

--

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 28 Jul 2021 00:37:19 +0100

Any large server operator out there who are using Cubic,
If you're willing to state whether or not your operations disable 
Hystart, pls do so in reply.
Then Praveen can cite this mailing list thread in the Hystart++ draft, 
as requested below.

If you are willing to reply privately, I would be willing to keep your 
confidences, and provide an anonymized result to the list.

Cheers

Bob

---

From: Yuchung Cheng <[email-masked]>
Date: Tue, 27 Jul 2021 16:46:21 -0700

Wait -- how is hystart "invariably disabled" in Linux (cubic)!?

What data indicates that

On Tue, Jul 27, 2021 at 4:37 PM Bob Briscoe <[email-masked]> wrote:

> If you're willing to state whether or not your operations disable Hystart,
> pls do so in reply.
> Then Praveen can cite this mailing list thread in the Hystart++ draft, as
> requested below.
>

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 28 Jul 2021 11:19:44 +0100

Yuchung,

It was during a Mar 2018 ad hoc workshop Jana had organized at the 
London IETF entitled 'BBR and the intersection with other work". I can't 
remember why I needed to know at the time, but during the break I 
approached the major server operators individually, established whether 
they used Cubic, and if so asked whether they used Hystart or disabled 
it. It's hard to anonymize the results, because IMMSMC all those that 
used Cubic said they disabled Hystart.

If this isn't correct, then maybe people who replied at the time thought 
they were being asked something else. Or maybe it was correct then but 
isn't now.

Bob

On 28/07/2021 00:46, Yuchung Cheng wrote:
> On Tue, Jul 27, 2021 at 4:37 PM Bob Briscoe <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Then Praveen can cite this mailing list thread in the Hystart++
>     draft, as requested below.
>
>     If you are willing to reply privately, I would be willing to keep
>     your confidences, and provide an anonymized result to the list.
>
>     Cheers
>
>
>
>     Bob
>
>>     Although Hystart is default enabled in Linux, it is invariably
>>     disabled. So, it's misleading to just say Hystart is default
>>     enabled, which implies it's widely used, when people clearly find
>>     it has problems (which motivates Hystart++). I found this out
>>     through an informal survey I did at the Mar'18 IETF in London by
>>     asking round the implementers of the most prevalent stacks (I
>>     would name names if I could find the note I later sent to someone
>>     or to some list, but I can't find it - sry).
>>>>>>
>>>>>>     I'd like some citations on this versus anecdata if possible
>>>>>>     before I add that caveat to the text. Do large deployments
>>>>>>     disable this? I haven't come across this suggestion in any
>>>>>>     Linux tuning guides to date.
>
>     -- 
>     Bob Briscoehttp://bobbriscoe.net/  <http://bobbriscoe.net/>
>
>     [email-masked] <mailto:[email-masked]>
>     <https://www.ietf.org/mailman/listinfo/tcpm>
>

---

From: Christian Huitema <[email-masked]>
Date: Wed, 28 Jul 2021 09:46:52 -0700

I certainly do not qualify as a "large operator", but when implementing 
Cubic for QUIC I found that Hystart was critical for performance. 
Specifically, the classical slow-start is very prone to overshooting the 
capacity of the path, which causes large batches of errors. With 
web-like traffic, these batches of errors cause increase latency for the 
first transactions in a session, and thus a drop in perceived quality. 
Hystart solves that. With Hystart, I observed that a large fraction of 
sessions do not experience any packet loss at all.

As an aside, we should strive for this suppression of packet losses. 
That's actually a big reason for moving from Cubic to BBR. Hystart 
suppresses the losses during the initial phase of the connection, but 
Cubic still relies on periodically testing the limit of path capacity 
and causing losses during the subsequent phase. BBR's probing for 
bottleneck bandwidth is much more conservative, does not cause such 
losses. It might be possible to adapt Cubic to also not cause losses, 
for example by ending an epoch early if too many CE marks are received 
or if the RTT increases. That would be worth trying.

-- Christian Huitema

On 7/28/2021 3:19 AM, Bob Briscoe wrote:
> London IETF entitled 'BBR and the intersection with other work". I 
> can't remember why I needed to know at the time, but during the break 
> I approached the major server operators individually, established 
> whether they used Cubic, and if so asked whether they used Hystart or 
> disabled it. It's hard to anonymize the results, because IMMSMC all 
> those that used Cubic said they disabled Hystart.
>
> If this isn't correct, then maybe people who replied at the time 
> thought they were being asked something else. Or maybe it was correct 
> then but isn't now.
>
>
> Bob
>
>
>>     -- ________________________________________________________________

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 28 Jul 2021 16:35:08 -0700

same experience with Chrisitian

For more than half a decade Google traffic used Linux CUBIC Hystart. This
includes google.com and YouTube public Internet traffic, as well as
internal Google TCP traffic. When we deployed pacing around 2013 we
disabled the Hystart ACK-train mechanism (that Hystart++ prohibits now) as
it causes false SS exit. But we continued to use the stock Hystart Delay
mechanism. ~2014-15 we switched to BBR which uses a different startup more
robust to delay jitters. But IMO the simpler hystart mitigating SS
over-shoot is still highly valuable for Cubic and other congestion
controls, as Internet bandwidth continues to hike.

Did those providers disable hystart b/c of its poor interaction between the
ack-train mode and pacing? Maybe it's time to disable the ack-train
approach in the upstream Linux when hystart++ is standardized.

On Wed, Jul 28, 2021 at 9:47 AM Christian Huitema <[email-masked]>
wrote:

> capacity of the path, which causes large batches of errors. With web-like
> traffic, these batches of errors cause increase latency for the first
> transactions in a session, and thus a drop in perceived quality. Hystart
> solves that. With Hystart, I observed that a large fraction of sessions do
> not experience any packet loss at all.
>
> suppresses the losses during the initial phase of the connection, but Cubic
> still relies on periodically testing the limit of path capacity and causing
> losses during the subsequent phase. BBR's probing for bottleneck bandwidth
> is much more conservative, does not cause such losses. It might be possible
> to adapt Cubic to also not cause losses, for example by ending an epoch
> early if too many CE marks are received or if the RTT increases. That would
> be worth trying.
>
> -- Christian Huitema
> It was during a Mar 2018 ad hoc workshop Jana had organized at the London
> IETF entitled 'BBR and the intersection with other work". I can't remember
> why I needed to know at the time, but during the break I approached the
> major server operators individually, established whether they used Cubic,
> and if so asked whether they used Hystart or disabled it. It's hard to
> anonymize the results, because IMMSMC all those that used Cubic said they
> disabled Hystart.
>
> <mailto:[email-masked]> <[email-masked]>> wrote:
>
> <http://bobbriscoe.net/>
>     [email-masked] <mailto:[email-masked]> <[email-masked]>
> tcpm mailing [email-masked]://www.ietf.org/mailman/listinfo/tcpm
>
>

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 29 Jul 2021 17:28:14 +0100

Yuchung,

On 29/07/2021 00:35, Yuchung Cheng wrote:
> For more than half a decade Google traffic used Linux CUBIC Hystart. 
> This includes google.com <http://google.com/> and YouTube public 
> Internet traffic, as well as internal Google TCP traffic. When we 
> deployed pacing around 2013 we disabled the Hystart ACK-train 
> mechanism (that Hystart++ prohibits now) as it causes false SS 
> exit. But we continued to use the stock Hystart Delay mechanism. 
> ~2014-15 we switched to BBR which uses a different startup more robust 
> to delay jitters.

[BB] You or Neal (sry can't remember which) were one of those I surveyed 
who said they had disabled Hystart. So perhaps this was a 
misunderstanding over the ACK-train part.

> But IMO the simpler hystart mitigating SS over-shoot is still highly 
> valuable for Cubic and other congestion controls, as Internet 
> bandwidth continues to hike.
>
> Did those providers disable hystart b/c of its poor interaction 
> between the ack-train mode and pacing?

[BB] I didn't ask them why.

Bob

> Maybe it's time to disable the ack-train approach in the upstream 
> Linux when hystart++ is standardized.
>
> On Wed, Jul 28, 2021 at 9:47 AM Christian Huitema <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     I certainly do not qualify as a "large operator", but when
>     implementing Cubic for QUIC I found that Hystart was critical for
>     performance. Specifically, the classical slow-start is very prone
>     to overshooting the capacity of the path, which causes large
>     batches of errors. With web-like traffic, these batches of errors
>     cause increase latency for the first transactions in a session,
>     and thus a drop in perceived quality. Hystart solves that. With
>     Hystart, I observed that a large fraction of sessions do not
>     experience any packet loss at all.
>
>     As an aside, we should strive for this suppression of packet
>     losses. That's actually a big reason for moving from Cubic to BBR.
>     Hystart suppresses the losses during the initial phase of the
>     connection, but Cubic still relies on periodically testing the
>     limit of path capacity and causing losses during the subsequent
>     phase. BBR's probing for bottleneck bandwidth is much more
>     conservative, does not cause such losses. It might be possible to
>     adapt Cubic to also not cause losses, for example by ending an
>     epoch early if too many CE marks are received or if the RTT
>     increases. That would be worth trying.
>
>     -- Christian Huitema
>
>>     It was during a Mar 2018 ad hoc workshop Jana had organized at
>>     the London IETF entitled 'BBR and the intersection with other
>>     work". I can't remember why I needed to know at the time, but
>>     during the break I approached the major server operators
>>     individually, established whether they used Cubic, and if so
>>     asked whether they used Hystart or disabled it. It's hard to
>>     anonymize the results, because IMMSMC all those that used Cubic
>>     said they disabled Hystart.
>>
>>     thought they were being asked something else. Or maybe it was
>>     correct then but isn't now.
>>
>>
>>     Bob
>>
>>
>>>     <mailto:[email-masked]> <mailto:[email-masked]>
>>>     <http://bobbriscoe.net/> <http://bobbriscoe.net/>
>>>
>>>     [email-masked] <mailto:[email-masked]> <mailto:[email-masked]>
>>>     <mailto:[email-masked]>
>>     https://www.ietf.org/mailman/listinfo/tcpm  <https://www.ietf.org/mailman/listinfo/tcpm>
>

---

From: Yuchung Cheng <[email-masked]>
Date: Thu, 29 Jul 2021 09:39:40 -0700

On Thu, Jul 29, 2021 at 9:28 AM Bob Briscoe <[email-masked]> wrote:

> robust to delay jitters.
> who said they had disabled Hystart. So perhaps this was a misunderstanding
> over the ACK-train part.
>
Yes this must be mis-communication. Since that was in 2018, it's either "we
don't use hystart b/c BBR doesn't use hystart" or "we don't use hystart
ack-train when hystart was used in Cubic before BBR".

> valuable for Cubic and other congestion controls, as Internet bandwidth
> continues to hike.
>
> Did those providers disable hystart b/c of its poor interaction between
> the ack-train mode and pacing?
>
>

---

From: Neal Cardwell <[email-masked]>
Date: Thu, 29 Jul 2021 12:58:51 -0400

On Thu, Jul 29, 2021 at 12:40 PM Yuchung Cheng <[email-masked]> wrote:

> Yes this must be mis-communication. Since that was in 2018, it's either
> "we don't use hystart b/c BBR doesn't use hystart" or "we don't use hystart

Yes, my sense of our experience with CUBIC Hystart matches Yuchung's,
including the likely sources of mis-communication.

thanks,
neal

---

From: Junho Choi <[email-masked]>
Date: Sat, 31 Jul 2021 00:02:00 -0700

Currently Cloudflare's QUIC library (quiche) and its production QUIC
traffic uses CUBIC (rfc8312bis-02) and HyStart++ (draft02) by default.
(https://blog.cloudflare.com/cubic-and-hystart-support-in-quiche/).

-- 
Junho Choi <junho dot choi at gmail.com> | https://saturnsoft.net