Subject: [tcpm] RTO seeding based on SYN RTT

From: Yuchung Cheng <[email-masked]>
Date: Fri, 16 Jul 2010 10:38:14 -0700

Hello folks,

I submitted a draft on seeding RTO based on SYN (or SYNACK) RTT.
https://tools.ietf.org/html/draft-ycheng-tcpm-rtosynrtt-00

"This document proposes seeding the retransmission timer by the RTT
   measured during the TCP connection establishment and updates the
   recommended approach in [RFC3390]. It discusses the motivation, the
   TCP  modification, and the experimental result."

Ideally I would like present the draft in tcpm. Since tcpm is not meeting so
I am presenting in iccrg on Friday but I have made the slides online at
http://code.google.com/speed/protocols/rtoseedsynrtt.ppt

Comments/questions are welcome

Yuchung

---

From: "Eddy, Wesley M. (GRC-MS00)[ASRC AEROSPACE CORP]" <[email-masked]>
Date: Fri, 16 Jul 2010 13:36:16 -0500

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On Behalf Of
>Yuchung Cheng
>Sent: Friday, July 16, 2010 1:38 PM
>To: [email-masked] Extensions
>Subject: [tcpm] RTO seeding based on SYN RTT
>
>Hello folks,
>
>Ideally I would like present the draft in tcpm. Since tcpm is not
>meeting so I am presenting in iccrg on Friday but I have made the slides
>online at

This looks like a good idea to me; at least for Experimental,
and possibly higher.  For standards track, I think there would
ideally be a little bit deeper analysis or experimentation
about potential downsides versus the current standard.  The
benefit is well motivated and understood, I think.

The plot on slide 7 is really informative.  One question, which
I don't know if the data is available to answer, is why the SYN
RTT and first data RTT sometimes differ by so much, with the SYN
sometimes being an order of magnitude faster.  That seemed odd.
It would be interesting to know in these cases what the average
RTT (or distribution of RTTs) measured for the connection was.

--
Wes Eddy
MTI Systems

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 16 Jul 2010 12:24:49 -0700

On Fri, Jul 16, 2010 at 11:36 AM, Eddy, Wesley M. (GRC-MS00)[ASRC AEROSPACE
> RTT (or distribution of RTTs) measured for the connection was.

That's a very good question. I can collect more data

Some usual suspects (yet to be verified):
a) browsers opening many connections at roughly the same time [1,2] plus
middle boxes having large buffers (>1sec) [3]. We (Google) have services
using by multiple domains on the same page (ex maps)

b) mobile connections where SYN RTT is often longer

[1]
http://www.stevesouders.com/blog/2008/03/20/roundup-on-parallel-connections/
[2] the network tab at http://www.browserscope.org/
[3] "Characterizing residential broadband networks"
http://www.mpi-sws.org/~mdischin/papers/07_imc_bb.pdf

---

From: Marco Mellia <[email-masked]>
Date: Fri, 16 Jul 2010 23:08:03 +0200

> a) browsers opening many connections at roughly the same time [1,2] 
> plus middle boxes having large buffers (>1sec) [3]. We (Google) have 
> services using by multiple domains on the same page (ex maps)
>

I may be misunderstand something here. The figure on slide 7 shows that 
there are cases in which the SYN RTT is much smaller than the 1st DATA 
RTT, i.e., all the dots above the diagonal.

That can be due to
a) some TCP stack using delayed ACK which artificially inflate the 1st 
DATA RTT
b) slow links along the path, for which the transmission delay is not 
negligible. A MSS=1460B long packet goes through a 256kbps ADSL link 
takes in about 50ms...

I vote for the second option, given that the curve RTO follows exactly 
the same shape...
 From slide 5, RTO = SRTT+ max(G, 4*RTTVAR) + D
Where D is the offset to consider an MSS segment going through a 56kbps 
link...

A part from this, I personally like this proposal, since today access 
link speed is much faster then it used to be when RC 2988 was specified.
I have some questions that might be interesting to investigate in case 
you have some data.
What happens on the client-server side? You are observing the 
server-client side of the connection, isn't it?
Do you have any idea of how often the 1st data segment is actually lost?
When I was working on the same issue some year ago, we estimated this 
figure. I'm just curious to know if this has changed.
See
Mellia, M. Meo, C. Casetti
TCP Smart Framing: a Segmentation Algorithm to Reduce TCP latency
IEEE/ACM Transactions on Networking, Vol. 13, No. 2, pp. 316-329, ISSN: 
1063-6692, April 2005
http://portal.acm.org/citation.cfm?id=1066626.1066634

Ciao,                    /\/\/\rco

+-----------------------------------+
| Marco Mellia - Assistant Professor|
| Skypeid: mgmellia                 |
| Tel: +39-011-564-4173             |
| Cel: +39-340-9674888              |   /"\  .. . . . . . . . . . . . .
| Politecnico di Torino             |   \ /  . ASCII Ribbon Campaign  .
| Corso Duca degli Abruzzi 24       |    X   .- NO HTML/RTF in e-mail .
| Torino - 10129 - Italy            |   / \  .- NO Word docs in e-mail.
|http://www.telematica.polito.it    |        .. . . . . . . . . . . . .
+-----------------------------------+
The box said "Requires Windows 95 or Better." So I installed Linux.

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 19 Jul 2010 11:08:37 -0700

On Fri, Jul 16, 2010 at 2:08 PM, Marco Mellia <[email-masked]> wrote:

> there are cases in which the SYN RTT is much smaller than the 1st DATA RTT,
> i.e., all the dots above the diagonal.
>
> Sorry I misread Eddy's email: I was addressing the cases where SYN RTT is
much larger than 1st data RTT, which is also very interesting.

>  That can be due to
> a) some TCP stack using delayed ACK which artificially inflate the 1st DATA
> RTT
>
Indeed, the 1st data RTT samples are spread across a 200ms "alley" (note the
plot is on log scale) which corresponds well to the Windows/Linux delayed
ACK timer values.

> negligible. A MSS=1460B long packet goes through a 256kbps ADSL link takes
> in about 50ms...
>
It's possible. But I suspect another possibility is simply congestion since
browser multiplex a few connections to download HTML resources. Also WiFi
adds more jitter at the last mile?

> I vote for the second option, given that the curve RTO follows exactly the
> same shape...
> A part from this, I personally like this proposal, since today access link
> speed is much faster then it used to be when RC 2988 was specified.
> I have some questions that might be interesting to investigate in case you
> have some data.
> What happens on the client-server side? You are observing the server-client
> side of the connection, isn't it?

> | http://www.telematica.polito.it   |        .. . . . . . . . . . . . .

---

From: Marco Mellia <[email-masked]>
Date: Tue, 20 Jul 2010 11:14:55 +0200

>
>     I may be misunderstand something here. The figure on slide 7 shows
>     that there are cases in which the SYN RTT is much smaller than the
>     1st DATA RTT, i.e., all the dots above the diagonal.
>
> Sorry I misread Eddy's email: I was addressing the cases where SYN RTT 
> is much larger than 1st data RTT, which is also very interesting.
>
>     That can be due to
>     a) some TCP stack using delayed ACK which artificially inflate the
>     1st DATA RTT
>
> Indeed, the 1st data RTT samples are spread across a 200ms "alley" 
> (note the plot is on log scale) which corresponds well to the 
> Windows/Linux delayed ACK timer values.
>
>     b) slow links along the path, for which the transmission delay is
>     not negligible. A MSS=1460B long packet goes through a 256kbps
>     ADSL link takes in about 50ms...
>
> It's possible. But I suspect another possibility is simply congestion 
> since browser multiplex a few connections to download HTML resources. 
> Also WiFi adds more jitter at the last mile?
>

Aren't you looking at the Server->Client->Server RTT?
Then, congestion shouldn't be there, unless google has poor servers ;)

M

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 21 Jul 2010 14:33:59 -0700

> negligible. A MSS=1460B long packet goes through a 256kbps ADSL link takes

I bucketed the IPs based on the bandwidth  (<100kbps, 100kbps - 1Mbps, 2Mbps
- 5Mbps etc) using our internal IP bandwidth database.  In all buckets,
there are connections with data RTTs >> SYN RTTs. This suggests there are
additional factors beside a) and b).

> Do you have any idea of how often the 1st data segment is actually lost?
We are observing ~2.5% retransmission rate from server -> client for the
experiment in the slides (slide 8), and ~2% client->server based on
measurements from our Chrome browsers. But since OSes like Windows or Linux
retransmit after 3 second, so there are certain degree of spurious
retransmissions.

---

From: Xiangsong Cui <[email-masked]>
Date: Thu, 22 Jul 2010 09:25:15 +0800

Hi Yuchung,

Thanks for your good job!

I am also interested in this topic, and I am thinking about what is the real reason
of different RTT, the packet type (SYN packet vs data packet) or the packet size?

So could you please do such a test:
in a certain bandwidth network, test the RTT of data packets with different size,
between the same two endpoint, so we can compare the RTT of  all cases.
(e.g. configure the MSS on both hosts with value 1 B, 10B, 100B, 1000B...)

I think this is important, if the RTT or one-way-delay of TCP packet is size-dependent.
because, if that was true, when host A sends TCP packet to host B,
if host B acknowledges the packet with an only ACK packet (small size, without data), 
then the RTT measured at host A is a smaller value;
if host B acknowledges the packet with an ACK-inside data packet (big size packet),
then the RTT measured at host A is a bigger value.

(Maybe we can also test the one way delay of alone ACK packet and ACK-inside data packet...)

If that happens and host B send data segments to host A intermittently, then host A 
would meet many unnecessary Retransmission timer expirements?

Thanks and best regards

Xiangsong

----- Original Message ----- 
From: "Yuchung Cheng" <[email-masked]>
To: "Marco Mellia" <[email-masked]>
Cc: <[email-masked]>
Sent: Thursday, July 22, 2010 5:33 AM
Subject: Re: [tcpm] RTO seeding based on SYN RTT

> retransmit after 3 second, so there are certain degree of spurious

--------------------------------------------------------------------------------

> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 26 Jul 2010 18:29:47 +0200

Hi Xiangsong,

I am not sure if such a test is needed b/c packet size definitely
affects the RTT on the same link.  There are studies showing "packet
types" or packets with different TCP flags have different RTTs (on the
same path). For example, SYN packets over cellular networks can
sometimes take much longer than subsequent data RTTs. In fact quite a
few data points in slide 7 below the x=y line are from mobile ISPs. In
addition, SYN packets could have extra delay due to router table or
ARP lookups.

But I agree if the receiver's ACK carries payload and is going through
narrowband, the current cushion D needs to be adjusted.

Yu-chung

On Thu, Jul 22, 2010 at 3:25 AM, Xiangsong Cui <[email-masked]> wrote:
> if host B acknowledges the packet with an only ACK packet (small size,
> without data), then the RTT measured at host A is a smaller value;
> If that happens and host B send data segments to host A intermittently, then
> host A would meet many unnecessary Retransmission timer expirements?
>
>
> Thanks and best regards
>
> Xiangsong
>
> ----- Original Message ----- From: "Yuchung Cheng" <[email-masked]>
>>> DATA
>>> takes
>>> link
>>> you

---

From: Xiangsong Cui <[email-masked]>
Date: Tue, 27 Jul 2010 01:52:35 +0800

Hi Yuchung,

Thanks for your response,

I also ever thought this point, but, in your previous mail, you mentioned:

>> I bucketed the IPs based on the bandwidth ?(<100kbps, 100kbps - 1Mbps,
>> 2Mbps
>> - 5Mbps etc) using our internal IP bandwidth database. ?In all buckets,

So, it seems that, if we excludes the ARP delay of SYN, we would get a more
bigger minus, I mean maybe data RTT >>> SYN RTT.
Considering you test in 5Mbps bandwidth, the additional time that a 1500 
Bytes data packet needs is (1500*8) / (5*1000*1000) = 2.4 ms, this is a 
really small value!

So I think, if we can measure the different RTT values (for different size
data packets), we can compare the different RTT values in different conditions,
and maybe find some new thread? (i.e. the element "additional factors beside
a) and b)" )

Best Regards,
Xiangsong

----- 原邮件 -----
发件人: Yuchung Cheng <[email-masked]>
日期: 星期二, 七月 27日, 2010 上午0:30
主题: Re: [tcpm] RTO seeding based on SYN RTT
收件人: Xiangsong Cui <[email-masked]>
抄送: Marco Mellia <[email-masked]>, [email-masked]

> > I am also interested in this topic, and I am thinking about what 
> is the real
> > reason
> > of different RTT, the packet type (SYN packet vs data packet) or 
> the packet
> > size?
> >
> > in a certain bandwidth network, test the RTT of data packets 
> with different
> > size,
> > between the same two endpoint, so we can compare the RTT of ?all 
> cases.> (e.g. configure the MSS on both hosts with value 1 B, 10B, 
> 100B, 1000B...)
> >
> >
> > I think this is important, if the RTT or one-way-delay of TCP 
> packet is
> > size-dependent.
> > if host B acknowledges the packet with an only ACK packet (small 
> size,> without data), then the RTT measured at host A is a smaller 
> value;> if host B acknowledges the packet with an ACK-inside data 
> packet (big size
> > packet),
> > (Maybe we can also test the one way delay of alone ACK packet 
> and ACK-inside
> > data packet...)
> >
> >
> > ----- Original Message ----- From: "Yuchung Cheng" 
> <[email-masked]>> To: "Marco Mellia" <[email-masked]>
> > Cc: <[email-masked]>
> >> On Fri, Jul 16, 2010 at 2:08 PM, Marco Mellia 
> <[email-masked]>>> wrote:
> >>
> >> I bucketed the IPs based on the bandwidth ?(<100kbps, 100kbps - 
> 1Mbps,>> 2Mbps
> >> - 5Mbps etc) using our internal IP bandwidth database. ?In all 
> buckets,>> there are connections with data RTTs >> SYN RTTs. This 
> suggests there are
> >>> A part from this, I personally like this proposal, since today 
> access>>> link
> >>> speed is much faster then it used to be when RC 2988 was 
> specified.>>> I have some questions that might be interesting to 
> investigate in case
> >>> you
> >>> have some data.
> >> measurements from our Chrome browsers. But since OSes like 
> Windows or
> >> Linux

---

From: Alexander Zimmermann <[email-masked]>
Date: Thu, 29 Jul 2010 11:01:24 +0200

Hi,

I read the doc and its looks like a good idea to me too.

One quick note. Supposed, the doc become a WG item, I highly
recommend that we integrate the scheme directly  in RFC2988bis.
I don't want to have one RFC for lowering initRTO, one for lowering
minimum RTO, one for seeding RTO based on SYN RTT, ...

Alex

Am 16.07.2010 um 19:38 schrieb Yuchung Cheng:

> Ideally I would like present the draft in tcpm. Since tcpm is not meeting so I am presenting in iccrg on Friday but I have made the slides online at

---

From: Yuchung Cheng <[email-masked]>
Date: Thu, 29 Jul 2010 16:02:15 +0200

I briefly discussed the possibility to merge the two drafts with the
2988bis authors in June but we didn't reach a consensus. One reason is
that this draft updates 5681 in addition to 2988. Since the submission
deadline was less than two weeks, I decided to send draft alone and
see tcpm's opinions.

Do other people on tcpm prefer integrating the two drafts (and
adopting it as one WG item :)  ?

Yuchung

2988bis: http://www.ietf.org/id/draft-paxson-tcpm-rfc2988bis-00.txt

---

From: Yuchung Cheng <[email-masked]>
Date: Thu, 29 Jul 2010 16:02:15 +0200

Yuchung

---

From: Mark Allman <[email-masked]>
Date: Thu, 29 Jul 2010 10:29:19 -0400

> adopting it as one WG item :)  ?

I am not following this thread right now, but am seeing this note since
I was Cc-ed.  So, let me try to be quite clear about two things ...

(1) There is *nothing* in RFC 2988 that prevents an RTT sample from
    being taken on the initial SYN.  In fact, stacks have done this for
    eons.  So, there is in some sense nothing to do to RFC 2988bis in
    this regard because this is already permitted.

    This provision that a TCP should not sample during the 3WHS came
    into play precisely because the initial cwnd was increased and at
    the time the consensus of the community was that an RTT sample taken
    using a small SYN packet might not be predictive of the RTTs that
    could be experienced based on a 4KB cwnd.  So, as I view it this is
    not a part of the RTO spec---which allows samples in the 3WHS---but
    a consideration that has to do with the increased initial cwnd.  We
    have an initial cwnd draft now, lets take it up there.  (I am not
    unsympathetic to the argument that this prohibition might not be
    needed in today's world of much faster networks.)

(2) This is really a very small quibble it seems to me.  I doubt there
    is really much of an overall impact.  I.e., the place where this
    helps is exactly this: when the TCP is using larger initial windows
    and therefore does not take an RTT sample during the 3WHS and then
    loses the first packet so as to not yet have gotten an RTT sample
    during data transmission when the RTO pops.  That no doubt happens
    sometimes.  I am not going to get excited about it.

If this is all sorts of stupid because I haven't inhaled the rest of the
thread (I will ... at some point) then ignore me.

allman

---

From: Yuchung Cheng <[email-masked]>
Date: Thu, 29 Jul 2010 18:05:56 +0200

>    needed in today's world of much faster networks.)
Totally fine with me to integrate with our initcwnd proposal if the
list and other authors like so.

>    sometimes.  I am not going to get excited about it.

Like you said, the key is this happens sometimes. When it does, the
impact is not small at least for interactive web applications. A 3sec
+ RTT pause at the beginning of the http session has side-effects:
browser (or users on fast network) timeout faster and retries on a new
connection and/or reload the whole page.

---

From: Mark Allman <[email-masked]>
Date: Thu, 29 Jul 2010 12:12:24 -0400

> Like you said, the key is this happens sometimes. 

Where---dollars to donuts---"sometimes" is an exceedingly small fraction
of the time.  At some point we have to just decide that we can't
engineer an optimal solution for every conceivable case.

> When it does, the impact is not small at least for interactive web
> applications. A 3sec + RTT pause at the beginning of the http session
> has side-effects: browser (or users on fast network) timeout faster
> and retries on a new connection and/or reload the whole page.

Fair enough, but (a) there is a proposal that lowers this 3sec to 1sec
and (b) there is always a flip side and that flip side is that you lost
the first data packet you tried to send and so *just maybe*
aggressiveness is not the right course of action here.

allman