Subject: Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0

From: Enke Chen <[email-masked]>
Date: Thu, 17 Dec 2020 13:02:14 -0800

Hi, Jakob:

The BGP keepalive locally generated would become "pending data", and would
trigger the TCP probe, and the then TCP_USER_TIMEOUT mechanism.

If one is paranoid that the BGP keepalive may not be generated locally,
then use the TCP_KEEPAVLIE option together with the TCP_USER_TIMEROUT
option. That would be more robust.

Thanks.  -- Enke

--------------------

"Jakob Heitz (jheitz)" <[email-masked]> Thu, 17 December 2020 20:53 UTCShow
header <https://mailarchive.ietf.org/arch/browse/idr/#>

TCP_USER_TIMEOUT times the peer's ACK.
That's only half the problem.
The other half is the zero window.
When the peer is advertising zero window, everything is acked. no?

Regards,
Jakob.

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 18 Dec 2020 07:44:33 -0500

Enke,

> On Dec 17, 2020, at 4:02 PM, Enke Chen <[email-masked]> wrote:
> 
> If one is paranoid that the BGP keepalive may not be generated locally, then use the TCP_KEEPAVLIE option together with the TCP_USER_TIMEROUT option. That would be more robust.

You're making the peculiar assumption that in broken situations that packets are getting somewhere.  This is Brian's point.

-- Jeff

---

From: Enke Chen <[email-masked]>
Date: Fri, 18 Dec 2020 10:09:17 -0800

Hi, Jeff:

No, I am not assuming that packets are getting somewhere. The
TCP_USER_TIMEOUT would work as long as there is "pending data" (either
unacked, or locally queued). The data can be from the local BGP Keepalives
or the TCP_KEEPALIVE.

Thanks.  -- Enke

On Fri, Dec 18, 2020 at 4:44 AM Jeffrey Haas <[email-masked]> wrote:

> You're making the peculiar assumption that in broken situations that
> packets are getting somewhere.  This is Brian's point.
>
> -- Jeff
>
>

---

From: Brian Dickson <[email-masked]>
Date: Fri, 18 Dec 2020 10:28:48 -0800

> unacked, or locally queued). The data can be from the local BGP Keepalives
Actually, my point was not only about packets getting somewhere, but also
that the LOCAL implementation of the TCP stack should not be assumed to be
bug-free (in relevant ways).

Your response is still assuming that those mechanisms actually work 100%
reliably 100% of the time.

Yes, if the implementation works correctly, TCP_USER_TIMEOUT would work.
However, I'm saying the BGP code should not assume that is the case, and
put some guard-rails around the behavior.
The overhead of some small amount of checking, regardless of how it is
done, is likely quite low.

(If packets are flowing, as viewed by updates and/or keepalives being seen
from the peer, for example, it might not be necessary to invoke those
checks? Or the check might only need to be done every $INTERVAL, like every
minute or two.)

Brian

>> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr
>

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 18 Dec 2020 14:20:23 -0500

On Fri, Dec 18, 2020 at 10:28:48AM -0800, Brian Dickson wrote:
> On Fri, Dec 18, 2020 at 10:09 AM Enke Chen <[email-masked]> wrote:

What's also important is that using this option removes the ability for the
BGP implementation to make its own decisions.

In the presence of some level of packet drop, the window may not be able to
advance because the ACK covering the head end isn't getting through.  So,
even if some data is getting through and helping open space in the buffer,
this feature may cause us to close the session.

Jakob also makes the point that zero window in the send direction isn't
really helped here.

> checks? Or the check might only need to be done every $INTERVAL, like every

My experience is that using the TCP information in an advisory fashion is
helpful, but hard to make work consistently or portably.  Features that tie
into the stack to try to assess liveness or close sluggish sessions are
helpful if you don't care about the impact.  BGP implementations tend to
care about being resilient, especially for ISP circumstances.

-- Jeff

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Fri, 18 Dec 2020 19:21:10 +0000

My point was a question.

Does TCP_USER_TIMEOUT work when TCP is in persist state?
It seems like it should work, but it helps to be sure.

Regards,
Jakob.

-----Original Message-----
From: Idr <[email-masked]> On Behalf Of Jeffrey Haas
Sent: Friday, December 18, 2020 11:20 AM
To: Brian Dickson <[email-masked]>
Cc: idr@ietf. org <[email-masked]>; Enke Chen <[email-masked]>
Subject: Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0

-- Jeff

---

From: John Scudder <[email-masked]>
Date: Fri, 18 Dec 2020 22:30:38 +0000

On Dec 18, 2020, at 1:09 PM, Enke Chen <[email-masked]> wrote:
> 
> No, I am not assuming that packets are getting somewhere. The TCP_USER_TIMEOUT would work as long as there is "pending data" (either unacked, or locally queued). The data can be from the local BGP Keepalives or the TCP_KEEPALIVE.

Apart from the other objections to relying on TCP_USER_TIMEOUT, which I think are sufficient, it’s not clear to me that implementations will provide the desired semantics. RFC 793 seems like it specifies the right semantics (“get this data to the peer within N seconds or close”):

        The timeout, if present, permits the caller to set up a timeout
        for all data submitted to TCP.  If data is not successfully
        delivered to the destination within the timeout period, the TCP
        will abort the connection.  The present global default is five
        minutes.

However the Linux man page documents different semantics:

       TCP_USER_TIMEOUT (since Linux 2.6.37)
              This option takes an unsigned int as an argument.  When the
              value is greater than 0, it specifies the maximum amount of
              time in milliseconds that transmitted data may remain
              unacknowledged before TCP will forcibly close the
              corresponding connection and return ETIMEDOUT to the
              application.  If the option value is specified as 0, TCP will
              use the system default.

The important difference being that whereas 793 implies data written to the socket, the Linux man page says “transmitted” data, which seems like it must mean data TCP has written to the network. These are two very different things! If Linux (or another stack) implements what the man page seems to say, it’s not useful for our purposes.

—John

---

From: William McCall <[email-masked]>
Date: Sat, 19 Dec 2020 10:38:00 +0000

> The important difference being that whereas 793 implies data written to the socket, the Linux man page says “transmitted” data, which seems like it must mean data TCP has written to the network. Th…

I was curious too. I read the manpage, relevant linux kernel code, the
RFC, and hacked up a test case (unicast me if you want the code).
Also, Cloudflare published a relevant blog entry[0]. For this specific
scenario, see under the sub-heading "Zero window ESTAB is...
forever?".

TCP_USER_TIMEOUT doesn't appear to kick in until there is unACKed
data, meaning that it has already been transmitted from TCP's
perspective. Stuff hanging around in the buffers due to persist state
doesn't seem to count, per the test results and the docs. Confirms
your thoughts from the reading I think.

[0] https://blog.cloudflare.com/when-tcp-sockets-refuse-to-die/

--
William McCall

---

From: Gyan Mishra <[email-masked]>
Date: Sat, 19 Dec 2020 16:36:07 -0500

Here is the RFC 5482 TCP User timeout options from TCPM WG.

https://tools.ietf.org/html/rfc5482

TCPM has a bis draft update to 793 that has more info then the original.

https://datatracker.ietf.org/wg/tcpm/documents/

https://tools.ietf.org/html/draft-ietf-tcpm-rfc793bis-19#page-42

>From quick read there are caveats with devices supporting or not supporting
the option.

Also I guess setting the value is tricky as well not too low or too high
that either could make matters worse with instability.

Going down this path of does seem a lot more complicated and risker then
using BFD.

Kind Regards

Gyan

On Sat, Dec 19, 2020 at 5:38 AM William McCall <[email-masked]>
wrote:

> > Apart from the other objections to relying on TCP_USER_TIMEOUT, which I
> think are sufficient, it’s not clear to me that implementations will
> provide the desired semantics. RFC 793 seems like it specifies the right
> semantics (“get this data to the peer within N seconds or close”):
> >
> > The important difference being that whereas 793 implies data written to
> the socket, the Linux man page says “transmitted” data, which seems like it
> must mean data TCP has written to the network. These are two very different
> things! If Linux (or another stack) implements what the man page seems to
> say, it’s not useful for our purposes.
> >
> > —John
-- 

<http://www.verizon.com/>

*Gyan Mishra*

*Network Solutions A**rchitect *

*M 301 502-134713101 Columbia Pike *Silver Spring, MD

---

From: Robert Raszuk <[email-masked]>
Date: Sat, 19 Dec 2020 23:18:48 +0100

Hi Gyan,

using BFD.

But BFD is not going to help at all to the problem at hand.

BFD is in the vast majority of cases distributed (and that is feature not a
bug) and responses are handled by line cards.

Here we are dealing with RE/RP based subsystems bugs regardless if those
are in TCP or BGP layer.

Thx,
R.

On Sat, Dec 19, 2020 at 10:36 PM Gyan Mishra <[email-masked]> wrote:

> From quick read there are caveats with devices supporting or not
> supporting the option.
>

---

From: Gyan Mishra <[email-masked]>
Date: Sat, 19 Dec 2020 18:20:21 -0500

Hi Robert

On the other thread it was not quite clear.

So if this scenario is completely devoid of link congestion and purely a
management plane TCP control plane processing BGP socket processing issue
then I agree BFD won’t help at all.

I agree with the poor RP design of management plane that either lead to RP
being overwhelmed high cpu and memory and or possibly memory leak or bug.

Do we know which vendor?

Something simple otter then messing with TCP parameters, if instead of
using the default 90 second BGP dead timer,  if that was reduced down a bit
to like 10 / 30, that could limit the time traffic is black hole and not
rerouted to alternate path until the hold timer expires.

Kind Regards

Gyan

On Sat, Dec 19, 2020 at 5:18 PM Robert Raszuk <[email-masked]> wrote:

> BFD is in the vast majority of cases distributed (and that is feature not
> a bug) and responses are handled by line cards.
>
>>> > Apart from the other objections to relying on TCP_USER_TIMEOUT, which
>>> I think are sufficient, it’s not clear to me that implementations will
>>> > The important difference being that whereas 793 implies data written
>>> to the socket, the Linux man page says “transmitted” data, which seems like
>>> it must mean data TCP has written to the network. These are two very
>>> different things! If Linux (or another stack) implements what the man page
>>> seems to say, it’s not useful for our purposes.
>>> >
>>> > —John
>> *M 301 502-134713101 Columbia Pike
>> <https://www.google.com/maps/search/13101+Columbia+Pike%C2%A0+Silver+Spring,+MD?entry=gmail&source=g>*Silver
>> Spring, MD
>> <https://www.google.com/maps/search/13101+Columbia+Pike%C2%A0+Silver+Spring,+MD?entry=gmail&source=g>
>>

*Gyan Mishra*

---

From: Gyan Mishra <[email-masked]>
Date: Sat, 19 Dec 2020 18:32:12 -0500

I think one critical point I am missing and please correct me if I am wrong.

When the 0 receive window is received and when the hold timer expires a
notification is sent but since the buffer is full on the other end the tcp
socket does not get the notification thus end up in an indefinite TCP pause
state until the default TCP user timeout is reached.

Completely agree BFD cannot resolve this and we need to mess with TCP user
timeout option is the only solution.

Kind Regards

Gyan

On Sat, Dec 19, 2020 at 6:20 PM Gyan Mishra <[email-masked]> wrote:

>> > Going down this path of does seem a lot more complicated and risker
>> then using BFD.
>>

*Gyan Mishra*

---

From: Robert Raszuk <[email-masked]>
Date: Sun, 20 Dec 2020 00:45:52 +0100

Hi Gyan,

to like 10 / 30

Sorry but again this is not the issue here.

The issue is that rcv peer is not terminating the session after holdtime
expires.

The sender can still keep receiving updates or keepalives just fine. This
is unidirectional issue.

The ask here is to have BGP trigger session RST or termination at TCP level
when we can no longer write to a TCP socket for N seconds.

- - -

To summarize watching this thread it seems that most folks agree that if we
do that the HOLD_SEND should be different then HOLD_RCV.

There is ongoing discussion to keep this at TCP level.

There is an apparent ask to make it a default with a knob to disable it.

Mechanics proposed seems to be to keep per peer HOLD_SEND timer and start
it at each socket write failure then stop+reset it at each socket write
success.

The other day I asked how often BGP is retrying to write to socket in most
widely deployed implementations - but did not get any answer :(

Best,
R.

---

From: Jeffrey Haas <[email-masked]>
Date: Sat, 19 Dec 2020 19:16:50 -0500

> On Dec 19, 2020, at 6:45 PM, Robert Raszuk <[email-masked]> wrote:
> 
> The other day I asked how often BGP is retrying to write to socket in most widely deployed implementations - but did not get any answer :( 
> 

I will publicly state that in each implementation I've worked on over twenty years that handling EAGAIN, EWOULDBLOCK and similar along with dealing with long-lived zero windowing regularly happens.

The majority of the time this is due to simple load issues on the receiver.  It occasionally is related to local resources being scarce (e.g. mbufs).

Once you start getting packet loss on the TCP session, you get these things much more often.

-- Jeff

---

From: Gyan Mishra <[email-masked]>
Date: Sun, 20 Dec 2020 02:47:40 -0500

Hi Robert

Thank you for the summarization.

I agree this is a critical issue to be solved as it can happen to any
global  routing system.

+1 with Tony, Jeff and others to apply the hold time to the transmit size,
and plan to updated text for RFC 4271 section 6.5 to reflect.

Excerpt from Tony:

“More generally, the proposal is that we apply the HOLD TIME on the
transmit side as well as the receive side. If we are not able to transmit
for that period of time, the receiver should give up and so should the
transmitter. The session is broken, updates cannot flow, and we no longer
have (eventual) consistency.”

Kind Regards

Gyan

On Sat, Dec 19, 2020 at 6:46 PM Robert Raszuk <[email-masked]> wrote:

> The ask here is to have BGP trigger session RST or termination at TCP
> level when we can no longer write to a TCP socket for N seconds.
>
> - - -
>
> To summarize watching this thread it seems that most folks agree that if
> we do that the HOLD_SEND should be different then HOLD_RCV.
>
>> I agree with the poor RP design of management plane that either lead to
>> RP being overwhelmed high cpu and memory and or possibly memory leak or
>> bug.
>>
>> Do we know which vendor?
>>
>>> BFD is in the vast majority of cases distributed (and that is feature
>>> not a bug) and responses are handled by line cards.
>>>
>>>> Also I guess setting the value is tricky as well not too low or too
>>>> high that either could make matters worse with instability.
>>>>
>>>> On Sat, Dec 19, 2020 at 5:38 AM William McCall <
>>>> [email-masked]> wrote:
>>>>
>>>>> > Apart from the other objections to relying on TCP_USER_TIMEOUT,
>>>>> which I think are sufficient, it’s not clear to me that implementations
>>>>> will provide the desired semantics. RFC 793 seems like it specifies the
>>>>> right semantics (“get this data to the peer within N seconds or close”):
>>>>> >

*Gyan Mishra*

---

From: Robert Raszuk <[email-masked]>
Date: Sun, 20 Dec 2020 13:25:21 +0100

Gyan,

Not sure what you are really trying to highlight in your last note, but If
we would apply the same HOLD TIME used today to detect BGP liveness to
detect "stuck" peer it would be pretty bad choice for obvious reasons.

Thx,
R

On Sun, Dec 20, 2020 at 8:47 AM Gyan Mishra <[email-masked]> wrote:

>> The other day I asked how often BGP is retrying to write to socket in
>> most widely deployed implementations - but did not get any answer :(
>>
>> Best,
>> R.
>>
>>
>>>> Here we are dealing with RE/RP based subsystems bugs regardless if
>>>> those are in TCP or BGP layer.
>>>>
>>>> Thx,
>>>> R.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>> > The important difference being that whereas 793 implies data
>>>>>> written to the socket, the Linux man page says “transmitted” data, which
>>>>>> seems like it must mean data TCP has written to the network. These are two
>>>>>> very different things! If Linux (or another stack) implements what the man
>>>>>> page seems to say, it’s not useful for our purposes.
>>>>>> >
>>>>>> > —John

---

From: Gyan Mishra <[email-masked]>
Date: Sun, 20 Dec 2020 12:23:36 -0500

Hi Robert

Sorry I mis spoke my last note as it was not complete.

I reread the thread again and the mail archive from 2010 below.

I agree with everything you stated in your initial reply.

Reiterating what you have stated on possible solution(s) so I am in sync
with you I am using the mail archive problem statement visual depictions
below.

https://mailarchive.ietf.org/arch/msg/idr/q0Sx5d3zZjfOmOQ4lO2OZAHh9Lc/

So the main problem is the 1 way BGP being up  and working from rtr-A to
rtr-B and the TCP stuck state of BGP from rtr-B to rtr-A due to control
plane or management plane congestion on rtr-A.

On rtr-B the TCP socket for BGP is stuck and cannot write to the socket to
send the BGP notification hold time expired.

“Mechanics proposed seems to be to keep per peer HOLD_SEND timer and start
it at each socket write failure then stop+reset it at each socket write
success.”

So what is being proposed is a method to detect the 1-way stuck peer with a
new per peer “HOLD_SEND” different from the “HOLD_RECEIVE” to trigger a BGP
session reset if we cannot write to the TCP socket for N second within the
new “HOLD_SEND” interval.

So after the first socket write failure a manual stop event and TCP RST to
recover the peer immediately.

So if the send buffer TCP socket can be written to successfully, don’t do
anything, but as soon as a socket write failure occurs for the first byte
attempted to be written, “stuck detected”, that would be the trigger for
the BGP session reset.

As far as GR if enabled discussed,  rtr-A to RTs-B is still sending BGP
updates, so we would set F flag to flush the BGP rib as we can’t trust the
routes from A as they could be stale due to its control plane congestion.
Correct?

In an example topology:

  rtr-A                   rtr-B
  (congested c-p)         (uncongested c-p)
  send window: >0         send window: 0
  recv window: 0          recv window: >0	

In this case we expect:
 a) rtr-B does not send any BGP packet (KEEPALIVE/UPDATE/NOTIFICATION)
to rtr-A in normal operating circumstances.
 b) rtr-A does not expect any KEEPALIVE/UPDATE packets from rtr-B. The
session remains established even if no packet is received in the
holdtime.
 c) rtr-A continues to send KEEPALIVE packets to rtr-B.

1) rtr-A signals control plane congestion through means of setting the
TCP recv window to 0, but continues to send KEEPALIVEs to rtr-B
2) rtr-B does not send any KEEPALIVE/UPDATE to rtr-A and indicates this
fact through CLI output.
3) After t=HOLDTIME, rtr-B sends NOTIFICATION to rtr-A indicating that
the hold time has expired.

On Sun, Dec 20, 2020 at 7:25 AM Robert Raszuk <[email-masked]> wrote:

>> +1 with Tony, Jeff and others to apply the hold time to the transmit
>> size, and plan to updated text for RFC 4271 section 6.5 to reflect.
>>
>> Excerpt from Tony:
>>
>>> The sender can still keep receiving updates or keepalives just fine.
>>> This is unidirectional issue.
>>>
>>> Mechanics proposed seems to be to keep per peer HOLD_SEND timer and
>>> start it at each socket write failure then stop+reset it at each socket
>>> write success.
>>>
>>>> So if this scenario is completely devoid of link congestion and purely
>>>> a management plane TCP control plane processing BGP socket processing issue

*Gyan Mishra*

---

From: Enke Chen <[email-masked]>
Date: Sun, 20 Dec 2020 22:52:34 -0800

Hi, John and William:

My reading of the Linux function tcp_probe_timer()  is that the data in the
socket buffer is checked. More specifically, when there is no
un-acked data, but there is data in the socket buffer,
the "icsk_user_timeout" would be checked, and the probe timer would be set
again in tcp_send_probe0().

I am not sure what could have caused the failure that William observed. We
will need someone who is familiar with the TCP code to take a look.
There might be one potential issue in tcp_check_probe_timer() where the
probe timer is not started (please see below).

Thanks.   -- Enke

----------------------
Linux v5.10-rc7-149-g33dc961

*diff --git a/include/net/tcp.h b/include/net/tcp.h*

*index d4ef5bf..0b28af1 100644*

*--- a/include/net/tcp.h*

*+++ b/include/net/tcp.h*

@@ -1328,7 +1328,8 @@ static inline unsigned long tcp_probe0_when(const
struct sock *sk,

 static inline void tcp_check_probe_timer(struct sock *sk)

 {

-       if (!tcp_sk(sk)->packets_out && !inet_csk(sk)->icsk_pending)

+       if (!tcp_sk(sk)->packets_out &&

+           (inet_csk(sk)->icsk_pending != ICSK_TIME_PROBE0))

                tcp_reset_xmit_timer(sk, ICSK_TIME_PROBE0,

                                     tcp_probe0_base(sk), TCP_RTO_MAX);
 }

----------

static void tcp_probe_timer(struct sock *sk)

{

        struct inet_connection_sock *icsk = inet_csk(sk);

        struct sk_buff *skb = tcp_send_head(sk);

        struct tcp_sock *tp = tcp_sk(sk);

        int max_probes;

        if (tp->packets_out || !skb) {

                icsk->icsk_probes_out = 0;

                return;

        }

        /* RFC 1122 4.2.2.17 requires the sender to stay open indefinitely
as

         * long as the receiver continues to respond probes. We support
this by

         * default and reset icsk_probes_out with incoming ACKs. But if the

         * socket is orphaned or the user specifies TCP_USER_TIMEOUT, we

         * kill the socket when the retry count and the time exceeds the

         * corresponding system limit. We also implement similar policy when

         * we use RTO to probe window in tcp_retransmit_timer().

         */

        if (icsk->icsk_user_timeout) {

                u32 elapsed = tcp_model_timeout(sk, icsk->icsk_probes_out,

                                                tcp_probe0_base(sk));

                if (elapsed >= icsk->icsk_user_timeout)

                        goto abort;

        }

On Sat, Dec 19, 2020 at 2:38 AM William McCall <[email-masked]>
wrote:

> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_idr&d=DwIFaQ&c=V9IgWpI5PvzTw83UyHGVSoW3Uc1MFWe5J8PTfkrzVSo&r=OPLTTSu-451-QhDoSINhI2xYdwiMmfF5A2l8luvN11E&m=P-eZWmrFtootouPUugKAk40aIyuZdrP9wLMCSS7GUTU&s=6oYcnalNTtK-8ktoh-vivM6BlWM0bCrW3WuHw19s7zo&e=
>
> [0]
> https://urldefense.proofpoint.com/v2/url?u=https-3A__blog.cloudflare.com_when-2Dtcp-2Dsockets-2Drefuse-2Dto-2Ddie_&d=DwIFaQ&c=V9IgWpI5PvzTw83UyHGVSoW3Uc1MFWe5J8PTfkrzVSo&r=OPLTTSu-451-QhDoSINhI2xYdwiMmfF5A2l8luvN11E&m=P-eZWmrFtootouPUugKAk40aIyuZdrP9wLMCSS7GUTU&s=M-HzefvcFBD2FU8OVERU_vTL_ObzcdQdlk0BUrADphk&e=
>
> --
> William McCall
>

---

From: Enke Chen <[email-masked]>
Date: Mon, 21 Dec 2020 12:50:52 -0800

Hi, William:

There have been multiple fixes to the TCP_USER_TIMEOUT feature in LInux.  I
listed several (see below) that look important.

One of the commits has the following comment:

    In addition, we change TCP_USER_TIMEOUT to cover (life or dead)

    sockets stalled on zero-window probes. This changes the semantics

    of TCP_USER_TIMEOUT slightly because it previously only applies

    when the socket has pending transmission.

Not sure which Linux version you used, and whether it has these fixes.

Thanks.   -- Enke

------

commit 8c72c65b426b47b3c166a8fef0d8927fe5e8a28d

Author: Eric Dumazet <[email-masked]>

Date:   Wed Sep 13 20:30:39 2017 -0700

    tcp: update skb->skb_mstamp more carefully

    liujian reported a problem in TCP_USER_TIMEOUT processing with a patch

    in tcp_probe_timer() :

          https://www.spinics.net/lists/netdev/msg454496.html

    ...

commit 4ab688793e086ef6d1744a0f803fe9770a1ae5d0

Author: Eric Dumazet <[email-masked]>

Date:   Sun May 21 10:39:00 2017 -0700

    tcp: fix tcp_probe_timer() for TCP_USER_TIMEOUT

    TCP_USER_TIMEOUT is still converted to jiffies value in

    icsk_user_timeout

    So we need to make a conversion for the cases HZ != 1000

    ...

commit b248230c34970a6c1c17c591d63b464e8d2cfc33

Author: Yuchung Cheng <[email-masked]>

Date:   Mon Sep 29 13:20:38 2014 -0700

    tcp: abort orphan sockets stalling on zero window probes

    Currently we have two different policies for orphan sockets

    that repeatedly stall on zero window ACKs. If a socket gets

    a zero window ACK when it is transmitting data, the RTO is

    used to probe the window. The socket is aborted after roughly

    tcp_orphan_retries() retries (as in tcp_write_timeout()).

    But if the socket was idle when it received the zero window ACK,

    and later wants to send more data, we use the probe timer to

    probe the window. If the receiver always returns zero window ACKs,

    icsk_probes keeps getting reset in tcp_ack() and the orphan socket

    can stall forever until the system reaches the orphan limit (as

    commented in tcp_probe_timer()). This opens up a simple attack

    to create lots of hanging orphan sockets to burn the memory

    and the CPU, as demonstrated in the recent netdev post "TCP

    connection will hang in FIN_WAIT1 after closing if zero window is

    advertised." http://www.spinics.net/lists/netdev/msg296539.html

    This patch follows the design in RTO-based probe: we abort an orphan

    socket stalling on zero window when the probe timer reaches both

    the maximum backoff and the maximum RTO. For example, an 100ms RTT

    connection will timeout after roughly 153 seconds (0.3 + 0.6 +

    .... + 76.8) if the receiver keeps the window shut. If the orphan

    socket passes this check, but the system already has too many orphans

    (as in tcp_out_of_resources()), we still abort it but we'll also

    send an RST packet as the connection may still be active.

    In addition, we change TCP_USER_TIMEOUT to cover (life or dead)

    sockets stalled on zero-window probes. This changes the semantics

    of TCP_USER_TIMEOUT slightly because it previously only applies

    when the socket has pending transmission.

    ...

On Sun, Dec 20, 2020 at 10:52 PM Enke Chen <[email-masked]>
wrote:

> Hi, John and William:
>
> My reading of the Linux function tcp_probe_timer()  is that the data in
> the socket buffer is checked. More specifically, when there is no
>         /* RFC 1122 4.2.2.17 requires the sender to stay open
> indefinitely as
>

---

From: Enke Chen <[email-masked]>
Date: Mon, 21 Dec 2020 19:20:46 -0800

Hi, John:

I checked a recent Linux version (v5.10-rc7-149-g33dc961), and confirmed
that the per-session USER_TIMEOUT value (icsk->icsk_user_timeout) is applied
 in all these cases:

  1) tcp_retransmit_timer()
              tcp_write_timeout()
              tcp_clamp_rto_to_user_timeout()

     2)  tcp_probe_timer()
             For zero-window event: when there is outstanding un-acked
data, or buffered data.

     3)  tcp_keepalive_timer()

So it seems that the implementation does provide the desired semantics
described in RFC 793, but the man page is out-dated.

Thanks.  -- Enke

---

From: Enke Chen <[email-masked]>
Date: Mon, 21 Dec 2020 20:08:09 -0800

Correction to 2), it should be:

2)  tcp_probe_timer()
             For zero-window event: when there is buffered data, but no
outstanding un-acked data.

---
The case with outstanding un-acked data would be taken care of by
tcp_retransmit_timer().

Thanks.   -- Enke

On Mon, Dec 21, 2020 at 7:20 PM Enke Chen <[email-masked]>
wrote:

> that the per-session USER_TIMEOUT value (icsk->icsk_user_timeout) is
> applied in all these cases:
>