Subject: Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0

From: Enke Chen <[email-masked]>
Date: Wed, 16 Dec 2020 18:40:01 -0800

Hi, Folks:

Regarding the patch for openBGPD pointed out by Job, I do not think it
would work. When the TCP rcv window from the remote is 0, the BGP keepalive
can still be queued to the socket buffer. It can take a long time for the
socket buffer to be filled up by BGP keepalives.

It seems that the TCP_USER_TIMEOUT option can be used for the persistent
zero-size window issue.  The timeout value could be multiples of the
holdtimer (with min and max adjustments), perhaps somewhere around 5 or 6
minutes.

Thanks.   -- Enke

----------

Job Snijders <[email-masked]> Tue, 15 December 2020 21:54 UTC
<https://mailarchive.ietf.org/arch/browse/idr/#>

[snip]
How to solve this? Claudio Jeker took a look at what it would take in
OpenBGPD and came up with the (tiny!) following patch, should be
readable to most: https://marc.info/?l=openbsd-tech&m=160796802508185&w=2

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 17 Dec 2020 11:20:59 +0100

Good catch Enke !

Also what if TCP rcv takes the BGP messages and passes it to BGP I/O InQ
which drops it for some reason right there ? Looks to me like we are not
going to detect any event like this here. But the problem we are trying to
address will persist. I think in this thread we are focusing too much on
transport vs application level detection.

And I will repeat the question already stated ... Why rcv would not close
the session in spite of missing KEEPALIVES or UPDATES ?

Tx,
R.

PS. Side note: BGP Operational Message addresses this type of
inconsistencies by periodically comparing BGP Adj_RIB_In and _Out counters.

On Thu, Dec 17, 2020 at 3:41 AM Enke Chen <[email-masked]>
wrote:

> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr
>

---

From: Gert Doering <[email-masked]>
Date: Thu, 17 Dec 2020 12:20:18 +0100

Hi,

> the session in spite of missing KEEPALIVES or UPDATES ?

I do not think this is a truly relevant question for the discussion at
hand.  Observably this is not what the receiver *did* in these cases.

It might be a system that crashed in a way that kernel level TCP is still
working (and sending TCP ACKs), but userland has crashed ("waiting on
a disk access to a dead hard drive") or whatever.  This happens.

Gert Doering
        -- NetMaster
-- 
have you enabled IPv6 on something today...?

SpaceNet AG                      Vorstand: Sebastian v. Bomhard, Michael Emmer
Joseph-Dollinger-Bogen 14        Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen                 HRB: 136055 (AG Muenchen)
Tel: +49 (0)89/32356-444         USt-IdNr.: DE813185279

---

From: Claudio Jeker <[email-masked]>
Date: Thu, 17 Dec 2020 12:49:13 +0100

> socket buffer to be filled up by BGP keepalives.

I agree that it can take time for the socket buffers to fill up. This
affects both send and recv socket buffers. If a client stops to recv(2) any
data then it will take a long time for an idle session to fill up all
these buffers with 19 byte KEEPALIVEs. At the same time an idle session
will not result in stale RIB entries.
In the DFZ there is enough update chatter to reduce the time significantly.

In the end the goal is to reduce the current state of keeping the
connection open forever to something more reasonable. As mentioned to 
fill up the receive buffer and closing the window to 0 already takes time.
This trigger will not be precise but limiting the socket buffer size can
help to reduce the time.
 
> holdtimer (with min and max adjustments), perhaps somewhere around 5 or 6

Not all systems implement TCP_USER_TIMEOUT. It is one way to detect send
side problems. With a large timeout it is probably a good mechanism.

-- 
:wq Claudio

---

From: Jeffrey Haas <[email-masked]>
Date: Thu, 17 Dec 2020 09:34:30 -0500

Enke,

Indirectly you are pointing out that an application at the end of a socket is loosely coupled to the TCP state of a session, at best.

This loose coupling gets even messier when you're on a "real" router with staged host paths, NSR features, etc.

For daemons that are roughly Unix, the usual pattern is likely "if your socket is blocked for the requisite period of time, peek into the TCP state, then decide whether to abruptly hangup or not".

The scenarios we've discussed haven't even gotten into the esoteric problems such as SOME types of TCP/IP packets might get through, but not others.  Two ready examples from my career include bugs in TCP-MD5 where a specific bit of data was mis-signed and wedged the TCP window, and a particular network I dealt with that IP checksumming issues that caused long-lived TCP sessions to die in it.  Another many of us have seen is path MTU discovery hiccups that result in wedged sessions.  

Middleboxes of all sorts, including security THINGs, only make this stuff worse.

-- Jeff

> On Dec 16, 2020, at 9:40 PM, Enke Chen <[email-masked]> wrote:
> 
> Regarding the patch for openBGPD pointed out by Job, I do not think it would work. When the TCP rcv window from the remote is 0, the BGP keepalive can still be queued to the socket buffer. It can take a long time for the socket buffer to be filled up by BGP keepalives.
> 
> It seems that the TCP_USER_TIMEOUT option can be used for the persistent zero-size window issue.  The timeout value could be multiples of the holdtimer (with min and max adjustments), perhaps somewhere around 5 or 6 minutes.
> 
>

---

From: Enke Chen <[email-masked]>
Date: Thu, 17 Dec 2020 11:48:21 -0800

Hi, Robert:

The receiver is broken for not closing the session after the holdtime
expires, and that certainly needs attention.

However, the rational for trying to do something on the sender seems to be
the following: as the session is broken and should have been terminated by
the other side, but it's not, the sender would like to have a way that
provides an "upper bound" for the session to be terminated
deterministically at the transport layer.

The TCP_USER_TIMEOUT option seems to be a good fit in this case.

Thanks.   -- Enke

On Thu, Dec 17, 2020 at 2:21 AM Robert Raszuk <[email-masked]> wrote:

>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__mailarchive.ietf.org_arch_browse_idr_-23&d=DwMFaQ&c=V9IgWpI5PvzTw83UyHGVSoW3Uc1MFWe5J8PTfkrzVSo&r=OPLTTSu-451-QhDoSINhI2xYdwiMmfF5A2l8luvN11E&m=FlndUknstuJ9j_Pf40oKOLGgDCHrXNgX1l6gQZsjVxE&s=Gy3ZgD4mwrmy1k7kEyDCeqqUBxyXkv33m4XaHfegXGA&e=>
>>
>> [snip]
>> readable to most: https://marc.info/?l=openbsd-tech&m=160796802508185&w=2 <https://urldefense.proofpoint.com/v2/url?u=https-3A__marc.info_-3Fl-3Dopenbsd-2Dtech-26m-3D160796802508185-26w-3D2&d=DwMFaQ&c=V9IgWpI5PvzTw83UyHGVSoW3Uc1MFWe5J8PTfkrzVSo&r=OPLTTSu-451-QhDoSINhI2xYdwiMmfF5A2l8luvN11E&m=FlndUknstuJ9j_Pf40oKOLGgDCHrXNgX1l6gQZsjVxE&s=4Ip2QeM5GZ1ohdD4z1RB3-XR1zvrkGa-gnnnxVzd3Gs&e=>
>>
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_idr&d=DwMFaQ&c=V9IgWpI5PvzTw83UyHGVSoW3Uc1MFWe5J8PTfkrzVSo&r=OPLTTSu-451-QhDoSINhI2xYdwiMmfF5A2l8luvN11E&m=FlndUknstuJ9j_Pf40oKOLGgDCHrXNgX1l6gQZsjVxE&s=hBMaxmukXgY-6uXgnTCoi6Zoz2jI0izuMOA06uP1Seg&e=>
>>
>

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 17 Dec 2020 21:07:09 +0100

Hey Enke,

> for the session to be terminated deterministically at the transport
layer.

Yes and I would 100% agree with that.

However the discussion seems to be around terminating it at/by the
application layer just from not being able to write to transport socket any
more for N seconds. This mismatch is something IMHO is a bit ugly.

Your suggestion to solve it at transport itself is sound !

Otherwise with each failed write we start a counter. How often do we retry
to write the same msg ? Then after many failed attempts to queue a BGP
message the time is exceeded and we RST.  That triggering logic seems quite
dirty at best .. just imagine dealing with 1000s of peers.

Cheers,
R.

---

From: Brian Dickson <[email-masked]>
Date: Thu, 17 Dec 2020 12:24:43 -0800

> The TCP_USER_TIMEOUT option seems to be a good fit in this case.

I think this thread is suffering from "impedance mismatch".

There is a known issue where the actual TCP stacks of some routers are
buggy in ways that breaks things.

This proposal (TCP_USER_TIMEOUT) assumes that the local speakers' TCP stack
isn't buggy, at least with regards to the handling of that option.

I'm not opposed to using that option, but I don't think relying on that
exclusively is sufficient.

There is a layer issue, involving BGP protocol and TCP transport, where
transport and/or protocol issues (or both) are causing (or at least have
caused) global problems.

Having the BGP implementation be cognizant of the state of the TCP
connections, and handle behavior violations or boundary condition problems
expeditiously, is probably a good idea.

(The common term is "belt and suspenders", or perhaps "trust but verify".)

I.e. if the session is "broken", use all the available mechanisms in
increasing order of effectiveness (or extreme-ness) until the connection
dies, possibly with some grace periods in the escalation steps.

Brian

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Thu, 17 Dec 2020 20:51:54 +0000

TCP_USER_TIMEOUT times the peer's ACK.
That's only half the problem.
The other half is the zero window.
When the peer is advertising zero window, everything is acked. no?

Regards,
Jakob.

From: Idr <[email-masked]> On Behalf Of Brian Dickson
Sent: Thursday, December 17, 2020 12:25 PM
To: Enke Chen <[email-masked]>
Cc: idr@ietf. org <[email-masked]>; Robert Raszuk <[email-masked]>
Subject: Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0

On Thu, Dec 17, 2020 at 11:49 AM Enke Chen <[email-masked]<mailto:[email-masked]>> wrote:
Hi, Robert:

The receiver is broken for not closing the session after the holdtime expires, and that certainly needs attention.

However, the rational for trying to do something on the sender seems to be the following: as the session is broken and should have been terminated by the other side, but it's not, the sender would like to have a way that provides an "upper bound" for the session to be terminated deterministically at the transport layer.

There is a known issue where the actual TCP stacks of some routers are buggy in ways that breaks things.

This proposal (TCP_USER_TIMEOUT) assumes that the local speakers' TCP stack isn't buggy, at least with regards to the handling of that option.

I'm not opposed to using that option, but I don't think relying on that exclusively is sufficient.

There is a layer issue, involving BGP protocol and TCP transport, where transport and/or protocol issues (or both) are causing (or at least have caused) global problems.

Having the BGP implementation be cognizant of the state of the TCP connections, and handle behavior violations or boundary condition problems expeditiously, is probably a good idea.

I.e. if the session is "broken", use all the available mechanisms in increasing order of effectiveness (or extreme-ness) until the connection dies, possibly with some grace periods in the escalation steps.

Brian