Subject: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0

From: Job Snijders <[email-masked]>
Date: Fri, 11 Dec 2020 19:23:50 +0000

Dear group,

Not too long ago an incident [1] in one Autonomous System resulted in
the global Internet being unusable in many parts of the world for
multiple hours. Some have reported the root cause was a 'configuration
error', however I believe much of the observed communication blackouts
in the global routing system stemmed from a pre-existing condition: a
specific implementation property present in multiple implementations
currently in use in the default-free zone. 

Usually when an incident happens in one AS, affected parties can through
unilateral action 'route around the problem', but the ability to 'route
around problems' critically depends on the ability to distribute
WITHDRAW or UPDATE messages. When messages are not processed, what
generally was assumed to be a unilaterally solvable problem, now requires
coordination between *all* neighbors of the suffering AS.

The global routing system requires every participant to process BGP
messages, because the alternative is intervention on thousands of BGP
devices to manually shutdown thousands of BGP sessions disconnecting the
AS suffering from an incident, to help the rest of the default-free
zone. I speak from experience when saying that coordinating a disconnection
of an AS at global scale is incredibly hard and slow, any many approval
levels must be worked through. It takes *hours* of phone calls & email
chains, a time window during which internet traffic is routed towards
stale (now blackholing) locations.

In the average ISP's network design using IBGP Route Reflectors, these
blackout effects are aggravated when BGP sessions landing in such
devices are not terminated when TCP causes the BGP session to stall.

The problem of how TCP and BGP-4 can interact has been discussed before,
but I'm not sure the working group followed up with any publication
detailing the problem and the solution.

    https://mailarchive.ietf.org/arch/msg/idr/q0Sx5d3zZjfOmOQ4lO2OZAHh9Lc/

Does everyone agree BGP-4 sessions MUST be terminated using a TCP RST
(instead of a BGP-4 Cease NOTIFICATION) if the peer has indicated for
the duration of the Hold Timer that the TCP receive window is zero?
I'm fine with there being buttons to make this different, but the
default for routers in the global Internet routing system should be to
consider the remote peer to be 'a lost cause' when it won't accept new
BGP messages for the duration of the hold timer.

Perhaps RFC 4271 Section 6.5 should be amended as following:

OLD:
    If a system does not receive successive KEEPALIVE, UPDATE, and/or
    NOTIFICATION messages within the period specified in the Hold Time
    field of the OPEN message, then the NOTIFICATION message with the
    Hold Timer Expired Error Code is sent and the BGP connection is
    closed.

NEW:
    If a system does not receive (or is unable to send) successive
    KEEPALIVE, UPDATE, and/or NOTIFICATION messages within the period
    specified in the Hold Time field of the OPEN message, then the
    NOTIFICATION message with the Hold Timer Expired Error Code is sent
    and the BGP connection is closed. If the NOTIFICATION message cannot
    be send the BGP connection is closed.

This is an ongoing problem. I suspect the BGP Nyancat's discoloration at
the left most eye might have been caused by an active TCP session
keeping a stale BGP session alive. But also the observations from "BGP
Zombies: an Analysis of Beacons Stuck Routes" [3] could be explained by
the problematic interaction between TCP and BGP.

I appreciate the work the IDR working group has done to *SOFTEN* the
blow from implementation defects on global routing (RFC 7606 is a
brilliant example of this), but I fear in this case there is no subtle
way to say goodbye when the peer doesn't process messages in a timely
fashion. It might be good to document this.

Kind regards,

Job

[1]: https://www.reuters.com/article/level-3-communi-outages-idUSL2N1CB00C
[2]: https://labs.ripe.net/Members/cteusche/bgp-meets-cat
[3]: https://www.iij-ii.co.jp/en/members/romain/pdf/romain_pam2019.pdf

---

From: Tony Li <[email-masked]>
Date: Fri, 11 Dec 2020 11:59:54 -0800

Hi Job,

> the duration of the Hold Timer that the TCP receive window is zero?

Not quite.  I agree that the session should be terminated if the TCP receive window
is zero for the duration of the Hold Timer. Clearly the receiver is stuck and the
receiver should be terminating the connection. For robustness, it makes sense that
the transmitter also close the connection. 

I would prefer to see the transmitter do both: queue a NOTIFICATION and reset
the TCP connection. Yes, the NOTIFICATION will probably never make it out. The advantage
of doing it this way is that the code path for errors will be more consistent: always send
a NOTIFICATION before the RST. This code path is also likely to send appropriate signaling to
the management plane. This is also useful in debugging: if you go through the
message queue, the NOTIFICATION will be sitting there in the queue.

>    be send the BGP connection is closed.

Your proposed text does what I prefer, not what you suggested above. I would suggest
two edits: (1) remove then parentheses in the first sentence, they are unnecessary, and
(2) remove the last sentence.

Regards,
Tony

---

From: John Scudder <[email-masked]>
Date: Fri, 11 Dec 2020 20:03:11 +0000

[all hats on]

Hi Job,

Thanks for bringing this up.

To take the liberty of summarizing your wall of text :-) you’re saying that you believe BGP should tear down its session if it’s unable to send a message for the duration of the hold time. 

Given that the conversation last time was inconclusive I think this is a good thing for the WG to discuss again. If you want to, you (or someone) could turn the idea into a short draft that updates RFC 4271, and we could have a WG adoption discussion about it. It might help focus the discussion but it’s not mandatory.

I’ll point out a few things to start with —

- Making it mandatory to apply hold time to the sending of messages would potentially make BGP peerings less stable. It clearly can’t make them *more* stable. Of course one can argue that if you haven’t been able to send a message for the hold time, the session has failed its metric of usefulness anyway, so any veneer of stability at this point is a harmful sham.
- If I recall correctly, RST doesn’t work (or may not work) if you’re using the MD5 TCP option. Nothing much to be done, but be aware.
- There is nothing stopping an implementation from doing what you describe now. The formalism that keeps you within the letter of 4271 would be that the implementation supplies a configuration option, that you set to enable the behavior. Once you’ve done that, when the implementation notices that the hold time has been exceeded in the outbound direction, it generates a ManualStop event for the session. 

Thanks,

—John

> On Dec 11, 2020, at 2:23 PM, Job Snijders <[email-masked]> wrote:
> 
> 
> Dear group,
> 
>    https://urldefense.com/v3/__https://mailarchive.ietf.org/arch/msg/idr/q0Sx5d3zZjfOmOQ4lO2OZAHh9Lc/__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkPhCc8cBA$
> 
> [1]: https://urldefense.com/v3/__https://www.reuters.com/article/level-3-communi-outages-idUSL2N1CB00C__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMkF2w4cg$
> [2]: https://urldefense.com/v3/__https://labs.ripe.net/Members/cteusche/bgp-meets-cat__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMry7Ktyw$
> [3]: https://urldefense.com/v3/__https://www.iij-ii.co.jp/en/members/romain/pdf/romain_pam2019.pdf__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkO8A78j8Q$
> 
> _______________________________________________
> Idr mailing list
> [email-masked]
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/idr__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMMXdwc-g$

---

From: Jeff Tantsura <[email-masked]>
Date: Fri, 11 Dec 2020 13:07:57 -0800

I’m with Tony, RST without Notification would be anything but useful/helpful for troubleshooting.
Doing both, and in that particular order would ensure consistency, and if Notification does make it through - assist with troubleshooting.

Regards,
Jeff

> On Dec 11, 2020, at 12:00, Tony Li <[email-masked]> wrote:
> 
> 
> Hi Job,
> 
> 
> https://www.ietf.org/mailman/listinfo/idr

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 11 Dec 2020 22:22:07 +0100

Hi,

Couple of observations/questions:

* I do share some concerns that this will make BGP peering less stable.

* Is the "unable to send" only possible under Window = 0 ? What if there is
a local NIC buffer full and we keep dropping it locally ? If we are going
there perhaps we could say "unable to successfully send" meaning send and
get an ACK for it ?

* The proposal is about reusing the HOLD TIME value to bring BGP down when
you are still receiving keepalives however peer sent ACK for the last
segment indicating zero window - is this right ?

* What happens if our side is stuck and not able to process subsequent ACKs
which potentially increase the window on the peer ?

* Most deployments use BFD to make sure peer is reachable. As BFD is often
offloaded from CPU this is not an indication of health of TCP or BGP path.
But what if some deployments can not use BFD (say not supported by the
peer) ? Then those typically reduce BGP HOLD TIME. Would it not be too
fragile in some cases to bring a TCP down in such cases too fast ? Aren't
we overloading HOLD TIME value here a bit ?

* Assume we bring TCP down ... when does it attempt to go up again ? Are we
ok to bring it up only after manual/script action from such a state ?

* As proposed it seems that the change will affect all AFI/SAFIs using
given single TCP session. Even if perhaps all are perfectly healthy and
running on separate cores. If each SAFI would run on a separate TCP session
this would not have such an impact.

* The change seems applicable to both iBGP and eBGP right ?

In summary the attempt here is to fix application issues by cutting the
transport. Sure half broken transport for whatever reason may not be a good
thing to keep in UP state. Especially when redundancy is in place. But my
main concerns are that we are only trying to focus on a single low level
trigger to detect it.

Wouldn't per AFI/SAFI heartbeat be a better option to detect if a peer's
BGP stack is still up and running fine for all applications ?

Thx,
R.

On Fri, Dec 11, 2020 at 9:04 PM John Scudder <jgs=
[email-masked]> wrote:

> To take the liberty of summarizing your wall of text :-) you’re saying
> that you believe BGP should tear down its session if it’s unable to send a
> message for the duration of the hold time.
>
> Given that the conversation last time was inconclusive I think this is a
> good thing for the WG to discuss again. If you want to, you (or someone)
> could turn the idea into a short draft that updates RFC 4271, and we could
> have a WG adoption discussion about it. It might help focus the discussion
> but it’s not mandatory.
>
> - Making it mandatory to apply hold time to the sending of messages would
> potentially make BGP peerings less stable. It clearly can’t make them
> *more* stable. Of course one can argue that if you haven’t been able to
> send a message for the hold time, the session has failed its metric of
> usefulness anyway, so any veneer of stability at this point is a harmful
> sham.
> - If I recall correctly, RST doesn’t work (or may not work) if you’re
> using the MD5 TCP option. Nothing much to be done, but be aware.
> - There is nothing stopping an implementation from doing what you describe
> now. The formalism that keeps you within the letter of 4271 would be that
> the implementation supplies a configuration option, that you set to enable
> the behavior. Once you’ve done that, when the implementation notices that
> the hold time has been exceeded in the outbound direction, it generates a
> ManualStop event for the session.
>
> Thanks,
>
> —John
>
> > [1]:
> https://urldefense.com/v3/__https://www.reuters.com/article/level-3-communi-outages-idUSL2N1CB00C__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMkF2w4cg$
> > [2]:
> https://urldefense.com/v3/__https://labs.ripe.net/Members/cteusche/bgp-meets-cat__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMry7Ktyw$
> > [3]:
> https://urldefense.com/v3/__https://www.iij-ii.co.jp/en/members/romain/pdf/romain_pam2019.pdf__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkO8A78j8Q$
> >

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 11 Dec 2020 22:30:23 +0100

> the TCP connection.

Attempting to queue/send NOTIFICATION is a good. But which error code ?

4 - Hold Timer Expired ?

That may be pretty confusing.

If so why don't we define new error code for this trigger ?

ex: Value 8 - "Peer's TCP Win Zero"

Thx,
R.

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Fri, 11 Dec 2020 22:07:57 +0000

Perhaps we are focusing too much on TCP.
The real issue is that BGP can't send a single byte for the duration of the hold timer.
We could instead say if the socket is send blocked.

Now, I think BGP should close the socket ungracefully, with close() and no shutdown()
and no NOTIFICATION. It cannot send a NOTIFICATION if the socket is blocked.
The management plane issue can be fixed in any number of ways.

I advocate for TCP to send a RST rather than wait for the window to open up and
drain the queue with a FIN at the end.
Allowing TCP to wait only drags out the pain. It must die. And it must die NOW.

Holding up WITHDRAWs is serious, because it does not allow redundant paths
to take effect.

Also, a RST allows a recovery using graceful restart, whereas NOTIFICATION and FIN precludes GR.
Notwithstanding RFC8538.

Regards,
Jakob.

From: Idr <[email-masked]> On Behalf Of Robert Raszuk
Sent: Friday, December 11, 2020 1:22 PM
To: Job Snijders <[email-masked]>
Cc: [email-masked]
Subject: Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0

Hi,

* Is the "unable to send" only possible under Window = 0 ? What if there is a local NIC buffer full and we keep dropping it locally ? If we are going there perhaps we could say "unable to successfully send" meaning send and get an ACK for it ?

* The proposal is about reusing the HOLD TIME value to bring BGP down when you are still receiving keepalives however peer sent ACK for the last segment indicating zero window - is this right ?

* What happens if our side is stuck and not able to process subsequent ACKs which potentially increase the window on the peer ?

* Most deployments use BFD to make sure peer is reachable. As BFD is often offloaded from CPU this is not an indication of health of TCP or BGP path. But what if some deployments can not use BFD (say not supported by the peer) ? Then those typically reduce BGP HOLD TIME. Would it not be too fragile in some cases to bring a TCP down in such cases too fast ? Aren't we overloading HOLD TIME value here a bit ?

* Assume we bring TCP down ... when does it attempt to go up again ? Are we ok to bring it up only after manual/script action from such a state ?

* As proposed it seems that the change will affect all AFI/SAFIs using given single TCP session. Even if perhaps all are perfectly healthy and running on separate cores. If each SAFI would run on a separate TCP session this would not have such an impact.

In summary the attempt here is to fix application issues by cutting the transport. Sure half broken transport for whatever reason may not be a good thing to keep in UP state. Especially when redundancy is in place. But my main concerns are that we are only trying to focus on a single low level trigger to detect it.

Wouldn't per AFI/SAFI heartbeat be a better option to detect if a peer's BGP stack is still up and running fine for all applications ?

Thx,
R.

On Fri, Dec 11, 2020 at 9:04 PM John Scudder <jgs=[email-masked]<mailto:[email-masked]>> wrote:
[all hats on]

Hi Job,

Thanks,

—John

> On Dec 11, 2020, at 2:23 PM, Job Snijders <[email-masked]<mailto:[email-masked]>> wrote:
>
>
> Dear group,
>
>    https://urldefense.com/v3/__https://mailarchive.ietf.org/arch/msg/idr/q0Sx5d3zZjfOmOQ4lO2OZAHh9Lc/__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkPhCc8cBA$<https://urldefense.com/v3/__https:/mailarchive.ietf.org/arch/msg/idr/q0Sx5d3zZjfOmOQ4lO2OZAHh9Lc/__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkPhCc8cBA$>
>
> [1]: https://urldefense.com/v3/__https://www.reuters.com/article/level-3-communi-outages-idUSL2N1CB00C__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMkF2w4cg$<https://urldefense.com/v3/__https:/www.reuters.com/article/level-3-communi-outages-idUSL2N1CB00C__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMkF2w4cg$>
> [2]: https://urldefense.com/v3/__https://labs.ripe.net/Members/cteusche/bgp-meets-cat__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMry7Ktyw$<https://urldefense.com/v3/__https:/labs.ripe.net/Members/cteusche/bgp-meets-cat__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMry7Ktyw$>
> [3]: https://urldefense.com/v3/__https://www.iij-ii.co.jp/en/members/romain/pdf/romain_pam2019.pdf__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkO8A78j8Q$<https://urldefense.com/v3/__https:/www.iij-ii.co.jp/en/members/romain/pdf/romain_pam2019.pdf__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkO8A78j8Q$>
>
> [email-masked]<mailto:[email-masked]>
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/idr__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMMXdwc-g$<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/idr__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMMXdwc-g$>

_______________________________________________
Idr mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/idr

---

From: Job Snijders <[email-masked]>
Date: Fri, 11 Dec 2020 22:24:22 +0000

Dear Jakob, Robert, Jeff, John, Tony,

On Fri, Dec 11, 2020 at 10:07:57PM +0000, Jakob Heitz (jheitz) wrote:
> Perhaps we are focusing too much on TCP. The real issue is that BGP
> can't send a single byte for the duration of the hold timer.  We could
> instead say if the socket is send blocked.
> 

Good feedback, I have a feeling we all are pointing at the same thing
but using different words to describe it. Increasing robustness in the
global routing system is the goal here, so with that in mind:

I agree with Tony's edit suggestions at https://mailarchive.ietf.org/arch/msg/idr/CS7VOx42V76RfGLmXhQqM8DyNjk/

I think the desired behavior is that locally the system emits a log
message 'hold time expired', but on the control plane really makes sure
the stuck BGP session is killed and discarded and withdraws are
generated. (Whether the Cease NOTIFICATION could be send or arrived or
was lost in transmission!)

Both as developer and operator you'd want to inspect the captured TCP
'outq' and SYSLOG facility, so there might be merit to Robert's
suggestion to make this an implementation checklist item
https://mailarchive.ietf.org/arch/msg/idr/c9uqI9l815RrOUayFdLdV3ntNc0/ 

> Holding up WITHDRAWs is serious, because it does not allow redundant paths

Yes, it is very problematic in the global context.

> Also, a RST allows a recovery using graceful restart, whereas
> NOTIFICATION and FIN precludes GR.  Notwithstanding RFC8538.

The interactions with GR I defer to gurus like yourself, Robert, Jeff,
John, and Tony.

Kind regards,

Job

---

From: Tony Li <[email-masked]>
Date: Fri, 11 Dec 2020 14:31:04 -0800

Hi Robert,

> * Is the "unable to send" only possible under Window = 0 ? What if there is a local NIC buffer full and we keep dropping it locally ? If we are going there perhaps we could say "unable to successfu…

There are many, many reasons why we might not be able to exchange bits. The specifics aren’t particularly relevant. The point is that we’re not able to make progress, so the session is clearly broken.

> * The proposal is about reusing the HOLD TIME value to bring BGP down when you are still receiving keepalives however peer sent ACK for the last segment indicating zero window - is this right ? 

More generally, the proposal is that we apply the HOLD TIME on the transmit side as well as the receive side. If we are not able to transmit for that period of time, the receiver should give up and so should the transmitter. The session is broken, updates cannot flow, and we no longer have (eventual) consistency.

> * What happens if our side is stuck and not able to process subsequent ACKs which potentially increase the window on the peer ? 

Then our side has some kind of bug. But the result is the same: the session is stuck and is not helpful.  Tearing down the session and starting over is probably the best that we can do.

> * Most deployments use BFD to make sure peer is reachable. As BFD is often offloaded from CPU this is not an indication of health of TCP or BGP path. But what if some deployments can not use BFD (s…

The HOLD TIME is already enforced by the receiver.  The only clarification that this makes is that the transmitter should also enforce it.  

> * Assume we bring TCP down ... when does it attempt to go up again ? Are we ok to bring it up only after manual/script action from such a state ? 

That is (and has always been) at the discretion of the implementation. Automatic periodic recovery would be preferable as a means of minimizing managerial overhead.

> * As proposed it seems that the change will affect all AFI/SAFIs using given single TCP session. Even if perhaps all are perfectly healthy and running on separate cores. If each SAFI would run on a…

Creating more TCP sessions is not likely to improve the behavior of a TCP receiver.

> * The change seems applicable to both iBGP and eBGP right ? 

Yes.  It’s fundamental.

> In summary the attempt here is to fix application issues by cutting the transport. Sure half broken transport for whatever reason may not be a good thing to keep in UP state. Especially when redund…

The point here is simply a clarification for robustness. If a (half) session is not making progress, then the receiver should terminate the session. With this clarification, we make it explicit that the transmitter may do so as well. The likely scenarios where this would come into play are serious software bugs where the transmitter or receiver is not able to make progress. This could be due to transport issues or infrastructure issues. As you note, trying to continue to work with the session is unlikely to be beneficial.

> Wouldn't per AFI/SAFI heartbeat be a better option to detect if a peer's BGP stack is still up and running fine for all applications ? 

That would add considerable complexity and still not address the stuck transmitter.

Regards,
Tony

---

From: Keyur Patel <[email-masked]>
Date: Fri, 11 Dec 2020 23:30:21 +0000

One comment inlined #Keyur

On 12/11/20, 12:04 PM, "Idr on behalf of John Scudder" <[email-masked] on behalf of jgs=[email-masked]> wrote:

    [all hats on]

    Hi Job,

#Keyur: +1 to what John said. This could very well be an implementation knob that generates ManualStop event.

Regards,
Keyur

    Thanks,

    —John

---

From: Jeff Tantsura <[email-masked]>
Date: Fri, 11 Dec 2020 15:57:20 -0800

The trade-off is (as often happens) between stability and convergence.
Given severity, I’d prefer formalized approach rather than implementation artifact ( at mercy of Product Manager in charge ;-))

Regards,
Jeff

> On Dec 11, 2020, at 15:30, Keyur Patel <[email-masked]> wrote:
>

---

From: Robert Raszuk <[email-masked]>
Date: Sat, 12 Dec 2020 02:19:52 +0100

Hi Tony,

> * Is the "unable to send" only possible under Window = 0 ? What if there
> is a local NIC buffer full and we keep dropping it locally ? If we are
> going there perhaps we could say "unable to successfully send" meaning send
> and get an ACK for it ?
>
> There are many, many reasons why we might not be able to exchange bits.
> The specifics aren’t particularly relevant. The point is that we’re not
> able to make progress, so the session is clearly broken.
>

I think we all see this and agree.
My point was to be able to detect more cases of unidirectional stuck
sessions resulting in more triggers to shut it down (then just watch the
RCV WND 0 for t=HT.

> * The proposal is about reusing the HOLD TIME value to bring BGP down
> when you are still receiving keepalives however peer sent ACK for the last
> More generally, the proposal is that we apply the HOLD TIME on the
> transmit side as well as the receive side. If we are not able to transmit
> for that period of time, the receiver should give up and so should the
> transmitter. The session is broken, updates cannot flow, and we no longer
> have (eventual) consistency.
>

That means as you also mentioned below that receiver should close the
session when HOLD TIME expires. I would like to actually better understand
why this is not taking place here, before we continue.

Creating more TCP sessions is not likely to improve the behavior of a TCP
> receiver.

Hmmm if socket it full to one SAFI and other socket(s) are working just
fine to some other ones I am not sure if this is not a safer bet.

Say IGP is churning and BGP-LS is taking all resources of TCP single
socket. Then 1/1 or 2/1 will die with it - when they could run just fine
...

Hi Jakob,

> whereas NOTIFICATION and FIN precludes GR.

Then it is pretty clear that we must start with RST. Hoping that subsequent
TCP session will be healthy. Only then perhaps we should kill it hard.

Best,
R.

---

From: Keyur Patel <[email-masked]>
Date: Sat, 12 Dec 2020 01:31:16 +0000

One more comment: 

The flapping of the session may result into persistent flaps (more network instability) if the session is brought up prematurely. Most issues are a side effect of excessive throttling of resources at the protocol/system level or bugs in the implementation. That means a manual intervention would be needed before the session is restarted.  

Regards,
Keyur

On 12/11/20, 3:57 PM, "Jeff Tantsura" <[email-masked]> wrote:

    Regards,
    Jeff

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Sat, 12 Dec 2020 03:29:02 +0000

Good point Keyur.
A receiver may be overwhelmed for a long time and not open its TCP window to avoid
silly window syndrome or some other reason. The receiver may still be functional
and able to clear its backlog, albeit in a long time. Resetting such a session
will only make the situation worse. Telling the difference between this case
and a receiver stuck in a bug is difficult.

Regards,
Jakob.

-----Original Message-----
From: Idr <[email-masked]> On Behalf Of Keyur Patel
Sent: Friday, December 11, 2020 5:31 PM
To: Jeff Tantsura <[email-masked]>
Cc: John Scudder <jgs=[email-masked]>; [email-masked]
Subject: Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0

One more comment: 

Regards,
Keyur

    Regards,
    Jeff

---

From: Robert Raszuk <[email-masked]>
Date: Sat, 12 Dec 2020 10:22:53 +0100

I went back and reread the thread:

Shouldn't it be better if we first ask implementations to provide show
command/api to list all peers and min-max durations of TCP Window being 0
without actually doing any automagic RST/NOTIFICATION/FIN ?

This could allow to better understand which peers are getting behind in
their control plane and perhaps also allow to set the RST timer under such
conditions by operator? If he chooses this to be equal to HOLD TIME so be
it but I am not sure this would be universally an optimal choice.

Along the same lines we should perhaps also list per BGP peer number of
DUPLICATE ACKS, RETRANSMISSIONS etc ...

Are there implementations already deployed in DFZ allowing such data to be
displayed per each BGP peer ?

Thx,
Robert.

---

From: Christoph Loibl <[email-masked]>
Date: Sat, 12 Dec 2020 14:58:13 +0100

Hi,

Isn’t it save to assume that if a system cannot send any messages to its BGP-neighbor (for whatever reason) for HOLD TIME seconds, that the neighbor on the other side has by that time already declared the BGP session dead (is already “trying" to deliver a NOTIFICATION and has removed the routes from its RIB). If this is the case I see no point in trying to keep the session alive, because it will *always* sooner or later lead to a new session-setup + flapping routes. The BGP session (if the NOTIFICATION is queued) cannot recover from that state anymore (can it?) and is useless, even if there are chances that messages may get delivered later. 

Cheers Christoph

-- 
Christoph Loibl
[email-masked] | CL8-RIPE | PGP-Key-ID: 0x4B2C0055 | http://www.nextlayer.at

> On 12.12.2020, at 10:22, Robert Raszuk <[email-masked]> wrote:
> 
> 
>    https://mailarchive.ietf.org/arch/msg/idr/q0Sx5d3zZjfOmOQ4lO2OZAHh9Lc/ <https://mailarchive.ietf.org/arch/msg/idr/q0Sx5d3zZjfOmOQ4lO2OZAHh9Lc/> 
> 
> Shouldn't it be better if we first ask implementations to provide show command/api to list all peers and min-max durations of TCP Window being 0 without actually doing any automagic RST/NOTIFICATION/FIN ? 
> 
> This could allow to better understand which peers are getting behind in their control plane and perhaps also allow to set the RST timer under such conditions by operator? If he chooses this to be equal to HOLD TIME so be it but I am not sure this would be universally an optimal choice. 
> 
> Along the same lines we should perhaps also list per BGP peer number of DUPLICATE ACKS, RETRANSMISSIONS etc ... 
> 
> Are there implementations already deployed in DFZ allowing such data to be displayed per each BGP peer ?
> 
> Thx,
> Robert.
>

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Sat, 12 Dec 2020 14:07:03 +0000

No.

Regards,
Jakob.

On Dec 12, 2020, at 6:01 AM, Christoph Loibl <[email-masked]> wrote:

 Hi,

Cheers Christoph

--
Christoph Loibl
[email-masked]<mailto:[email-masked]> | CL8-RIPE | PGP-Key-ID: 0x4B2C0055 | http://www.nextlayer.at

On 12.12.2020, at 10:22, Robert Raszuk <[email-masked]<mailto:[email-masked]>> wrote:

Thx,
Robert.

---

From: Christoph Loibl <[email-masked]>
Date: Sat, 12 Dec 2020 15:23:02 +0100

Jakob,

> On 12.12.2020, at 15:07, Jakob Heitz (jheitz) <[email-masked]> wrote:
> 
> No.

I guess, this may be the reason why we are having this conversation. ;-) This is why I am asking: Because from reading the RFC4271 Section 6.5 I would assume that not sending messages to the neighbor for HOLD TIME will *always* result in a HoldTimer_Expires event, that leads to a NOTIFICATION, release of all BGP resources, drop of TCP connection, … and no way to recover from that.

Cheers Christoph

> On 12.12.2020, at 15:07, Jakob Heitz (jheitz) <[email-masked]> wrote:
> 
> No.
> 
> Regards,
> Jakob.
> 
> 
>> [email-masked] <mailto:[email-masked]> | CL8-RIPE | PGP-Key-ID: 0x4B2C0055 | http://www.nextlayer.at <http://www.nextlayer.at/>
>> 
>> 
>> 
>>> On 12.12.2020, at 10:22, Robert Raszuk <[email-masked] <mailto:[email-masked]>> wrote:
>>> 
>>> 
>>> [email-masked] <mailto:[email-masked]>

---

From: Jared Mauch <[email-masked]>
Date: Sat, 12 Dec 2020 10:21:19 -0500

One of the issues I've seen here is the tcp stacks in routers be very ... unique. Much of this is because there is significant desire for state handling to be replicated between dual routing daemons and line cards. 

I spent many hours with cisco about the perils of non default settings for their stack for things like keepalives. These are all the choice of bad OS level defaults and the applications not having consistent socket behavior. This example with XR is mostly to air the details of their implementation and how it would impact the discussion. 

These implementation details matter as window 0 isn't clear and may not go up the stack or may not come down enough. 

There's a delicate balance here in consume the data as quickly as possible and run the rib and fib downloading in alternative threads and when it should cause a blocking event and when the back pressure should cause the notification or tcp tear down. 

The applications should be more aware of the transport state and I'm worried this runs into the "please don't write bugs" category. It's at least close to it. 

Sent from my iCar

> On Dec 12, 2020, at 9:07 AM, Jakob Heitz (jheitz) <jheitz=[email-masked]> wrote:
> 
>  No.
> 
> Regards,
> Jakob.
> 
>

---

From: Jared Mauch <[email-masked]>
Date: Sat, 12 Dec 2020 10:32:24 -0500

> On Dec 12, 2020, at 9:23 AM, Christoph Loibl <[email-masked]> wrote:
> 
> drop of TCP connection, … and no way to recover from that.

Some vendors have really bad TCP stacks. I had one that would ack the wrong number of bytes and without setting the keep alive on the socket would block forever as it never knew it went away. 

Some have also not implemented window scaling or other modern rfcs. 

Then when you find the bugs they are concerned about the fix due to how core TCP is for the device. 

This isn't unique to any vendor but those with multiple flavors and OS it's quite interesting to say the least. 

- Jared

---

From: William McCall <[email-masked]>
Date: Sat, 12 Dec 2020 15:38:37 +0000

> and a receiver stuck in a bug is difficult.

Byzantine Generals again.

We ran into this problem recently. The update group behavior made a
single session's
slowness cascade to all of the others, so there was much screaming.

At a certain point, a slow neighbor is a pointless neighbor. If we saw a number,
say 2 days backlog, it would be an obvious problem that requires
reset. Lower numbers
get trickier.

It seems to me that we're looking at 3 categories:

1) Normal sessions - don't kill
2) Slow sessions - don't kill yet, do the slow session things.
3) Too slow - goodbye session.

Maybe a "minimum update rate" algo that would kick off #2 and #3 is a
possibility. A bit
more flexible than just looking at queue depth or getting too deep
into other layers by
looking at buffer characteristics. I know some implementations implement #2, but
I never bothered to dig into how it works. #3 just seems like a
logical extension.

When we ran into this problem recently, the only discussion of window
size was as an affirmation
that the broken box was, in fact, the broken box. But the problem
description is always
something like "stale routes" or "routing loops" (undoubtedly because
they are stale).

---

From: Job Snijders <[email-masked]>
Date: Sat, 12 Dec 2020 16:54:20 +0000

On Fri, Dec 11, 2020 at 08:03:11PM +0000, John Scudder wrote:
> that you believe BGP should tear down its session if it’s unable to
> send a message for the duration of the hold time. 

Wall of text you say... here is a lighthearted attempt to visually
explain how routes become stale routes on the Internet. ;-)

    http://sobornost.net/~job/bgp_holdtimer_tcp_deathgrip.gif

Sessions with broken peers are not torn down, which prevents much
needed WITHDRAW messages from being generated.

> Given that the conversation last time was inconclusive I think this is
> a good thing for the WG to discuss again. If you want to, you (or
> someone) could turn the idea into a short draft that updates RFC 4271,
> and we could have a WG adoption discussion about it. 

will do

Kind regards,

Job

---

From: Jared Mauch <[email-masked]>
Date: Sat, 12 Dec 2020 13:24:25 -0500

> On Dec 12, 2020, at 11:54 AM, Job Snijders <[email-masked]> wrote:
> 

Yeah but is this a protocol issue (BGP) or a transport implementation issue, or just a BGP implementation issue? We have mostly stopped using XR for a variety of issues and I'm happy to not be dealing with the transport bugs in 2020. I hope they clean up that stack soon, but given how long it's been I kinda a doubt it. 

I'm also sure the bugs aren't limited to a single vendor. Everyone's transport implementation will be different.

---

From: Randy Bush <[email-masked]>
Date: Sat, 12 Dec 2020 10:55:02 -0800

> One of the issues I've seen here is the tcp stacks in routers be very
> ... unique.

way back when, we looked at the tcp behavior of bgp on a few routers.
we gave up in disgust and did not publish (beyond a talk at nag, i
think).

maybe, over the years, newer or revised implementations are running on
linux, bsd, or other more widely reviewed and tested stacks.  this would
be a good thing.

but even behind a decent stack, the receiver might not be able to keep
up (for other disgusting reasons, historical cite albeit ospf [0]), so
the window narrows.  so i see this discussion as useful.

randy

[0] - Aman Shaikh and Albert Greenberg, “Experience in Black-box OSPF
      Measurement,” in Proc. ACM SIGCOMM Internet Mea- surement Workshop
      (IMW), November
      2001. http://conferences.sigcomm.org/imc/2001/imw2001-papers/82.pdf

---

From: John Scudder <[email-masked]>
Date: Tue, 15 Dec 2020 20:00:06 +0000

Hi Christoph,

On Dec 12, 2020, at 9:23 AM, Christoph Loibl <[email-masked]<mailto:[email-masked]>> wrote:

On 12.12.2020, at 15:07, Jakob Heitz (jheitz) <[email-masked]<mailto:[email-masked]>> wrote:

No.

When he kicked off this thread Job cited a much older thread where this had been discussed. In the first message in *that* thread (https://mailarchive.ietf.org/arch/msg/idr/q0Sx5d3zZjfOmOQ4lO2OZAHh9Lc/), Rob gives a rationale, a reasonable one in my opinion, for why it’s not safe to assume the session will be reset. Actually there are two reasons. Rob touches on one, which comes down to “if I explicitly tell you not to send anything to me, it is not reasonable to penalize you for not sending anything to me.” Admittedly, this requires some creativity in reading the spec (hint: you can pack a lot into your interpretation of “receive” in Section 6.5).

The second reason, however, is pragmatic and requires no creative spec interpretation at all. The window closing and staying closed for HOLDTIME might be a hint that the peer has lost its mind and can’t be counted on to run the protocol correctly anyway. Consider that one of the simplest reasons for failing to drain the socket would be that the daemon has fallen into an infinite loop. In such a situation, you can’t count on the peer’s behavior.

Also, a nit:

Isn’t it save to assume that if a system cannot send any messages to its BGP-neighbor (for whatever reason) for HOLD TIME seconds, that the neighbor on the other side has by that time already declared the BGP session dead (is already “trying" to deliver a NOTIFICATION

I think you are talking about this scenario. I’ll copy the example from Rob’s message cited above:

  rtr-A                   rtr-B
  (congested c-p)         (uncongested c-p)
  send window: >0         send window: 0
  recv window: 0          recv window: >0

In this case we expect:
 a) rtr-B does not send any BGP packet (KEEPALIVE/UPDATE/NOTIFICATION)
to rtr-A in normal operating circumstances.
 b) rtr-A does not expect any KEEPALIVE/UPDATE packets from rtr-B. The
session remains established even if no packet is received in the
holdtime.
 c) rtr-A continues to send KEEPALIVE packets to rtr-B.

You’re saying that you think Router A will declare Router B dead (debatable, but let’s assume it will), and that as a result it’ll produce a NOTIFICATION (yep). I’m not sure why then you say it would be “trying” to deliver that NOTIFICATION. Router B hasn’t closed its window, so Router A’s NOTIFICATION should be delivered — unless you want to stipulate other problems as well.

But this isn’t central to the discussion.

—John

---

From: Christoph Loibl <[email-masked]>
Date: Tue, 15 Dec 2020 21:57:47 +0100

John,

Thanks for answering my question in more detail. Maybe I was unclear (but reading your email I think we are talking about the same).

> On 15.12.2020, at 21:00, John Scudder <[email-masked]> wrote:
> 

The part I have a problem to understand is b). It is clear that rtr-A will not receive any packets from rtr-B because rtr-B cannot send them (send window: 0). But does "rtr-A does not expect any KEEPALIVE/UPDATE packets from rtr-B” mean that rtr-A has essentially suspended its hold-timer until it is ready to receive new messages and opens up its recv window? If yes, why? I would expect timers to run independently of the transport protocol.

Cheers Christoph

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Tue, 15 Dec 2020 21:08:54 +0000

There is a queue from rtr-B BGP to rtr-A BGP. This queue stretches through the TCP implementations on both routers. That means it consists of the TCP send buffer in rtr-B and the TCP receive buffer in rtr-A, whatever might be in the forwarding buffers on both routers and maybe some IPC queues and whatever else. rtr-A BGP only knows what it is pulling from its receive socket. BGP does not know or care what wnd=0 means. As long as it can keep pulling from its receive socket, it cannot expire its hold timer. An important check that a BGP implementation must do before expiring its hold timer is: "is my TCP receive socket empty?".

Regards,
Jakob.

-----Original Message-----
From: Christoph Loibl <[email-masked]> 
Sent: Tuesday, December 15, 2020 12:58 PM
To: John Scudder <[email-masked]>
Cc: Jakob Heitz (jheitz) <[email-masked]>; [email-masked]; Robert Raszuk <[email-masked]>
Subject: Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0

John,

Cheers Christoph

---

From: John Scudder <[email-masked]>
Date: Tue, 15 Dec 2020 21:47:54 +0000

Agreed.

—John

On Dec 15, 2020, at 4:08 PM, Jakob Heitz (jheitz) <jheitz=[email-masked]<mailto:jheitz=[email-masked]>> wrote:

[External Email. Be cautious of content]

Regards,
Jakob.

-----Original Message-----
From: Christoph Loibl <[email-masked]<mailto:[email-masked]>>
Sent: Tuesday, December 15, 2020 12:58 PM
To: John Scudder <[email-masked]<mailto:[email-masked]>>
Cc: Jakob Heitz (jheitz) <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; Robert Raszuk <[email-masked]<mailto:[email-masked]>>
Subject: Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0

John,

On 15.12.2020, at 21:00, John Scudder <[email-masked]<mailto:[email-masked]>> wrote:

 rtr-A                   rtr-B

In this case we expect:

Cheers Christoph

--
Christoph Loibl
[email-masked]<mailto:[email-masked]> | CL8-RIPE | PGP-Key-ID: 0x4B2C0055 | https://urldefense.com/v3/__http://www.nextlayer.at__;!!NEt6yMaO-gk!Qm49_lCxrC4mUilTalG2MTz0dehIKXi0GAQ2hUdoNkILySKQdek0AWSPrbUfyQ$

_______________________________________________
Idr mailing list
[email-masked]<mailto:[email-masked]>
https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/idr__;!!NEt6yMaO-gk!Qm49_lCxrC4mUilTalG2MTz0dehIKXi0GAQ2hUdoNkILySKQdek0AWQy5xj9EA$

---

From: Job Snijders <[email-masked]>
Date: Tue, 15 Dec 2020 21:54:01 +0000

On Tue, Dec 15, 2020 at 09:57:47PM +0100, Christoph Loibl wrote:
> Thanks for answering my question in more detail. Maybe I was unclear
> (but reading your email I think we are talking about the same).
> The part I have a problem to understand is b). It is clear that rtr-A
> will not receive any packets from rtr-B because rtr-B cannot send them
> (send window: 0). But does "rtr-A does not expect any KEEPALIVE/UPDATE
> packets from rtr-B” mean that rtr-A has essentially suspended its
> hold-timer until it is ready to receive new messages and opens up its
> recv window? If yes, why? I would expect timers to run independently
> of the transport protocol.

Yeah, I'd expect that too. We've seen congested BGP implementations
continue to send KEEPALIVEs but not accept (or send!) other BGP
messages. And rtr-B's attempts at KEEPALIVE just be TCP ACked with zero
window.

I'd argue in the above scenario rtr-A is simply broken and rtr-B MUST
proceed to close down the session towards rtr-A, rtr-B must cleanup and
generate WITHDRAWs for any routes pointing to rtr-A. By doing the
clean-up rtr-B does both itself and rtr-A a favor. If the issue was
transcient rtr-A and rtr-B will re-establish a few minutes later
(IdleHoldTimer, right?) and things will normalize.

Arguably and measurably, rtr-A is operating its Loc-RIB (forwarding)
based on stale routing information (assuming rtr-A is working at all!):
rtr-A has not received any WITHDRAWs, UPDATEs (or somewhat less
importantly KEEPALIVEs) from rtr-B.

Rtr-B is fully aware of this stale situation, because rtr-B was not able
to write these BGP messages to the network: the messages are still in
OutQ. Rtr-A didn't accept any KEEPALIVE (or UPDATE/WITHDRAW) from
rtr-B.

How to solve this? Claudio Jeker took a look at what it would take in
OpenBGPD and came up with the (tiny!) following patch, should be
readable to most: https://marc.info/?l=openbsd-tech&m=160796802508185&w=2

Ben Cox helped me create a 'EBGP peer from hell': a publicly accessible
EBGP multihop instance which can reliably produce the undesirable
TCP/BGP behavior we're discussing here. This 'peer from hell' will do
the OPEN exchange but then manipulates the TCP recvwindow towards zero.

All BGP implementations tested so far (5 famous ones) appear vulnerable
because they continue to consider the BGP session healthy & stable
(meanwhile OutQ keeps growing endlessly and zero BGP messages go across
the wire).

One network operator (with thousands of EBGP sessions in the DFZ)
reported to me the above stalled-TCP scenario is *not* a common case on
the Internet. On a normal day, a network operator will see no (zero)
sessions stuck this way, which leads me to believe 'recvwind=0' ...
*for the duration of the hold timer* is a very strong indicator for a
really broken situation which should be attempted to automatically
resolve.

I believe BGP implementations are not helping any known deployment
scenarios by *not* disconnecting a stuck peer, however on the other we
now know about various operational examples where honoring recvwind=0
for (hours, days) longer than $holdtimer led to global scale problems.

As the 'not-at-all progressing OutQ' situation seems somewhat rare in
the wild (yet continues to happen from time to time) I think it is worth
discussing & documenting how implementers can attempt to avoid this
state from happening. It might help make the Internet 1% more robust.

BGP implementers (or operators wanting to test their equipment) feel
free to contact me off-list if you'd like to set up an EBGP multihop
session towards the 'peer from hell' testbed. Testing potential
solutions this way is quite easy, the behavior can be triggered within a
few seconds.

Kind regards,

Job

ps. At this moment we have (1) an attempt at problem description, (2) a
demonstration BGP-4 implementation of a 'problem causer', and (3) a
different BGP-4 implementation with a 'solution'. This enables IDR to
test interopability & (potentially revised) protocol compliance,
hopefully moving the problem a bit from theoretical to practical
reality? :)

---

From: John Scudder <[email-masked]>
Date: Tue, 15 Dec 2020 22:55:54 +0000

It came up earlier in the discussion that HOLDTIME serves at least two different purposes: neighbor liveness detection, and as a metric of BGP protocol health. If the proposal is adopted, the former use is potentially problematic in deployments where the hold time gets cranked down to a very small number.[*] Of course, you can argue that this is a bad use of the BGP hold time, and that the operator should use BFD instead, but sometimes this isn’t done, for whatever reason. 

Because of this sad state of affairs, it seems to me that if we adopt this proposal, there should be a new hold time introduced, call it TX_HOLDTIME if you like. It could have the same default value as HOLDTIME, but (and this is the important bit) would not inherit any non-default value of HOLDTIME, it would have to be explicitly changed separately.

—John

[*] I assume this is obvious; I’m prepared to detail why if people don’t agree.

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 16 Dec 2020 00:18:36 +0100

Hi Job,

Putting all other concerns aside I have few questions ...

1. Is this BGP which should trigger the session RST or FIN or TCP ?

2. If this is BGP (TCP would not be aware of HOLD_SEND) how exactly do we
know that peer's window is 0 for HOLD_SEND TIME ?

3. Which TCP socket option will return BGP an error that for the duration
of X sec window for a given peer was 0 ? I presumed even if it jumped for
100 ms above 0 the timer would be reset indicating peer is still alive ?

>From your bgpd example you are not checking anything other then BGP's
ability to write to out queue. So is this the suggestion now forgetting all
about TCP layer ? Simply if I can not write anything to a peer for over X
sec RST the session ?

Hi John,

I think the suggestion is to add a second HOLD_SEND TIME different from
normal HOLD TIME.

Also there could be lost of different type of peers so unless HOLD_SEND
would be say 5 x HOLD putting all peers under same time value may be
suboptimal.

Thx,
R.

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Tue, 15 Dec 2020 23:39:52 +0000

If you tell the socket to shutdown and then close, it will attempt to
send everything in the queue with the FIN at the end.
Then wait for the FIN ACK and all manner of nonsense to bore rtr-B to tears.
So, to get on with it, send the RST.

Next question is what to do if GR is in effect.
rtr-A will dutifully retain all the routes from rtr-B and Job's beloved WITHDRAW
will still not happen.
The new session will come up (maybe), rtr-B will send all its routes again and
(if it doesn't get stuck again) will send its EOR. Only now can Job breathe easy.

Might we need a new bit in the GR capability in the OPEN message?
"WITHDRAW ALL MY ROUTES NOW"

Regards,
Jakob.

From: Idr <[email-masked]> On Behalf Of Robert Raszuk
Sent: Tuesday, December 15, 2020 3:19 PM
To: Job Snijders <[email-masked]>
Cc: [email-masked]
Subject: Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0

Hi Job,

2. If this is BGP (TCP would not be aware of HOLD_SEND) how exactly do we know that peer's window is 0 for HOLD_SEND TIME ?

3. Which TCP socket option will return BGP an error that for the duration of X sec window for a given peer was 0 ? I presumed even if it jumped for 100 ms above 0 the timer would be reset indicating peer is still alive ?

From your bgpd example you are not checking anything other then BGP's ability to write to out queue. So is this the suggestion now forgetting all about TCP layer ? Simply if I can not write anything to a peer for over X sec RST the session ?

Hi John,

I think the suggestion is to add a second HOLD_SEND TIME different from normal HOLD TIME.

Also there could be lost of different type of peers so unless HOLD_SEND would be say 5 x HOLD putting all peers under same time value may be suboptimal.

Thx,
R.

On Tue, Dec 15, 2020 at 10:54 PM Job Snijders <[email-masked]<mailto:[email-masked]>> wrote:

Kind regards,

Job

---

From: john heasley <[email-masked]>
Date: Wed, 16 Dec 2020 00:15:05 +0000

> and a receiver stuck in a bug is difficult.

It seems that closing after HOLDTIME would be fragile at boot-time of the
receiver or recovery of an IxP interface, when there is high demand -
which I think is keyur's comment.  Maybe this should not be enforced until
an EoR marker, allowing a receiver to retard new, GR, or RtRefresh peers?

Could a sender test the liveliness of a peer by attempting to open a
new session?  would a successful 3-way and commencement of BGP OPEN be
an indication that it should be more patient, increase its "deadtimer"
(HOLDTIME < STUCKTIME < PATHETICTIME)?  Clearly the remote has been
sending bgp keepalives, so perhaps not for all implementations.

could an implementation more tightly coupled to its tcp use the urgent
pointer to test liveliness?

---

From: Tony Li <[email-masked]>
Date: Tue, 15 Dec 2020 16:43:31 -0800

> On Dec 15, 2020, at 4:15 PM, john heasley <[email-masked]> wrote:
> 

Hi Heas,

In modern implementations that are multi-threaded or multi-process, the ability to open 
a new connection only demonstrates liveness of the destination stack. The upper
level thread may not be alive, either due to protocol implementation issues or
OS issues.

Tony

---

From: Keyur Patel <[email-masked]>
Date: Wed, 16 Dec 2020 00:44:28 +0000

Comments inlined #Keyur

On 12/15/20, 4:15 PM, "john heasley" <[email-masked]> wrote:

#Keyur: Or a normal route-refresh as well.

#Keyur: Please note that the problem here is mostly a slow/faulty receiver. Any session reset is less likely to solve issues unless the receiving side processing related issues are solved first.

---

From: Keyur Patel <[email-masked]>
Date: Wed, 16 Dec 2020 00:55:07 +0000

My comments #Keyur

On 12/15/20, 2:56 PM, "Idr on behalf of John Scudder" <[email-masked] on behalf of jgs=[email-masked]> wrote:

    It came up earlier in the discussion that HOLDTIME serves at least two different purposes: neighbor liveness detection, and as a metric of BGP protocol health. If the proposal is adopted, the former use is potentially problematic in deployments where the hold time gets cranked down to a very small number.

#Keyur: Yep and it has been cranked down to a very small number in number of networks.

[*] Of course, you can argue that this is a bad use of the BGP hold time, and that the operator should use BFD instead, but sometimes this isn’t done, for whatever reason. 

#Keyur: The TX_HOLDTIME would need be negotiated with a higher value (lower negotiated value can result in sending speaker somewhat abusing the receiver speaker) being chosen. __ 

Even if you have a new value (negotiated), it isn’t going to always solve Job's problem at hand. __ Assuming receiver is stuck it is not guaranteed that receiver would reset the session (sender would have to do it anyways). Then why not have a sender implement a knob that essentially says the same: Reset a session if a sending speaker cant write for more than configured interval of time (or 3*Holdtime or Holdtime interval).

Regards,
Keyur

    —John

---

From: Brian Dickson <[email-masked]>
Date: Tue, 15 Dec 2020 17:07:32 -0800

Top-thread-comment-reply (apologies to those who consider it rude).
I think this may boil down to what some of the knobs or timers measure or
impact.
The current combinations of knobs/timers in the BGP protocol are being used
to infer stuff about the TCP situation.
What really matters is, is the thing at the front of the BGP out-queue
stuck or not?
And by "stuck", the meaning is "stuck for whatever $TX_HOLDTIME is".
Attempting to infer that by adding anything to the queue itself is likely
unreliable or misleading.

Suggesting (but not requiring) implementations determine this situation
more directly, either by looking at the TCP window, or by literally
examining the out-queue, or any other technique that works, is I think what
Job is saying is required.

What to do when that situation is discovered should be pretty clear.
Nothing is going out, and nothing other than RST is likely to prevent bad
things from happening.
Any of the other corner cases being discussed would not align with the
described situation. If the queue is progressing (not necessarily draining,
which implies getting shorter), other mechanisms/techniques can/should be
used.
Only the case where the head of the queue isn't being sent is the thing
that needs to be handled, outside of the BGP state machine proper.

If I have not summarized the problem statement or required response, my
apologies, but I think making those two (problem and fix) as simple as
possible is needed to make *this* queue progress. How's that for meta?

The only correct response amounts to "kill it with fire". And I think
everything else should handle the situation on the other end. If the other
end is in a mild bug state but recovers, that's great. If otherwise, the
resulting withdrawals are a necessary side effect to keep the Internet
running.

Btrian

On Tue, Dec 15, 2020 at 4:55 PM Keyur Patel <[email-masked]> wrote:

> My comments #Keyur
>
> On 12/15/20, 2:56 PM, "Idr on behalf of John Scudder" <
> [email-masked] on behalf of jgs=[email-masked]> wrote:
>
>     It came up earlier in the discussion that HOLDTIME serves at least two
> different purposes: neighbor liveness detection, and as a metric of BGP
> protocol health. If the proposal is adopted, the former use is potentially
> problematic in deployments where the hold time gets cranked down to a very
> small number.
>
> [*] Of course, you can argue that this is a bad use of the BGP hold time,
> and that the operator should use BFD instead, but sometimes this isn’t
> done, for whatever reason.
>
>     Because of this sad state of affairs, it seems to me that if we adopt
> this proposal, there should be a new hold time introduced, call it
> TX_HOLDTIME if you like. It could have the same default value as HOLDTIME,
> but (and this is the important bit) would not inherit any non-default value
> of HOLDTIME, it would have to be explicitly changed separately.
>
> #Keyur: The TX_HOLDTIME would need be negotiated with a higher value
> (lower negotiated value can result in sending speaker somewhat abusing the
> receiver speaker) being chosen. __
>
> Even if you have a new value (negotiated), it isn’t going to always solve
> Job's problem at hand. __ Assuming receiver is stuck it is not guaranteed
> that receiver would reset the session (sender would have to do it anyways).
> Then why not have a sender implement a knob that essentially says the same:
> Reset a session if a sending speaker cant write for more than configured
> interval of time (or 3*Holdtime or Holdtime interval).
>
> Regards,
> Keyur
>
>     —John
>

---

From: Claudio Jeker <[email-masked]>
Date: Wed, 16 Dec 2020 10:17:52 +0100

> Putting all other concerns aside I have few questions ...

If you are talking about the change to OpenBGPD that Job referenced to
then the answers are...
 
> 1. Is this BGP which should trigger the session RST or FIN or TCP ?

This is pure BGP protocol level. OpenBGPD runs on UNIX systems and uses
the socket interface. There is no way to alter the TCP session state other
than calling close(2) or shutdown(2).
 
> know that peer's window is 0 for HOLD_SEND TIME ?

The TCP window is indirectly visible in the sendbuffer. If the send buffer
is full calls to select(2), poll(2) or kevent/epoll will keep the file
descriptor non-writable until there is again space on the send buffer.
So when a send(2)/write(2) call succeeds to write data we can assume that
the other side removed some data from the sendbuffer.
This may be not perfect but it is the best we can do (TCP state is a
blockbox for userland applications including BGP).
 
> 100 ms above 0 the timer would be reset indicating peer is still alive ?

There is no need to add an extra socket option. Just being able to write
data to the socket implies that progress is possible. The peer did remove
data from its recv buffer which opened the recv window. Sure it will take
some time to fill the send buffer once the recv window is stuck at 0 but
the main goal is to detect stuck sessions and reset them once it is
obvious that no progress happened during a long time (hold time).
 
> about TCP layer ? Simply if I can not write anything to a peer for over X

Yes, if a peer can not write a single byte for hold time seconds then
there is no way the peer could read any KEEPALIVES in that time and should
have sent a NOTIFICATION some time ago. Since that did not happen the peer
violated the RFC and because of that it is probably best to close the
session.

-- 
:wq Claudio

---

From: Claudio Jeker <[email-masked]>
Date: Wed, 16 Dec 2020 10:36:14 +0100

> "WITHDRAW ALL MY ROUTES NOW"

GR should not be an issue since the connection is closed with a
NOTIFICATION. At least the system detecting the stuck session will flush
and WITHDRAW all routes. In the next OPEN message this system will neither
set the R flag nor the F flag and so the stuck system will WITHDRAW
all routes as well.

The per AF "Forwarding State" bit already acts as a withdraw all my routes
now indicator.

Cheers,
-- 
:wq Claudio

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Wed, 16 Dec 2020 16:21:53 +0000

No. It's not closed with a NOTIFICATION. The send queue is frozen. No data, not even a NOTIFICATION is going to get to rtr-A. The only thing that will get there is a TCP RST and/or a new TCP SYN.

Regards,
Jakob.

> On Dec 16, 2020, at 1:36 AM, Claudio Jeker <[email-masked]> wrote:
>

---

From: Job Snijders <[email-masked]>
Date: Wed, 16 Dec 2020 16:29:47 +0000

Dear all,

A follow-up with some PCAP data to illustate the issue

>  c) rtr-A continues to send KEEPALIVE packets to rtr-B.

A PCAP showing the above scenario is available here:

    webhosted decoded: https://www.cloudshark.org/captures/6b120e111c76
    original .pcap file: http://sobornost.net/~job/bgp_tcp_bad_interaction.pcap

rtr-A == 165.254.255.17, rtr-B == 45.138.228.4
The PCAP was captured on a wiretap applied between these BGP nodes.

rtr-B is trying to send a full routing table (~ 800,000 routes) to
rtr-A, rtr-A in turn has only a few (stale) downstream routes.

Things are normal during the first 22 seconds (packets 1-149). At packet
150 it becomes clear rtr-B is not able to send UPDATEs (or KEEPALIVEs or
WITHDRAWs) to rtr-A. This situation persisted for ~ 46 minutes, at which
point I manually killed the sesssion.

During those 46 minutes (wall clock) the OutQ on rtr-B climbed to
thousands. Even if rtr-B has decent OutQ deduplication (which could
somewhat hide the detrimental effects of this situation), it is clear
from the packet capture that rtr-B is not able to follow the spirit and
intent of the BGP Hold Timers: BGP communication is completely and fully
stalled in one direction. All of rtr-A's BGP messages are TCP ACKed, but
zero progress is made sending anything from rtr-B to rtr-A.

About the OpenBGPD example solution: OpenBGPD ('bgpd') is an integral
part of the OpenBSD Network Operating System. The OpenBSD developers are
responsible for bgpd, userland, the kernel, ssh, TCP subsystem, NIC
drivers, all of it. Conceptually such a 'complete fullstack
implementation' is no different than its more impressive siblings Junos
(a complete operating system with an embedded BGP implementation), IOS
XR, or SR-OS. Customers are interested in the complete product.

I think vendors are expected to have full ownership of their entire
product: customers most likely won't care how "Inside the router
chassis" the BGP daemon(s), kernel(s), hypervisor, filesystem, NIC
drivers, etc are separate components. What matters is what actually
happens on the wire between between individual BGP nodes.

I think Takt offered a valuable insight:

    "Usually there is some tx buffering going on. Just because write()
    was successful doesn't mean a message actually arrived on the other
    hand. But if write() blocked for an Holdtimer interval it is sure
    there is an issue."
    source: https://twitter.com/taktv6/status/1338223595487719436

It's up to each implementation/vendor how to pull up state from lower
layers into their BGP engine. In the case of OpenBGPD we are fortunate
to have some infrastructure in place to accomodate this type of
improvement. Other implementations might have to come up with different
solutions depending how they designed to handle IO or buffer BGP
messages.

Compliance testing for the yet-to-be-submitted internet-draft will be
gauged simply by looking at what happens on the wire between two nodes.

Kind regards,

Job

---

From: John Heasley <[email-masked]>
Date: Wed, 16 Dec 2020 16:42:28 +0000

> level thread may not be alive, either due to protocol implementation issues or

Hey tli,
That is why I added "commencement of the BGP OPEN," but I suppose a
threaded implementation could use separate threads for various
operations, not just i/o vs other ops.

rfc721 is another thought, but rfc7805 likely kills that and Stevens IIRC
suggested the implementation was varied - some might separate it from the
normal send/recv buffers (ie: win=0 is irrelevant) and some might not
provide a way to retrieve the data in parrallel.  I haven't tinkered with
it since ~1989 on Encore Umax.

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Wed, 16 Dec 2020 17:02:53 +0000

Sorry, I responded incorrectly. The iPad only showed me a portion of the message, so I misinterpreted.

Regards,
Jakob.

> On Dec 16, 2020, at 8:22 AM, Jakob Heitz (jheitz) <jheitz=[email-masked]> wrote:
>

---

From: Claudio Jeker <[email-masked]>
Date: Wed, 16 Dec 2020 18:43:19 +0100

On Wed, Dec 16, 2020 at 04:21:53PM +0000, Jakob Heitz (jheitz) wrote:
> No. It's not closed with a NOTIFICATION. The send queue is frozen. No
> data, not even a NOTIFICATION is going to get to rtr-A. The only thing
> that will get there is a TCP RST and/or a new TCP SYN.

While true the sending system doing the reset will consider it as if it
sent a NOTIFICATION and will therefor not do a GR. Once the new session is
established rtr-A will flush its table because rtr-B will not include the
F flags. Considering that rtr-A is probably not handling any route updates
it is more important that rtr-B routes traffic away from rtr-A and this
happens.

-- 
:wq Claudio

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Wed, 16 Dec 2020 18:50:51 +0000

The restarting speaker in this case did not actually restart.
It just restarted this one session. There is no reason for it
to delete any forwarding state. There is no evidence of any
problem with its received routes, only with the routes it sent
to the stuck peer. It may still set the (F) bit to force the
stuck peer to WITHDRAW.

Regards,
Jakob.

-----Original Message-----
From: Claudio Jeker <[email-masked]> 
Sent: Wednesday, December 16, 2020 9:43 AM
To: Jakob Heitz (jheitz) <[email-masked]>
Cc: Job Snijders <[email-masked]>; [email-masked]
Subject: Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0

 

-- 
:wq Claudio

---

From: Brian Dickson <[email-masked]>
Date: Wed, 16 Dec 2020 11:17:37 -0800

> to the stuck peer. It may still set the (F) bit to force the

The transitive nature of bgp pretty much requires the safest choice should
be taken.

Which is to say, there is EVERY reason to delete forwarding state.
If a peer's router is so messed up that it is not accepting any TCP
packets, the only safe assumption is that the problem is AS-wide for that
peer.
In which case, the only data point available (the TCP session) indicates
the problem is very likely affecting other routing announcements towards
that ASN.
And the only SAFE behavior is to tear down the session completely,
including removing received routes from the IN-RIB, update the RIB and FIB,
and go on with life.

The incident that I believe lead to this proposal was super nasty, and only
a fix like this could handle that.
There is no reason to believe this can't/won't happen again in future.
As Jared notes, there are likely other implementations that would fare as
poorly if they encountered the triggering situation.

While this is my opinion on the best way to handle it, the underlying facts
aren't arguable.
An AS-wide situation (stuck receivers with no TCP progress) would never
result in the AS sending withdrawals.
The current handling of that is demonstrably broken.
The assumption that the BGP state machine can be fully relied upon is no
longer sound.

It probably never was a completely safe assumption.
IFF the state machine is working correctly, this corner case would never
occur.
It has occured and can occur, ergo it needs to be handled outside of the
state machine proper.

Brian

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Wed, 16 Dec 2020 19:26:16 +0000

Brian,

We went through this discussion when developing RFC 7606<https://tools.ietf.org/html/rfc7606>.
What errors deserve what consequences?
In implementations, we ended up making them all configurable.
We can certainly make this one optional too.

Regards,
Jakob.

From: Brian Dickson <[email-masked]>
Sent: Wednesday, December 16, 2020 11:18 AM
To: Jakob Heitz (jheitz) <[email-masked]>
Cc: Claudio Jeker <[email-masked]>; [email-masked]
Subject: Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0

On Wed, Dec 16, 2020 at 10:51 AM Jakob Heitz (jheitz) <jheitz=[email-masked]<mailto:[email-masked]>> wrote:

The transitive nature of bgp pretty much requires the safest choice should be taken.

Which is to say, there is EVERY reason to delete forwarding state.
If a peer's router is so messed up that it is not accepting any TCP packets, the only safe assumption is that the problem is AS-wide for that peer.
In which case, the only data point available (the TCP session) indicates the problem is very likely affecting other routing announcements towards that ASN.
And the only SAFE behavior is to tear down the session completely, including removing received routes from the IN-RIB, update the RIB and FIB, and go on with life.

The incident that I believe lead to this proposal was super nasty, and only a fix like this could handle that.
There is no reason to believe this can't/won't happen again in future.
As Jared notes, there are likely other implementations that would fare as poorly if they encountered the triggering situation.

While this is my opinion on the best way to handle it, the underlying facts aren't arguable.
An AS-wide situation (stuck receivers with no TCP progress) would never result in the AS sending withdrawals.
The current handling of that is demonstrably broken.
The assumption that the BGP state machine can be fully relied upon is no longer sound.

It probably never was a completely safe assumption.
IFF the state machine is working correctly, this corner case would never occur.
It has occured and can occur, ergo it needs to be handled outside of the state machine proper.

Brian

---

From: Job Snijders <[email-masked]>
Date: Wed, 16 Dec 2020 19:35:56 +0000

Jumping in to pick a nit about perceived asymmetry and practical
reality.

On Wed, Dec 16, 2020 at 06:50:51PM +0000, Jakob Heitz (jheitz) wrote:
> There is no evidence of any problem with its received routes, only
> with the routes it sent to the stuck peer. 

* The routes received from the stuck peer might block a healthier route from being used

* The routes to be sent towards the stuck peer might contain routing information for healthier routes for destinations the stuck peer is advertising.

Both the local node and the stuck peer are operating on stale routing
information. Even though the signalling observably is stuck in 'only
one' direction, the staleness applies to both directions because BGP
nodes converge together: a problem in 1 direction is a problem for both
directions, for both nodes.

Kind regards,

Job

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Wed, 16 Dec 2020 19:42:39 +0000

How far do you want to push this?
Does anyone want to restart the router?

Regards,
Jakob.

-----Original Message-----
From: Job Snijders <[email-masked]> 
Sent: Wednesday, December 16, 2020 11:36 AM
To: Jakob Heitz (jheitz) <[email-masked]>
Cc: Claudio Jeker <[email-masked]>; [email-masked]
Subject: Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0

Kind regards,

Job

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 16 Dec 2020 20:53:25 +0100

Job,

I think your observations may not always apply.

Imagine you are peering with a stub customer and he is only getting default
route from you while advertising few routes to your ASN.

The fact that you can not send to him for some time BGP UPDATES is normal
here. Queuing BGP KEEPALIVE messages could indicate some issue with the
peer's control plane, but his data plane may be running just fine.

So you want to cut his edge router off the Internet in this case ? I would
think we should be very careful not to break in any way data plane
especially when GR is in effect.

At min make before break (new session should be established - if possible)
before killing the old one.

/* I think while most implementations do not enforce the same BGP ID check
at eBGP OPEN there is something in the specs which at least originally
mandated it. */

Thx,
R.

On Wed, Dec 16, 2020 at 8:39 PM Job Snijders <[email-masked]> wrote:

> * The routes to be sent towards the stuck peer might contain routing
> information for healthier routes for destinations the stuck peer is
> advertising.
>

---

From: Job Snijders <[email-masked]>
Date: Wed, 16 Dec 2020 20:07:27 +0000

> route from you while advertising few routes to your ASN.

The stub customer still is not be receiving KEEPALIVEs, the stub has no
idea should you want to announce routes in addition to the default
route, or withdraw the default route (for some type of maintenance). All
these important BGP messages simply never make it to the stuck peer.

Flapping the session increases chances of recovery, or at least draws
operator attention.

> At min make before break (new session should be established - if
> possible) before killing the old one.

What if new session can't establish? The old session still is stuck and
most likely hurting the users of the network. As stated before in the
thread: this 'stuck' situation does *not* appear under normal
circumstances, many BGP sessions have been inspected. We now know of
multiple situations where automatic disconnection of 'stuck' peers would
have helped improve global routing.

A consession: vendors are free to make it possible to disable the new
improved behavior. If an operator knows of a deployment scenario where
after exchanging the OPEN & sending a single UPDATE (that default route)
no further bi-directional communication is required, sure. However, I'd
probably recommend considering RIP instead of BGP at that point ;-)

Kind regards,

Job

ps. No snark intended: I appreciate the working group looking at each
and every corner case. I understand it is quite unusual for someone to
point at the BGP-4 FSM and say 'I think we are hurting here', decades
into the deployment.

---

From: Job Snijders <[email-masked]>
Date: Wed, 16 Dec 2020 20:17:49 +0000

> Does anyone want to restart the router?

Not very far! :-)

(1) A node detecting its peer is stuck (in either sending or receiving
direction) only needs to stop and start the affected session.

(2) Separately, restarting the stuck router (the one advertising
recvwind=0) might also be required, but such action would be orthogonal
to the node from (1) proceeding to cleanup routes & try again.

It won't be possible to specify a mechanism for part (2), but (1) very
much is within reach. Part (2) probably is done by field engineers,
while part (1) seems an appealing automated response to improve
robostness in multi-node networks.

Kind regards,

Job

---

From: Brian Dickson <[email-masked]>
Date: Wed, 16 Dec 2020 12:58:02 -0800

> We can certainly make this one optional too.
The main thing is, even if it is configurable, it needs to have a sane
default. This is analogous to the requirement to have a configured routing
policy (any policy at all).
It should be, IMNSHO, extremely rare that anyone should want to do that
(make that override) given the global consequences of a stuck ASN.

Pretty much by definition, this is the stuff that normal control plane
stuff (including RFC 7606) could never handle per se.

A router which is slow handling updates would not trigger this issue.
It would only occur if it completely stopped working for UPDATE timer
duration.
At which point, it is arguable as to whether the routes it has are still
reasonable and legitimate.

It may be forwarding packets, but there is zero ability to guarantee it
isn't causing a routing loop.
It is quite likely that it would/could, which would further exacerbate any
problems the router is experiencing.

In which case, the result is likely better in 99.99% of potential
scenarios, to tear down the session.
I'd say 100% but someone may have a counter-example...

Brian

> Regards,
>
> Jakob.
>
>
>
> *From:* Brian Dickson <[email-masked]>
> *Sent:* Wednesday, December 16, 2020 11:18 AM
> *To:* Jakob Heitz (jheitz) <[email-masked]>
> *Cc:* Claudio Jeker <[email-masked]>; [email-masked]
> *Subject:* Re: [Idr] TCP & BGP: Some don't send terminate BGP when
> holdtimer expired, because TCP recv window is 0
>
>
>
>
>
>
>

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 16 Dec 2020 16:53:15 -0500

> (meanwhile OutQ keeps growing endlessly and zero BGP messages go across

Sadly, this is a general problem of TCP implementations.  There were some
rather interesting intentional window congestion attacks going around a few
years ago.

> really broken situation which should be attempted to automatically

I understand the intent of your text above.  However, in the presence of
very short hold timers (say 3-10 seconds), this may be too short.

It's my experience across three implementations that during extremely scaled
scenarios that a BGP receiver may indicate zero windowing easily for
multiple seconds.  Basically, if you don't manage to get CPU or other
resources to drain the socket a bit at your end of things, and you have
enough sockets, you don't get around to this fast enough.

So, while crafting your attempted mitigations, remember that a box that goes
high enough CPU to impact sessions during reconfig (e.g.) may not be the
time you want to drop the session.  It's exactly these sort of situations
that pushed implementations to perhaps ignore timers expiring when they
notice there's still data on the socket to read.

> for (hours, days) longer than $holdtimer led to global scale problems.

FWIW, implementations that have peer groups getting hung up on slow peers
simply exacerbate the issue.  However, I think the scenario should be
treated separably.

-- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 16 Dec 2020 16:55:20 -0500

On Tue, Dec 15, 2020 at 10:55:54PM +0000, John Scudder wrote:
> small number.[*] Of course, you can argue that this is a bad use of the
> BGP hold time, and that the operator should use BFD instead, but sometimes
> this isn’t done, for whatever reason. 
> 
> Because of this sad state of affairs, it seems to me that if we adopt this
> proposal, there should be a new hold time introduced, call it TX_HOLDTIME
> if you like. It could have the same default value as HOLDTIME, but (and
> this is the important bit) would not inherit any non-default value of
> HOLDTIME, it would have to be explicitly changed separately.

+1 to this.  Don't overload holdtime (negotiated) with the transmit blocked
semantic.  As noted elsewhere in the thread, a new timer that perhaps takes
hints from tcp session state (a layer violation! but not the first we'd
have) is sufficient to trigger a Stop event in the FSM.  

And that said, we'd likely need an Ungraceful Stop for RST to avoid gumming
up lingering tcp state.

-- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 16 Dec 2020 17:01:22 -0500

Brian,

> packets, the only safe assumption is that the problem is AS-wide for that

While your later text does cover concerns about AS-wide events, I'd like to
suggest that your assumption doesn't necessarily hold.

A situation where we enter a half-duplex state and simply can't get a
response to packets we're pushing might simply impact a single peering
session.  The demonstration machinery Job mentions elsewhere in-thread is
an example of this.  Active attacks against TCP windowing mechanisms are
another.

> result in the AS sending withdrawals.

For the incident in question, it's not possible for a single BGP
implementation to decide that something AS-wide is happening.  And even so,
auto-mitigation of this triggered by a single session on a single device
would be unwise.

> It has occured and can occur, ergo it needs to be handled outside of the

For the demonstrated case, a coordinated response was needed.  To some
extent, the argument is for tooling to permit easy shutdown of sessions.

Operators have plenty of provisioning machinery.  Writing up the use case
and motivations for encouraging such a thing seems like something
appropriate to an operational forum.

-- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 16 Dec 2020 17:05:37 -0500

Robert,

Using your post to make a meta point:

> displayed per each BGP peer ?

Gathering TCP state from stacks is messy at the best of times even from
standard stacks.  Doing so usually incurs the equivalent of a system call
and probably isn't something you might not be able to count on doing
regularly without potential impact on the router.

That said, it's my experience that tracking the windowing state
cross-correlated with hints of dropped packets (e.g. missing TCP ACKs) tends
to provide a good feel for the health of sessions when done in a time series
basis.  But such machinery really belongs in the TCP stacks and BGP as an
application is simply one such consumer of such telemetry.

This has the feel of work that should be spun to TSV, if any standardization
is to be done at all.

-- Jeff

---

From: Brian Dickson <[email-masked]>
Date: Wed, 16 Dec 2020 15:08:47 -0800

> auto-mitigation of this triggered by a single session on a single device
Fair enough.

Thinking a bit bigger-picture, who could or should be able to (a) detect,
and (b) respond to, a situation like this in future?
What are the pros/cons of different approaches, in terms of risk (of
accidental or malicious outages induced), or effectiveness?

I'll start:
A large network peering with another large network, is likely to have more
visibility.
If all of the sessions are stuck (but still up), that's a much stronger
indicator, and maybe that'd be a good situation when auto-reaction would be
appropriate.
Assuming the auto-reaction was limited to large peers of a large network, I
think this is less risky, and still very effective.
(There would still be a challenge of how to share this state discovery
across an ASN, but that's a more constrained problem to solve, IMHO.)

> appropriate to an operational forum.
A large ASN may want a reliable, secure method of shutting down peers for
some modest duration. Is that also something to consider developing a
solution for?
Basically, a "Shoot me now, shoot me now." kind of thing.
I don't see that possible without signatures a la RPKI, but don't know if
it's something anyone would really want to have available.
The conceptual model would be that of a rapid administrative shutdown of
peering sessions, possibly with a pre-configured timer to re-enable
sessions and/or a start time?
Maybe signed with both a personal PGP key and an RPKI key, so as to have a
great deal of control over its use and trustworthiness.

Brian

---

From: Jeffrey Haas <[email-masked]>
Date: Thu, 17 Dec 2020 09:28:38 -0500

Brian,

> On Dec 16, 2020, at 6:08 PM, Brian Dickson <[email-masked]> wrote:
> Thinking a bit bigger-picture, who could or should be able to (a) detect, and (b) respond to, a situation like this in future?
> What are the pros/cons of different approaches, in terms of risk (of accidental or malicious outages induced), or effectiveness?
> 
> I'll start:
> A large network peering with another large network, is likely to have more visibility.
> If all of the sessions are stuck (but still up), that's a much stronger indicator, and maybe that'd be a good situation when auto-reaction would be appropriate.
> Assuming the auto-reaction was limited to large peers of a large network, I think this is less risky, and still very effective.
> (There would still be a challenge of how to share this state discovery across an ASN, but that's a more constrained problem to solve, IMHO.)

This sort of analysis is amenable to current centrally collected telemetry situations.  It's just not one that would be in people's playbooks.

> A large ASN may want a reliable, secure method of shutting down peers for some modest duration. Is that also something to consider developing a solution for?

Arguably, how is this different than your billing system deciding someone hasn't paid the bill and you shut down their BGP?  In this case, it's just a matter of finding all of either the impacted peering sessions, or having a list of all peering sessions by AS.

> I don't see that possible without signatures a la RPKI, but don't know if it's something anyone would really want to have available.

Pushing this action into someone else's system is probably a non-starter.  If you think people were upset at the Dutch court attack scenario...

> The conceptual model would be that of a rapid administrative shutdown of peering sessions, possibly with a pre-configured timer to re-enable sessions and/or a start time?

I think in the scenario in question, automated and persisted shutdown is the desire.  Figuring out whether you have stable BGP, much less safe transit BGP through the provider is trickier.

-- Jeff

---

From: Gyan Mishra <[email-masked]>
Date: Fri, 18 Dec 2020 15:44:27 -0500

Jeffrey

+ Greg Mirsky

Would a simple solution be to use BFD RFC 5880 for liveliness detection
single hop in async mode with BGP to bring down the protocol BGP registered
with BFD.

I don’t know which vendor router as was not mentioned, however wanted to
point out that the default MTU for BGP is 560 and if pmtud is not enabled
on the peer which sets the MSS to interface MTU - 40 then that could have
caused the receive buffer to fill resulting in a receive window 0 situation.

Since BGP utilizes TCP window scaling it utilizes the standard TCP
congestion control algorithm.

After the initial SYN/ACK during the 3 way handshake the initial window
size is established by the SYN/ACK sent by the receiver.

So now a BGP update comes along and sends either an advertisement or
withdrawal of the internet table.

RtrA = Sender
RtrB = Receiver

RTRB sends win 65535 *ws=-28
RTRA sends BGP updates until the last segment
RTRB send Ack

** all of a sudden RTRA management plane is not able to keep up with the
BGP updates as the TCP receive buffer is full**

At this point the TCP session goes into a “Paused” state and the sender
RtrA as it has received a TCP receive window 0 message, it stops sending
BGP updates waiting for an ACK from RTRB with a new non zero window size to
unpause the TCP session so data segments can be sent again.

As the application is not a file transfer between end hosts, and is two
routers running BGP I don’t know if BGP implementation has a IPC call that
signals BGP to hang on let’s wait for the receiver RTB to clear his buffer
and signal with non zero ack.  If BGP could sense the TCP receive window 0
via IPC that would be best and immediately tear down BGP and send
notification hold timer expired.

However, regardless of the TCP to BGP IPC communication here is what
happens without BFD which resulted in the internet outage.

RTRA is still waiting for a ACK with non zero window before it can send
data.

During this time until the BGP hold time expires default 90 seconds traffic
is not able to reroute on an alternate path and we are black holding
traffic until RTRB sends BGP notification hold time expired followed by TCP
RST and BGP peer session torn down.

6.5 <https://tools.ietf.org/html/rfc4271#section-6.5>.  Hold Timer
Expired Error Handling

With BFD RFC 5880 liveliness detection single hop async mode with sub
second timers set.

Once the BFD session is 2 way established the path is declared operational
continuity test. ( excluded from S-BFD if used - highly recommend to use
standard BFD for MetroE continuity check prior to bringing up routing
protocol).

A separate BFD session is created for each communication path to monitor
the data plane bi directional liveliness detection with the registered
protocol BGP interval  X multiplier and if the number of async multiplier
control packets are not received by the other end in a row the BFD session
is declared down and BFD pulls down the registered routing protocol in this
case BGP.

In this case we are guessing that the TCP receive buffer is full because
the link is congested and so cannot process any more packets on the NIC
including BGP or BFD control packets.

So in this particular case with BFD Asynchronous mode enabled let’s say
with interval 50ms and multiplier 3 as soon as soon as Receiver RTR-B
misses 3 consecutive BFD control packets it pulls down the BGP session
within 150ms at which time RTR-B sends notification log message that the
hold time has expired and TCP RST is sent closing the session to RTR-A.

In this particular case if BFD asynchronous was enabled and was on a LAG
802.1ax but was not using BFD LAG RFC 7130 where a micro BFD session is per
member link in the LAG.

https://tools.ietf.org/html/rfc7130

So even if the vendor implementation is running non RFC 7130 micro BFD
session per member link and let’s say a single BFD session over one of the
member link where visibility is not present with all the memes links at
least in this particular case as it’s a TCP receive buffer issue and not
link integrity issue I think it would not impact BFD taking down the link.

I doubt this was the case but if there was not any link congestion which
would mean the BFD liveliness detection would not fail.  So in this case
maybe highly unlikely if only TCP was in a stuck paused state for the BGP
session however their was no link congestion.

BFD used UDP 6784 and is checking link integrity liveliness which would be
fine and not fail if the link is not congested.  So then if BGP is having
an issue with the TCP session being in a paused state is their IPC TCP to
BGP to BFD.

Would BFD be able to detect that BGP is hung due to TCP Paused state from
receive window 0?

My guess is as BGP protocol is registered with BFD, that BFD would be
monitoring the BGP protocol status and state and I would guess detect the
stuck state and bring down BGP.

I think this second scenario where the link is not congested and TCP is
stuck can be easily tested in a lab with a Spirent traffic generator.

Gyan

On Thu, Dec 17, 2020 at 9:29 AM Jeffrey Haas <[email-masked]> wrote:

> > Thinking a bit bigger-picture, who could or should be able to (a)
> detect, and (b) respond to, a situation like this in future?
> > Assuming the auto-reaction was limited to large peers of a large
> network, I think this is less risky, and still very effective.
> This sort of analysis is amenable to current centrally collected telemetry
> situations.  It's just not one that would be in people's playbooks.
>
> > A large ASN may want a reliable, secure method of shutting down peers
> for some modest duration. Is that also something to consider developing a
> solution for?
>
> Arguably, how is this different than your billing system deciding someone
> hasn't paid the bill and you shut down their BGP?  In this case, it's just
> a matter of finding all of either the impacted peering sessions, or having
> a list of all peering sessions by AS.
>
> > I don't see that possible without signatures a la RPKI, but don't know
> if it's something anyone would really want to have available.
>
> Pushing this action into someone else's system is probably a non-starter.
> If you think people were upset at the Dutch court attack scenario...
>
> I think in the scenario in question, automated and persisted shutdown is
> the desire.  Figuring out whether you have stable BGP, much less safe
> transit BGP through the provider is trickier.
>
> -- Jeff
>
-- 

<http://www.verizon.com/>

*Gyan Mishra*

*Network Solutions A**rchitect *

*M 301 502-134713101 Columbia Pike *Silver Spring, MD

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 18 Dec 2020 16:49:48 -0500

Gyan,

> On Dec 18, 2020, at 3:44 PM, Gyan Mishra <[email-masked]> wrote:
> 
> Jeffrey 
> 
> + Greg Mirsky 
> 
> Would a simple solution be to use BFD RFC 5880 for liveliness detection single hop in async mode with BGP to bring down the protocol BGP registered with BFD.

BFD is used for BGP regularly.  The use of it for ISP to ISP connections, as in the issue described, is not very typical.  Resiliency of the session is far more important for ISP to ISP communication than fast failure.

For BFD sessions to customers, fast failure is sometimes used.

> 
> As the application is not a file transfer between end hosts, and is two routers running BGP I don’t know if BGP implementation has a IPC call that signals BGP to hang on let’s wait for the receiver RTB to clear his buffer and signal with non zero ack.  If BGP could sense the TCP receive window 0 via IPC that would be best and immediately tear down BGP and send notification hold timer expired.

BGP implementations vary quite a bit.  Simpler implementations that pay only attention to basic socket APIs would simply see things like EWOULDBLOCK, EAGAIN or similar if doing async stuff.  If they're doing blocking sockets (unusual!), the implementation simply hangs.

FWIW, blocked sockets for this sort of thing is usually a socket programmer's first introduction to things that cause zero-windowing.

> During this time until the BGP hold time expires default 90 seconds traffic is not able to reroute on an alternate path and we are black holding traffic until RTRB sends BGP notification hold time expired followed by TCP RST and BGP peer session torn down.

It's important for the general case to realize that just because BGP is wedged up (control plane) that the forwarding plane may - or may not - be fine.  You can't tell from BGP.

What you do know in the abstract is that you care about the sessions being healthy in particular:
1. If you're not able to receive updates from your peer, you may end up with stale forwarding via that peer.
2. If you have stuff to send to the peer, they may end up with stale forwarding to you.

In that second case, you have a better local sense as to how urgent being stuck is.  If you have thousands of updates queued, it's probably dire.  If you have a few... is it?  If it's for a low priority network, maybe not.  If it's for google, probably much more important.

But in general, being stuck or out of sync is a problem.

But similarly, in general, the cost of dropping and re-establishing a peering session is very high.  So, there's resistance to knocking a session over because it's had some level of "temporary" hiccup.  Your definition of "temporary" will vary, and thus part of the motivation for this conversation.

> In this case we are guessing that the TCP receive buffer is full because the link is congested and so cannot process any more packets on the NIC including BGP or BFD control packets.

The fate is potentially shared, but not a guarantee.  If the congestion is happening because traffic is selectively dropping for your BGP session, BFD may behave fine.  Perhaps you have a congestion issue to your  router's CPU, but the line card's BFD is fine.

> 
> So in this particular case with BFD Asynchronous mode enabled let’s say with interval 50ms and multiplier 3 as soon as soon as Receiver RTR-B misses 3 consecutive BFD control packets it pulls down the BGP session within 150ms at which time RTR-B sends notification log message that the hold time has expired and TCP RST is sent closing the session to RTR-A.

This would be way too short for most ISP scenarios. 

> BFD used UDP 6784 and is checking link integrity liveliness which would be fine and not fail if the link is not congested.  So then if BGP is having an issue with the TCP session being in a paused state is their IPC TCP to BGP to BFD.

TCP session state is very decoupled from UDP state, so the best inference you can make is "BFD works, TCP hopefully can get through?"  But as I noted above, there's no guarantee of that.

For a different flavor of this type of problem, IS-IS doesn't use IP transport.  This means IP forwarding can be broken but you can get ISO packets through.

> I think this second scenario where the link is not congested and TCP is stuck can be easily tested in a lab with a Spirent traffic generator.

I'd suggest playing with selective packet loss for a link for a busy TCP session.  You should find that with no more than 15% of TCP packet loss that your throughput becomes terrible, and sessions may simply fail because the TCP ACK necessary to advance the window may simply not get through.

-- Jeff

---

From: Gyan Mishra <[email-masked]>
Date: Fri, 18 Dec 2020 18:29:14 -0500

Hi  Jeff

In-line

Thanks

Gyan

On Fri, Dec 18, 2020 at 4:49 PM Jeffrey Haas <[email-masked]> wrote:

> BFD is used for BGP regularly.  The use of it for ISP to ISP connections,
> as in the issue described, is not very typical.  Resiliency of the session
> is far more important for ISP to ISP communication than fast failure.

    Gyan>. Understood. Agreed.  Resiliency and stability.  Generally tuning
timers for fast convergence within an operators domain or PE-CE customer
side where I agree ISP to ISP it’s about stability and may leave default
BGP 90 second dead timer as in this case.  However,  I believe ISPs are
starting to use BFD and S-BFD for inter ISP but are very careful as don’t
want to create inter ISP instability from flapping link due to tight
timers.

> For BFD sessions to customers, fast failure is sometimes used.

   Gyan> Agreed, BGP fast external failure works well for L3 connections
where as soon as you loose connected from link down the peer immediately
goes down.  The gain with BFD is with MetroE links where OAM fault
propagation far end link down is not sent or where link stays UP connected
 to L2 switch at an IXP NAP peering point where you now end up relying on
default BGP timers for convergence.  BFD also even in LAN under floor L3
links where you can rely on fast external failover, BFD or S-BFD may still
be beneficial for one way fiber scenarios.

> BGP implementations vary quite a bit.  Simpler implementations that pay
> only attention to basic socket APIs would simply see things like
> EWOULDBLOCK, EAGAIN or similar if doing async stuff.  If they're doing
> blocking sockets (unusual!), the implementation simply hangs.

   Gyan> Understood.  I guess this could be a case where the BGP version
draft could be handy troubleshooting these types of issues.

https://datatracker.ietf.org/doc/draft-abraitis-bgp-version-capability/

>
> FWIW, blocked sockets for this sort of thing is usually a socket
> programmer's first introduction to things that cause zero-windowing.

   Gyan> Yep

>
>
> > During this time until the BGP hold time expires default 90 seconds
> traffic is not able to reroute on an alternate path and we are black
> holding traffic until RTRB sends BGP notification hold time expired
> followed by TCP RST and BGP peer session torn down.
>
> It's important for the general case to realize that just because BGP is
> wedged up (control plane) that the forwarding plane may - or may not - be
> fine.  You can't tell from BGP.
>

   Gyan> Good point.  The management plane is critical to monitor health
such as resources, memory, cpu etc do TCP state is part of the management
plane so BGP process being split between TCP sockets  part of the
management plane and BGP functions themselves.  Due to BGP use of TCP
socket it does make it more vulnerable to DDOS and of course security
concerns and ensuring authentication is enabled.  The NOC usually is
looking for down peers and would not catch missing routes until reported by
Customer outage.  Health of a peer telemetry is critical for operators
which is usually any non zero number for received routes and stale routes
or missing routes is very difficult to troubleshoot and easily missed until
you get a ticket.

In that second case, you have a better local sense as to how urgent being
> stuck is.  If you have thousands of updates queued, it's probably dire.  If
> you have a few... is it?  If it's for a low priority network, maybe not.
> If it's for google, probably much more important.

  Gyan> Yep.  I think operators for any critical peering including inter SP
should use BFD to mitigate quickly prolonged outages where every second
counts.

> But in general, being stuck or out of sync is a problem.

   Gyan> Agreed.  I think in general the stuck state is generally
 management plane TCP socket related and stale could be many other reasons.

>
> But similarly, in general, the cost of dropping and re-establishing a
> peering session is very high.  So, there's resistance to knocking a session
> over because it's had some level of "temporary" hiccup.  Your definition of
> "temporary" will vary, and thus part of the motivation for this
> conversation.

   Gyan> I disagree in the case of stuck but not stale.  I can see stale
could recover and normalize on its own possibly, where stuck prevents
convergence between onto an alternate path which once the peer bounces and
is normalized it can now take traffic.  Most ISPs load balance all their
inter ISP connections BGP multipath ECMP paths so if you reset the peer you
are much better of then black hole of traffic during the duration of the
hold timer.

> The fate is potentially shared, but not a guarantee.  If the congestion is
> happening because traffic is selectively dropping for your BGP session, BFD
> may behave fine.  Perhaps you have a congestion issue to your  router's
> CPU, but the line card's BFD is fine.
>

Gyan> Agreed.  I brought up that scenario below which I was not sure
happened in this particular instance but could happen and how would BFD
help if a localized router management plane issue.

> This would be way too short for most ISP scenarios.

    Gyan> Agreed.  Just giving an example but for inter ISP would be around
a second like 750ms is reasonable.

>
> > BFD used UDP 6784 and is checking link integrity liveliness which would
> be fine and not fail if the link is not congested.  So then if BGP is
> having an issue with the TCP session being in a paused state is their IPC
> TCP to BGP to BFD.
>
> TCP session state is very decoupled from UDP state, so the best inference
> you can make is "BFD works, TCP hopefully can get through?"  But as I noted
> above, there's no guarantee of that.

    Gyan> As BFD is detecting bi directional liveliness if the BFD control
packet is not making it especially with RFC 5880 3 way handshake session
establishment continuity test that if the BFD session cannot establish more
then likely their is a fiber cut L1 issue.  If running S-BFD it still
detects data plane bi directional liveliness but without the 3 way
handshake session establishment continuity test.

>
>
> For a different flavor of this type of problem, IS-IS doesn't use IP
> transport.  This means IP forwarding can be broken but you can get ISO
> packets through.
>

   Gyan> BFD single hop RFC 5881 can still register ISIS with BFD tuning
the timers down for fast RFC 5880 session establishment async mode link
failure detection to bring down ISIS neighbors for convergence to avoid
black hole of traffic.

> I'd suggest playing with selective packet loss for a link for a busy TCP
> session.  You should find that with no more than 15% of TCP packet loss
> that your throughput becomes terrible, and sessions may simply fail because
> the TCP ACK necessary to advance the window may simply not get through.

    Gyan> Will give it a shot

I think overall for link congestion or failure where bidirectional
continuity needs to be detected their is tremendous gain to using BFD
single hop async for BGP, OSPF or ISIS convergence.

It would be nice maybe if BFD or S-BFD or IPPM IOAM internally on the
router maybe it could run on router management plane to detect the control
plane health of socket establishment.  I would have to noodle if possible
but that could be a new innovative draft.

*Gyan Mishra*

---

From: Gyan Mishra <[email-masked]>
Date: Sun, 20 Dec 2020 16:05:38 -0500

Hi Job

I had a question about the adj-RIB-in and adj-RIB-out during the TCP paused
BGP stuck state.

Did the rtr-A rib in and rtr-B rib out did that look normal showing 800k in
out ?

So from a NOC perspective the peering looked almost normal except for the
rtr-A red flag with 0 or few routes.

rtr-A GR enabled ?
adj-RIB-in  ? did this show few routes or 0 did it change throughout the
stuck duration
adj-RIab-out 800k routes - Looked Normal

rtr-B GR enabled ?
adj-RIB-in  800k routes - Looked Normal
adj-RIab-out  did this show 800k routes out to the peer mentioned

Also from an interface level for the peer I am guessing was showing 0
traffic in either direction during the 46 minutes you probably also had a
ton of drops on the inter ISP peer in either direction.

That was probably an immediate red flag for the NOC of the stuck state.

So when the NOC discovered the issue they saw the 1 way control plane.

Did the NOC think of bouncing the peer to recover?

Bounce immediately would have forced the withdrawal and immediate
convergence.

Very interesting real world problem and thank you again for bringing to
IDR.  We definitely need to find a solid solution which looks like we are
on our way.

Kind Regards

Gyan

On Wed, Dec 16, 2020 at 11:30 AM Job Snijders <[email-masked]> wrote:

>     original .pcap file:
> http://sobornost.net/~job/bgp_tcp_bad_interaction.pcap
>
-- 

*Gyan Mishra*