Subject: [Idr] New Draft Notification:draft-dong-idr-fsm-subcode-00

From: Jie Dong <[email-masked]>
Date: Wed, 03 Mar 2010 11:49:33 +0800

Dear all, 

There is a new internet draft: draft-dong-idr-fsm-subcode-00:
http://tools.ietf.org/html/draft-dong-idr-fsm-subcode-00

The abstract is as follows: 

This document defines several subcodes for BGP Finite State Machine Error
that could provide more information to help network operators in diagnosing
BGP FSM issues and correlating network events. It also provides a mechanism
for BGP speaker to notify its peer the reason for last time session flap
after the session comes up again.

Comments are welcomed!

Thanks a lot.

Best regards,
Jie

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 3 Mar 2010 15:03:58 +0000

Jie,

Here are some comments on your draft.

> http://tools.ietf.org/html/draft-dong-idr-fsm-subcode-00

This document deals with two scenarios:
1. Reporting additional information in the NOTIFICATION message.
2. Providing information to the local and remote system about the nature of
transport session issues.

In the case of the NOTIFICATION extensions, the changes are relatively
minimal and should not impact the interoperability of existing BGP
implementations.  There are potential conformance issues which I will
address separately.  The addition of the notification subcodes provides good
additional information as to why the FSM error occurred.  However, it
doesn't supply enough information to determine what message caused the error
in question.  Additional clarification could be given in this situation by
setting the NOTIFICATION Data field to be the value of the BGP Message type
that caused the error.

Subcode 4, error cause by TCP connection, is interesting.  In this case it
is not a notification we're receiving from the peer but one that we are
effectively pretending to have received.  One place this could be visible
would be in the BGP MIBv2, e.g. bgp4V2PeerLastErrorSubCodeReceived or in the
older MIB, bgpPeerLastError.

While these subcodes are potentially interesting, they also introduce
headaches for people who have to deal with conformance testing for their
implementation.  Despite the fact that the semantics of the additional
information available in either the code/subcode and data for a given
notification may be clear, errors are often flagged by conformance tools
when there are deviations from the core BGP specification.  While this
shouldn't be a primary motivation to avoid changes to the spec, especially
for a message type meant to tear down the peering session, I would suggest
that there should be a reasonably high bar to introducing these sorts of
changes.

I am also confused about the motivation for this draft.  With the exception
of the TCP error functionality, all of these subcodes should be seen only in
cases where a BGP implementation is broken.  Seeing one of these subcodes in
an operational network would seem to me to be an indication that the router
causing it should be pulled from production immediately.  In other words,
these subcodes seem mostly useful for development of an implementation, not
deployment.

With regard to the session information capability, I'm not sure a capability
is the best way to carry this information.  It's certainly the easiest to
implement.  However, I have issues with the proposed content:

For last flap cause, this will in every case but the transport error one be
the notification message that the receiving peer had sent to the
transmitting peer.  In the absence of additional data about the message type
that caused the problem, knowing that a FSM error that occurred isn't useful
enough to help troubleshoot the broken BGP implementation.

With regard to the BGP Speaker Description information type, while US-ASCII
is certainly easy to implement, the IETF OPS group has been urging
operational content to be displayed in MIBs in an international-friendly
format with the encoding of UTF-8.

-- Jeff

---

From: Ananth Suryanarayana <[email-masked]>
Date: Wed, 3 Mar 2010 14:56:30 -0800

Thanks for your review Jeff. My replies for some of your comments in line.

-- 
> setting the NOTIFICATION Data field to be the value of the BGP Message type

TCP based errors may not be associated with a particular BGP message, as TCP
is stream based. For others, yes, we could pass on some part of the unpexpected
message received.

> that there should be a reasonably high bar to introducing these sorts of

I believe, idea here is to not only use new codes for trouble-shooting,
but also for potential BGP outbound performance improvements, by isolating
peers with TCP errors away from the rest of the normal and healthy peers
in a peer group. 

> these subcodes seem mostly useful for development of an implementation, not

I agree that TCP based error is certainly the most important one among the
defined FSM error sub codes. That is the reason, we defined a way to carry
it and inform the peer in subsequent open message.

However, making it more explicit to cover all other FSM related errors would
not hurt. (Agreed, conformance tools need awareness as well to avoid false
positives)

> implement.  However, I have issues with the proposed content:

Currently in BGP, there does not seem to be an easier alternative to add
features/add new information without breaking backward compatibility which
is very criticial obviously.

> enough to help troubleshoot the broken BGP implementation.

Intention is to consider a session 'flap'ed only when the BGP session state
moves out of established state. That really limites this last flap cause
to two of the 4 FSM error codes (3 - Receive Unexpected Message in
Established State and 4 - Error Caused by TCP connection problem)

> format with the encoding of UTF-8.

I see. We will certainly looking into UTF-8 format.

Another factor that I thought about is in traditional BGP packet debuggers like
tcpdump, wireshark, etc. They all can interpret ASCII already. So can, most of
the current BGP Implementations. 

If BGP open messages were to carry a little more information in user friendly
text form, then it seems quite handy during BGP packet dump analysis as well,
especially when dealing with dumps/logs of multiple bgp sessions.

Thanks
-- 
Regards,
Ananth

---

From: Jie Dong <[email-masked]>
Date: Thu, 04 Mar 2010 11:32:51 +0800

Hi Jeffrey,

Thanks for your comments. Please see my replies inline.

> conformance issues which I will address separately.  The 
> addition of the notification subcodes provides good 
> additional information as to why the FSM error occurred.  
> However, it doesn't supply enough information to determine 
> what message caused the error in question.  Additional 
> clarification could be given in this situation by setting the 
> NOTIFICATION Data field to be the value of the BGP Message 
> type that caused the error.
Agreed. And in this draft we just defined this in section 4 : 
" The Data field is a 1-octet unsigned integer which indicates type of the
unexpected message."

> While these subcodes are potentially interesting, they also 
> introduce headaches for people who have to deal with 
> conformance testing for their implementation.  Despite the 
> fact that the semantics of the additional information 
> available in either the code/subcode and data for a given 
> notification may be clear, errors are often flagged by 
> conformance tools when there are deviations from the core BGP 
> specification.  While this shouldn't be a primary motivation 
> to avoid changes to the spec, especially for a message type 
> meant to tear down the peering session, I would suggest that 
> there should be a reasonably high bar to introducing these 
> sorts of changes.
> 
> I am also confused about the motivation for this draft.  With 
> the exception of the TCP error functionality, all of these 
> subcodes should be seen only in cases where a BGP 
> implementation is broken.  Seeing one of these subcodes in an 
> operational network would seem to me to be an indication that 
> the router causing it should be pulled from production 
> immediately.  In other words, these subcodes seem mostly 
> useful for development of an implementation, not deployment.
As new message types may be introduced into BGP, the FSM may become more
complex than before, and implementations may need this tool for finer FSM
error diagnosing.

> With regard to the session information capability, I'm not 
> sure a capability is the best way to carry this information.  
> It's certainly the easiest to implement.  However, I have 
Yes, backward compatibility is what we concern most here.

> For last flap cause, this will in every case but the 
> transport error one be the notification message that the 
> receiving peer had sent to the transmitting peer.  In the 
> absence of additional data about the message type that caused 
> the problem, knowing that a FSM error that occurred isn't 
> useful enough to help troubleshoot the broken BGP implementation.
For FSM subcode 0-3, the message type that caused the problem is carried in
the data field of Notification message. And last flap cause is mainly used
to notify FSM errors which cannot be sent to peer using ordinary
Notification procedure.

> 
> With regard to the BGP Speaker Description information type, 
> while US-ASCII is certainly easy to implement, the IETF OPS 
> group has been urging operational content to be displayed in 
> MIBs in an international-friendly format with the encoding of UTF-8.
Thanks for pointing out this. 

Best regards,
Jie

---

From: Jeffrey Haas <[email-masked]>
Date: Thu, 4 Mar 2010 14:55:16 +0000

Ananth and Jie,

Thanks for pointing out that I had missed the notification data in section 4
of your draft.   This addresses one of the concerns I had.

> peers with TCP errors away from the rest of the normal and healthy peers

This information is already available to the local implementation at the
socket primitive level.  I'm not sure what providing this information to the
remote peer does.  Could you provide an example scenario?

> is very criticial obviously.

An implementation that has code to understand administrative information in
capabilities could just as easily advertise a very small capability that
defines that a new message type may be exchanged.  You may wish to examine
prior IDR threads regarding the INFORM proposal.

> Established State and 4 - Error Caused by TCP connection problem)

For each of these scenarios:
3. If the implementation receives an unexpected message that is the result
of a FSM error, what exactly is the implementation to do about it?  In all
of the cases that I'm aware of, this would be the result of an
implementation bug.  In such cases, existing debugging mechanisms for BGP
sessions are better suited for providing information to the vendor for
fixing the issue.  This would include packet traces of the involved messages
and not just the message type.

4. In the case of a TCP error, each side is aware that TCP dropped.  Each
side must already take this into consideration for the delay-open and the
damp-oscillation mechansisms of the FSM.  What additional information is
intended to be provided here?

> I see. We will certainly looking into UTF-8 format.

One thing to be aware of for UTF-8 is that US-ASCII is a subset of UTF-8.
This makes implementations simple.  Latin-1 (ISO-8859-1), however, cannot be
encoded without additional work.

> the current BGP Implementations. 

Adding UTF-8 decoding support to these programs is not a big deal.

My biggest concern is that human-readable text should only be used for
informative purposes and should never have an impact on the BGP FSM.

I would suggest that you consider splitting your draft into two drafts: 
1. New FSM subcodes.
2. The administrative information notification mechanism.

I clearly don't speak for the IDR chairs, but I suspect that 1. wouldn't be
very controversial and may gain WG support more readily than the combined
draft.

-- Jeff

---

From: Ananth Suryanarayana <[email-masked]>
Date: Thu, 4 Mar 2010 21:13:22 -0800

Hi,

> remote peer does.  Could you provide an example scenario?

Basically, the idea was to cover scenarios in which one side of the TCP has
no or different knowledge about the failure in the communication channel when
compared to the other side of the session. 

At times, I have seen one side of a BGP session erring out as "Hold Timer
Expired" (Because it did not receive any message for a while, though it was able
to send periodically).

The real cause could be in the other side's TCP. We thought it might be useful
to log such scenarios more accurately in both the sides. Also, if one were to
split peer-groups into slow and fast peer groups based on TCP errors, then 
it makes sense to base it on TCP errors happened not just in the local side, 
but also in the remote side.

> prior IDR threads regarding the INFORM proposal.

OK. It always seems me that incremental cost of adding a new capability is quite
low when compared to adding a new message (and hence some what altering the state
machine, etc.). For the sake of just exchanging additional nice to have
"information" about a session, a new message type seems too big a hammer to me.

And including it inside the open message seems like a better fit. (Like how
we have the 'My-AS' field today)

> very controversial and may gain WG support more readily than the combined

If the IDR community does not see the need for "exchanging" TCP related error
information, then I agree with you that we should split the draft into two
parts, and use this one with focus solely on defining the new FSM error subcodes.

Thanks again, for your kind review.
-- 
Regards,
Ananth