Subject: Re: questions on INFORM draft

From: "George Sheng" <[email-masked]>
Date: Sat, 17 Aug 2002 05:38:06 +0000

Jeff,

>From: Jeffrey Haas <[email-masked]>
>To: George Sheng <[email-masked]>
>CC: [email-masked]
>Subject: Re: questions on INFORM draft
>Date: Thu, 15 Aug 2002 17:48:06 -0400
>
>On Thu, Aug 15, 2002 at 09:20:32PM +0000, George Sheng wrote:
> > If this is only for logging, then
> > why its particular to bgp?
>
>Not speaking for the INFORM authors, the general idea is to
>allow a remote BGP speaker to send O&M type information to you.
>
>Thus, its a "I've done this, you *might* want to know about it."

that is cool. but still the question is why bgp? there are a
couple of protocols need inter-noc communications, such as PIM,
DVMRP, RSVP, LDP. For some large/huge networks, even the IGPs
at different areas are monitored by different departments/NOCs.
wouldn't they all need some version of INFORMs?

if the local SNMP traps get to "local" NMS, then a policy base
rules can be defined to process them, identify them, and take
appropriate actions. Those actions may include send an
inter-noc trap to another NOC, or send an automatic email to
notify them. This way, you don't just restrict something to
only BGP, but to anything including DDOS attack, weird traffic
pattern and other protocol INFORM msgs. Sometimes you don't
want to "INFORM" them every time it hits a condition, you
want to only do one for every 20 times; or combine two events,
if X happens, and then Y also happens, then we send them a
notification. If you want to do all those things inside BGP,
its going to have a very fun configurations. In the example
of "too many routes", for every peer, you may need to define
a "too many routes" limit since they can have different
requirements. Also those limits do change with time. Can ISPs
keep up with those configurations? You may also need to define
lots of inform-groups(similar to bgp peer-groups) to group
all the INFORM parameters for some peers. do we really want
to get into this?

you are probably thinking this(local NMS sending traps to
other NOCs) puts to much development burden
on ISPs. Yes. It can be developed in house if ISPs want to
take this type of control; or it can be new features for
NMS software vendors; or it can be a router software function.
Does "SAA" ring a bell? I think it stands for "service
assurance agent". Can we do something like that? call it
"INENA"(Inter-Noc Event Notification Agent), it should allow
event mapping into rules, and send out snmp traps to diff
NMS stations.

>--
>Jeff Haas
>NextHop Technologies

george

_________________________________________________________________
Join the world’s largest e-mail service with MSN Hotmail. 
http://www.hotmail.com

---

From: Jeffrey Haas <[email-masked]>
Date: Sat, 17 Aug 2002 14:07:58 -0400

Avi,

On Sat, Aug 17, 2002 at 03:07:47AM -0400, Avi Freedman wrote:
> Isn't this also something to eventually replace the NOTIFY behavior?

Not at all.
Again, I'll let the authors comment.

> Right now the default behavior is pretty evil, and every time 
> someone leaks a reserved AS which is generally transitied all 
> the paranoid vendors' routers reset their Cisco sessions.

Oh ye gods and little fishes.
BGP doesn't care a rats suitcase about "reserved AS" numbers.
You see reserved AS's leaking into the Net all the time.
Please help to stop the spread of this insidious rumor.

The error in question was the fact that a CONFEDERATION segment
(see RFC 3065) leaked outside of a confederation boundary and was
propagated across the Net.  When a router that didn't have this
bug received a confederation segment and the peer was not a 
confederation peer, it was treated as an error and the NOTIFY sent.

If you wont believe me, I'll actually take the time to find a 
freeware BGP packet generator and build the packets for you.
I'll even build it with non-private-as confederation segments.
I'll leave it to you to find instances of the propagator and
also the receiver who sends the NOTIFY.

Normally, you would only see this from an adjacent peer and the
confederation segment would be right at the start of the as path.
If you get this from a peer that you don't have configured as
a confederation peer, someone has a screwed up configuration and
you probably don't want their routes.

The fact that these confederation segments were able to occur in
the middle of a path means that:
1) Someone received the confederation segment and wasn't a
   confederation peer.
2) Propagated them.

1 is bad enough.  2 means you're leaking buggy stuff.

Even if you discard the confederation segment, you now have an AS
that doesn't have itself in the path and can lead to internal
loops for that AS.  You also very likely have routes that are
being leaked to the net unintentionally and a lot of missing policy
that should have been applied.

You treat your confederation peers different policy-wise than your
AS external neighbors?  Thought so.

> I don't love the idea of a BGP extension just for status but
> something that would help people get comfortable moving 
> "broken-packet" notifications into something that some providers
> could watch for and drop sessions and some could not...

I think the idea of "fixing" broken packets is generally abhorrent
as a general case and am queasy at the notion even in specfically
understood well-known cases.  At the very least, these belong in
the "museum of broken packets".

I happen to like the idea of being able to pass along some information
between routers.  INFORM, in this sense, isn't horrible.  The trick
is to make it work generically and not use it where a NOTIFY really
should be used.

> Avi

-- 
Jeff Haas 
NextHop Technologies

---

From: Jeffrey Haas <[email-masked]>
Date: Sat, 17 Aug 2002 14:19:27 -0400

George,

On Sat, Aug 17, 2002 at 05:38:06AM +0000, George Sheng wrote:
> that is cool. but still the question is why bgp?

Why not?  Has the idea been previously proposed for other protocols?

> there are a couple of protocols need inter-noc communications,

For each of these, there may be appropriate information that should
be conveyed.

> For some large/huge networks, even the IGPs

In the case of IGP's, SNMP is sufficient.

> Those actions may include send an
> inter-noc trap to another NOC,

Exactly.  But:
1. Do you want SNMP coming from the other provider?
2. Why should the information come from a remote NMS rather than
   the adjacent router?  You do provide some useful justifications:

> This way, you don't just restrict something to
> pattern and other protocol INFORM msgs.

I'm going to take the cop-out on this.
A generic inter-noc protocol is out of scope for BGP.
If you can provide such a thing, we should use that instead. :-)

> In the example
> requirements.

This is the way many providers already work.

> Also those limits do change with time.

And this is the way providers usually deal with things.

> Can ISPs keep up with those configurations?

As soon as someone proposes what they want the traps to look like,
they'll show up in the bgp v2mib.

> You may also need to define

Why would you?  Just like you would NOTIFY a given peer, you INFORM
them.  Its up to the remote implementation to forward the information
to the party that cares about it.

> other NOCs) puts to much development burden on ISPs.

What I am thinking is that ISPs must already provide monitoring
services by monitoring their routers already.  This just provides
another piece of information that must be collected, summarized and
processed.  As stands, I don't think INFORM is good enough to
be summarized and easy to process.  But its a young proposal yet.

-- 
Jeff Haas 
NextHop Technologies

---

From: Jeffrey Haas <[email-masked]>
Date: Tue, 20 Aug 2002 10:17:43 -0400

On Sat, Aug 17, 2002 at 02:58:52PM -0400, Avi Freedman wrote:
> That said, I will go back and recheck the behavior of various
> routers for reserved ASs appearing outside confederation segments.
> Perhaps that's not an issue any more.

That would be appreciated.
It may even be worthy of an ID - or at least something for the
museum of broken packets.

> I think we might be agreeing - I'm not suggesting that fixing broken
> packets is the desired thing, but rather that not requiring that
> one broken packet being propagated take down a peering session is
> the desired thing.

Here is the trick:
When I receive a packet that is either well- or mal-formed
that wasn't originated by the adjacent router, how can I tell that
the information is the individual fields is mostly correct?

Pedro from Juniper has made this point a few times before.  If
we think the data in the packet is invalid, we may not be able
to trust the NLRI fields.  Given that BGP is a stateful protocol,
you MUST be able to trust at least that much for purposes of deciding
"foo" path attribute is broken and potentially:
o Ignorable/Discardable
o Fixable

and then doing the right thing with the prefixes that go along with
the brokenness.

The best heuristic we can do is know about certain well-known types
of brokenness and provide knobs to work around them.  Anything else,
you'd be playing Russian Roulette with your routes.

If you can provide a solid heuristic of the type of potential
brokenness you're willing to put up with in your network, let us know. :-)

-- 
Jeff Haas 
NextHop Technologies

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 23 Aug 2002 10:36:12 -0400

On Tue, Aug 20, 2002 at 01:02:17PM -0400, Avi Freedman wrote:
> That would be a good idea.  Even just a museum web page.

I look forward to seeing such a thing.  

> I don't care; my suggested heuristic is ignore if any part of it
> is syntactically or semantically incorrect.  Thus, also don't
> pass it on.  But I don't want to kill my sessions because of
> one or 1000 malformed UPDATEs.

BGP is stateful.  If, for example, you get state for Yahoo,
and you get a corrupt packet for Yahoo's NLRI, do you really want
to keep using it?  What if Yahoo's NLRI isn't in that packet
but it should have been?

We currently don't have a way, in the presence of error 
conditions, to assert that we have good or bad state.

That prefix you propagate for Yahoo may cause you more calls to your
NOC than dropping the peering session.  This could be because
the path information is wrong and you've introduced routing blackholes
or loops.

What is your actual concern with dropping the peering session
operationally?  The fact that you've lost *their* routes or
that *your* routes (which are perfectly fine) get flapped?
Or is it that this may happen repeatedly?

-- 
Jeff Haas 
NextHop Technologies

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 23 Aug 2002 11:11:00 -0400

On Fri, Aug 23, 2002 at 10:51:53AM -0400, Avi Freedman wrote:
> I haven't seen or heard of any packets that were corrupted in
> the NLRI.  AS_PATH mostly, and usually by the origin AS or
> the immediate upstream AS.

This may be why we aren't seeing quite eye-to-eye.
You've seen some particular bugs related to AS_PATHS.
Many of the people on this list implement the protocols.  Buffer
creation (painting) errors are pretty common.  Thankfully software
engineer, debugging and QA work and you generally don't see these
in the real world except under edge circumstances that weren't
caught by some test.

My concerns are about the general issue, not a specific instance of
it.  Well known bugs (started that ID yet? :-) are worth working around.

> That's true.  But of course that hints at another problem,
> somewhat operational, which is that we have no idea whether
> any origin AS should be announcing a particular prefix.

Different issue, but the origin AS issue is definitely worth
addressing.  BGP is likely not the place to do it.  Policy
implementations and a distributed policy distribution system
(RPSL et al) are a good candidate for it.

> Right now there's a huge amount more churn in the BGP system than
> there 'should' be from the topology changes happening at the edge.
> Noone knows 'why' this is, though people are starting to try to
> figure it out.

And its very interesting stuff to watch.  For my part, I expect
a pretty big amount of churn any time a topology change happens.
Its a expected side-effect of path vector.

> As someone who wants the Internet to work, I don't want the NOTIFY
> mechanism to be an attack vector on the Internet infrastructure.
> (Someone getting 10-20 t1s, speaking BGP using their own implementation,
> and injecting UPDATEs for a few minutes at a time that cause Ciscos
> and Junipers to break sessions to each other).

The right answer to this is conformance and exception testing.
Problems need to be caught where they occur - not later down the chain.

-- 
Jeff Haas 
NextHop Technologies