Subject: [Idr] BGP Auto-Discovery Protocol State Requirements

From: Jeffrey Haas <[email-masked]>
Date: Tue, 16 Mar 2021 17:02:03 -0400

https://tools.ietf.org/html/draft-ietf-idr-bgp-autoconf-considerations-00

To avoid cluttering the adoption call thread, this is sent separately.

Section 2.3 lists the following requirements.  This thread is to discuss
them.

: BGP Session Transport State:
: o  IP addresses
: o  Transport security parameters
: o  GTSM [RFC5082] configuration, if any
: o  BFD [RFC5880] configuration, if any
: 
: BGP Session Protocol State:
: o  AS Numbers
: o  BGP Identifier
: o  Supported AFI/SAFIs
: o  Device Role

A valid criticism I'll take as editor of the document is the session
protocol state section should have had additional material noting that it
may be procedurally optional.  Whether it's optional depends on the desired
behavior of the protocol.  The proposals to date in most cases cover one or
more of these items and tend to universally lack justification as to why
they're in a given document - or not.  

The following attempts to discuss the motivations from design team
discussion during 2021.

-----

To get to a stage where BGP can even send an OPEN message, you have to 
be able to get TCP to connect.  This state is covered by the BGP Session
Transport State.

IP address: MANDATORY
BGP can't come up without knowing what the IP address is.

This state may be carried explicitly in the protocol, or potentially derived
from the protocol transport.  For example, the source address of the
discovery protocol, if it uses IP.  

Even if it uses IP, there are design choices:
- Do you require a discovery protocol message in the address family of the
  peer?  For example, IPv4 discovery message to carry only IPv4?

  + This doubles messages if you support v4 and v6.

- What do you do about mixed-mode peering, e.g. RFC 5549?

---

Transport security parameters: MANDATORY
That parameter may be "NONE" and thus plain TCP.  

If you're requiring TCP-MD5, TCP-AO, or something a bit more exotic like
IPsec, the TCP state machine may not even advance unless you're using the
correct parameters.  

Keychain hints may be needed if you want to use one of these.  

---

GTSM: MANDATORY
If you're using GTSM for your sessions, TCP may not even establish unless
it's on.

Robert points out existing security considerations about using GTSM.
They apply here as well.

---

BFD Configuration: OPTIONAL?

Some BGP implementations don't let their sessions come up or stay up if
there's a mis-match in BFD between the systems.

See draft-ietf-idr-bgp-bfd-strict-mode.  Making support for this draft
mandatory would let this state requirement be dropped.

======================================================================

Once you have the capability of bringing up a BGP session, you have to
decide if you even want to.  There are at least two design choices, each
with specific consequences:

1. The implementation is willing to determine if a peer is acceptable or not
   based on the information exchanged in the BGP OPEN message.  Call this
   "discover at OPEN".  A further implication of this mode is the
   implementation must be willing to service incoming sessions that may drop
   just so they can figure out if they find each other mutually agreeable.
   This becomes a scaling discussion.

2. The implementation wants to know the properties of its potential peer to
   decide if it wants even try to bring up a session.  This enables "peer
   selection".

   The good property is that the peer sending such messages doesn't have to
   service sessions only to immediately drop them.

   The bad property is that this increases disclosure of some BGP
   information to parties that may not eventually be permitted to peer.
   (Security consideration.)

---

A concomitant consideration for implementations using either strategy is the
retry mechanism.  Presume that upon discovering a BGP speaker using the
auto-configuration mechanism that either participant decides that it doesn't
want to continue peering.  Simple examples may be wrong AS number (relevant
for data center fabrics), existing peering session on a different link
(resolved by BGP Identifier), or mis-match in capabilities.

When things aren't acceptable, how should you retry?

1. If you use "discover at OPEN", you have to figure out how often to try to
   bring up a session to see if things have changed.  Perhaps you require a
   manual reset after some interval.  Either way, you're repeatedly trying to
   establish a session and thus it's a scaling consideration.

2. If the peer selection criteria is present in the auto-discovery
   mechanism, the discovering peer can simply wait until it discovers sessions
   with acceptable properties.

   Such a mechanism does introduce the edge case where the discovered
   session's propeties may not match those in the BGP OPEN message.

Auto-discovery protocols need to document how retry will be expected to
work.  The security and transport ADs will want the considerations dealt
with prior to moving to RFC.

-----

The typical state for session selection are the AS Number(s), BGP
Identifier, and supported AFI/SAFIs.  Multiple of the proposed protocols
carry this state.

Device role should not have been in the MUST state, but discussion on how to
auto-configuration is deployed specifically for BGP datacenter fabrics has
raised questions that point to some form of this repeatedly.  This items is
understood to be a contentious point and was going to require IDR list
discussion.

-- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Thu, 18 Mar 2021 12:41:23 -0400

Off-list discussion with Tony Przygienda did suggest the following can use
some clarification:

> mandatory would let this state requirement be dropped.

These two are effectively a bit each.

GTSM needs to know if it's running or not.  

BFD needs to know if it's running or not.  BFD itself handles negotiating
sessions.  No parameters need to be exchanged.  BFD session type is derived
from the IP address of the session.

-- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Thu, 18 Mar 2021 15:19:36 -0400

Further off-list conversation with Tony Przygienda offer a related
observation on the BGP Session State: They serve as the "offer", and the
management of its lifetime is to a large extent the critical detail covered
by the "retry" behavior.

This potentially creates an option between 1 and 2 below.  A BGP
Auto-Discovery listener that is implementing "discover at OPEN" mostly needs
the hint that "something has changed" for the parameters that may be
discovered at OPEN.

A version/generation or similar field in the auto-discovery protocol would
suit this purpose and permit a "discover at OPEN" BGP Speaker to minimize
the load upon the auto-discovered device.  

This would permit the entire set of BGP Session Protocol State to be omitted
from the protocol, as long as "discover at OPEN" was acceptable.  It also
addresses the consistency issue noted in section 3.6 of the
autconf-considerations draft.

---

Note that the majority of proposals analyzed for BGP auto-configuration in
Appendix A published some subset of the state in the BGP Session Protocol
State requirements.  Authors of those proposals may wish to comment on their
reasoning for that state in their protocol.

-- Jeff

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 19 Mar 2021 10:54:19 +0100

Jeff,

I am not sure if by "discover at OPEN" you mean just convention to be used
in the draft or some flag/field.

If this is just the convention it is fine. Discover at OPEN is exactly what
I had in mind writing
https://tools.ietf.org/html/draft-raszuk-idr-bgp-auto-session-setup-01

If however you think about signalling it as part of auto-discovery then I
think we do need to keep in mind that BGP4 MUST continue to operate fine
without auto discovery. That means that all negotiation and checks still
MUST happen natively in BGP - so they will happen anyway.

Or maybe you wanted to relax this and say if lots of things have been
already auto discovered we can send OPEN-lite ?

Of course down the road we may reach conclusion that configuration of BGP
session is the thing of the past and bump BGP version to operate in the
"modern" style but this moment I think still a bit ahead of us.

Cheers,
R.

On Thu, Mar 18, 2021 at 7:58 PM Jeffrey Haas <[email-masked]> wrote:

> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr
>

---

From: Tony Przygienda <[email-masked]>
Date: Fri, 19 Mar 2021 11:38:54 +0100

Once you go to version number the issue of underlyng transport behavior
starts to come to the fore. UDP & sometimes L2 can misorder, _badly_, from
experience. In most extreme cases in virtualized environment I saw UDP
being buffered & misorderd up to 5 seconds delay. That''s surely extremely
and until seen bits hard to believe but here you are and you start to look
once your adjacencies with 3 sec lifetime start to bounce & do _really_
weird stuff on AF negotiation (since one node talks to the other being 5
sec in the past ;-)  I debugged that one by establishing a baseline clock
using reflected numbers in hellos because otherwise none of the logs made
any sense ;-)

-- tony

---

From: "Acee Lindem (acee)" <[email-masked]>
Date: Fri, 19 Mar 2021 12:14:34 +0000

The avoidance of the these complexities is one advantage of an independent discovery protocol. Another advantage is that it could serve use cases beyond BGP. 

Thanks,
Acee

On 3/18/21, 3:00 PM, "Idr on behalf of Jeffrey Haas" <[email-masked] on behalf of [email-masked]> wrote:

    ---

    -- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 19 Mar 2021 09:05:32 -0400

> in the draft or some flag/field.

As I tried to make clear in a few formats, there are two things you need to
discover a BGP speaker, or perhaps more precisely an ACCEPTABLE one:
1. You need enough information to bring up your BGP peering session.
2. You need to see enough of the session's parameters to know whether you'll
   keep the session, or drop it.

"Discover at OPEN" means you don't get the BGP session's parameters in the
discovery protocol itself.  You have to get them by connecting to the
discovered peer.

> https://tools.ietf.org/html/draft-raszuk-idr-bgp-auto-session-setup-01

See Appendix A.3.2 for the analysis.

> MUST happen natively in BGP - so they will happen anyway.

Discovery doesn't change any of that.

> "modern" style but this moment I think still a bit ahead of us.

Discovery doesn't change the BGP state machine.

I think a point lacking in most of the proposals is how they're intending
the feature to be used.  That's setting a number of conflicting expectations
in the heads of the authors.

Examples:
1. Whatever you plug into this port will be fine, just connect me!
2. I'm willing to only connect to peers over ipv6 link local, or some
   specific family.
3. I'm only interested in peers that support specific AFI/SAFI
4. I'll only connect to peers that acceptable ASes in this range.
5. I only want one peering session with a given router.

etc.

To be able to determine if a peer is going to be acceptable, you have to
know something about what it is going to do.  You can get that information
from the auto-discovery protocol and not need to establish a BGP session, or
you can establish a bgp session and get it that way.  

If you continue because the session is acceptable, this is the easy case.

If you decide the session unacceptable, what do you do?
1. If you get the session parameters from discovery protocol, you wait until
   you see an offer that is acceptable.  "Shouting it to the world" is cheap.
2. If you get the session parameters from starting a bgp session, how do you
   decide whether you try again later?  

The "keep hammering on BGP" option is problematic.  Several implementations
start rate limiting incoming session requests.  This is because an incoming
session causes resources to start to be allocated, even if they're just
sockets.  

This is generally not a problem for existing BGP speakers because you don't
configure what you don't want to use.  But once you introduce
auto-configuration, you end up with the same problem as hundreds of sessions
that are misconfigured trying to get into your box every X seconds.

-- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 19 Mar 2021 09:11:32 -0400

Tony,

> using reflected numbers in hellos because otherwise none of the logs made

Thankfully, I think that this mechanism is likely to be slow and dumb enough
for misordering to not be too much of a problem.

The intent for BGP auto-configuration is to tell a listener "go peer here".
Truth exists in the actual BGP session.

Stale information means that we're trying with potentially wrong state, but
correcting that in a few seconds would likely be fine.  Changes to session
parameters are likely being done at human time scales, from one
configuration commit to another.

The version is mostly there to say "go try again", and is there in this form
because the BGP Session Paramters are discovered from the BGP session
itself.  It's almost simple enough that a single bit of "I've changed since
last time" would be sufficient, but we know that's not good enough. :-)

-- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 19 Mar 2021 09:16:22 -0400

Acee,

> The avoidance of the these complexities is one advantage of an independent discovery protocol. Another advantage is that it could serve use cases beyond BGP. 

I guess that depends on what things we're talking about.

For "how do you get a TCP speaker to come up for an application with
partially known parameters", longer term it might be nice if the tcpm
Working Group took this up.

They've partially addressed this using TCP-ENO (RFC 8547).  At some point
BGP would benefit from having an informational document describe
interactions with it.  The GTSM option doesn't fit well there.

BFD state for BGP is an artifact of interoperability headaches for the BGP
FSM and the BFD FSM as you well know. :-)

-- Jeff

---

From: Tony Przygienda <[email-masked]>
Date: Fri, 19 Mar 2021 13:56:27 +0100

I don't see how the "discovery protocol" chosen matters except the
transport things I mentioned (and fragmentation if you kitchen sink it and
choosing how simple it is to implement vs. how close to hardware, UDP is
one point in spectrum, the other taking LLC SNAP type AFAIS ;-).

the key distinction I see here is whether you go along the philosophy of
"only minimum to reliably bring session(s) up" (which is already enough,
BFD, AF mixes, faith sharing and all the stuff IGP does today for BGP are
non trivial to get right) or "kitchen sink incl. replicating capabilities".

after that you still have to deal with semantics of an "offer". In RIFT
after pondering that for quite a while and despite Jeffrey pushing for
independent protocol the whole negotiation of ZTP stuff including levels
etc has been put onto LIEs (hellos) for the reason that LIEs have timeouts
on them so it's pretty clear how long something is of importance.
Obviously, IGPs are much simpler because the lifetime of offer is @ same
time lifetime of the "session" whereas here the TCP session is kind of
"floating" around the offers. Thinking of which, the 3-way stuff in IGPs
has saved our bacon more than once and this discovery protocol should
probably put that in also deal with stuff like MTU size given that
mismatching those makes for a bad hair day and there is no IGP now to do
all the heavy lifting for the mighty BGP to demean itself to run the
allmighty TCP protocol over it ;-)

-- tony

On Fri, Mar 19, 2021 at 1:14 PM Acee Lindem (acee) <acee=
[email-masked]> wrote:

> The avoidance of the these complexities is one advantage of an independent
> discovery protocol. Another advantage is that it could serve use cases
> beyond BGP.
>
> Thanks,
> Acee
>
> On 3/18/21, 3:00 PM, "Idr on behalf of Jeffrey Haas" <
> [email-masked] on behalf of [email-masked]> wrote:
>

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 19 Mar 2021 09:50:26 -0400

Tony,

> one point in spectrum, the other taking LLC SNAP type AFAIS ;-).

There's some discussion in the draft about what transport layer we use, and
a few of the headaches.  For BGP auto-discovery, the major implication is
how wide your want your messages to travel.

I won't argue your issues about the headaches in a given layer. 

> non trivial to get right) or "kitchen sink incl. replicating capabilities".

Do you have to bother the BGP speaker to figure out if you want to talk to
them or not?  

If you do, how do you handle retries.

Those questions have to be answered.  You can not like a given mechanism all
you like, but if you want to provide constructive criticism, describe how to
do it better.

> on them so it's pretty clear how long something is of importance.

Lifetime is a good point, and one that isn't in the current draft.  Section
3.2 mostly talks about how chatty to be, which might hint a bit about
implicit lifetime.  (E.g. 2-3x the advertising window.  BFD makes that
explicit for reasons you'll likely get to.)

> allmighty TCP protocol over it ;-)

The IGPs might get it right if they use padding TLVs, but it was certainly
not baked in up front.

The autoconf considerations draft is intentionally silent on whether the
design MUST have some sort of adjacency.  Some of the proposals are "shout
into the dark" multicast style.  draft-xu-idr-neighbor-autodiscovery
basically replicates LDP state machinery, including a full adjacency.  But
it also discusses impacts on the BGP FSM by tying the fate of the BGP
session to the auto-discovery session.  

MTU mis-matches are definitely a topic of interest elsewhere in IETF.  My
personal opinion is that trying to solve it in auto-discovery is problematic
in a few respects.  The biggest one is that the discovery protocol may run
at a different layer (L2/L3) or different address family than the actual
session.  Even if we did discover an acceptable MTU, most stacks want to
interact with this either through configuring TCP MSS or performing PMTU-D.
Whether you want this stuff to get added to your RFC 4271 style BGP session
setup is where that leads.

-- Jeff

---

From: Tony Przygienda <[email-masked]>
Date: Fri, 19 Mar 2021 14:36:08 +0100

agreed roughly. I don't have any particular likes or dislikes but observe
that until you get an agreement on whether you use the Occam's razor of
"only and only what gets TCP connected" or not amongst yourself you will
not have an easy time to reach agreement on a specific design

the mtu observation about packets going over different transport than TCP
session is pretty good. On IGPs the padding is bit of a kludge (but in a
sense a good kludge since it checks the "reality" of packets passing), I
would venture that including MTU & other parameters description in the
packet to make sure both sides have acceptable things and with 3-way also
see each other and don't shout two-way only is a better design.

-- tony

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 19 Mar 2021 10:07:43 -0400

> not have an easy time to reach agreement on a specific design

Hence the dt document.  We have 2 (now 2.5) possible solutions to the state
problem.  The proposals analyzed use both of them in different flavors.  The
authors of a proposal should be able to justify why they put the state they
did into them.

The dt document says "this is a problem to be solved".  The WG gets to pick
what solution it likes.

> see each other and don't shout two-way only is a better design.

BFD for large packets[1] leverages similar tricks and is intended to be used
as a service.  But much like the observations karp made about crypto,
running even a generic service is problematic based on your bootstrapping
considerations.

An IGP with MTU detection can protect BGP... if BGP uses an IGP.
BFD can protect multiple services... if BFD gets to come up first - but that
may depend on routing.
Etc.

-- Jeff

[1] https://tools.ietf.org/html/draft-ietf-bfd-large-packets

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 19 Mar 2021 14:49:05 +0100

Jeff,

> how wide your want your messages to travel.

And this is precisely the fundamental point we would need to agree on.

Currently the answer is any transport. I recommended to limit it *only* to
L2 links (physical or emulated).

For the DC use case I think this is more then enough.

And if you take that as a base then to automatically establish BGP session
is not that hard. In fact you may not need any protocol at all.

*All you need is two steps: *

*Step 1 - Find out who is on your link (directly via ICMP or by looking at
likely already running LLDP or CDP :)*
*Step 2 - Among peers discovered in step 1 for each address you check if he
is listening on 179. If so you send him an OPEN. *

Done.

Some suggested to add this info to LLDP from step 1 which is not bad idea
either. But maybe IDR has no power to recommend additions to LLDP.

The above works fine for p2p and p2mp ethernets just fine. You can bring up
BGP for your DC core and you can bring BGP to your computes from TOR or
virtual instances running on computes as long as they talk
direectly to the underlay LAN.

- - -

Side comment: When we worked on single side BGP we had very similar
discussions. But large SP was very clear with the requirements ... If I
enable auto peering on my LAN towards customer all I care to accept his
sessions is MD5 match.

Cheers,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 19 Mar 2021 15:00:03 +0100

See inline ...

> 5. I only want one peering session with a given router.

Most if not all of that can be expressed in the auto peering policy
template.
If needed, operators can define rules managing auto peering and placing
some additional controls on it.

Also part of such a template maybe limiting one side to be passive only.

> that are misconfigured trying to get into your box every X seconds.

If you do not expose port 179 to those 100s of peers no one will ring your
door bell from that side. If you do then it is no different then today ...
anyone can send TCP OPEN on that port which you need to respond to in some
way.

But again is that really a real risk if we scope it for DC use case ?

Thx,
R.

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 19 Mar 2021 10:34:48 -0400

Robert,

> And this is precisely the fundamental point we would need to agree on.

Certainly one of them.

> For the DC use case I think this is more then enough.

Does that mean:
1. L2 encapsulated multicast
2. L2 encapsulated unicast to a well known MAC?
3. L3 multicast targeted to a single link (e.g. all-routers)?
4. Other?

> likely already running LLDP or CDP :)*

See Appendix A.2.

ARP/ND can provide such a mechanism, but triggering them can be tricky.

"Ping the broadcast address" can cover this as well, and is an old and
common hack.

LLDP has a proposal. It had to add some of the state discussed in the design
team draft and is somewhat incomplete.  See Appendix A.1.1

CDP is proprietary and isn't a good candidate for IETF work.

Coworkers working on our implementation of A.2 note that "Router
Advertisement" works fine for this property, although it has extra semantics
that are possibly not desirable for this use case.

> is listening on 179. If so you send him an OPEN. *

"check" how?

As noted at the beginning of this thread, you may not be able to get your
SYN+ACK if you don't agree on security mechanisms or GTSM.  So, you need
that state either in your discovery mechanism, or your provisioning.

If you require it in your provisioning, you're making a design choice for
your mechanism.  Other implementors may not want to require explicit
provisioning for those things.

> either. But maybe IDR has no power to recommend additions to LLDP.

I suggest you read the cited appendix A.1.1 for a proposal.

I'll leave it to IETF's IEEE liaisons to comment on what happens if we want
a L2-specific protocol.  IEEE has previously had concerns over where such
work happens, and is one of the potential reasons we keep our proposals to L3.

L3DL will have worked through part of this already.  They also handle
authentication and fragmentation considerations.  See appendix A.1.2

> enable auto peering on my LAN towards customer all I care to accept his

That's certainly one of the valid use cases.

Some providers may be fine with no security.

Other providers may want to limit peering based on some property of the
session.

The mechanism we specify needs to accommodate them.

-- Jeff (in general, read the draft)

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 19 Mar 2021 15:17:19 +0100

> 2. L2 encapsulated unicast to a well known MAC?

That one + broadcast.

Thx,
Robert (who read the draft).

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 19 Mar 2021 10:42:11 -0400

> some additional controls on it.

See Section 2.5.

> Also part of such a template maybe limiting one side to be passive only.

Agreed.

> But again is that really a real risk if we scope it for DC use case ?

See Section 5.2.

Minimally, the Security ADs will require this stuff to be discussed.

While we can think it foolish to do auto-discovery for "interface all",
it'll probably happen.  Knowing what occurs when it does will be one of
their questions.

Even for DC.

-- Jeff

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 19 Mar 2021 15:22:05 +0100

> that state either in your discovery mechanism, or your provisioning.

You keep bringing GTSM here.

Aren't both DC nodes we are to establish a BGP session between under the
very same administration ? I don't get the need to put this in
auto discovery if we are sticking to the scope.

Even if we talk about Compute to TOR .. for one this is still under same
admin. And it is actually a security feature not to add this to auto
discovery as this info can be passed out of band to what value to set
your TTL such that TOR answers.

Thx,
R.

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 19 Mar 2021 10:46:57 -0400

> auto discovery if we are sticking to the scope.

You've already given an example of a provider you've worked with in this
context that wanted to use TCP-MD5.  The same question could be asked there.

*YOU* may think it's out of scope.  That doesn't mean other providers do.

The proposal must be able to support GTSM, or no GTSM.  If, for some reason,
you are thinking the draft says you MUST use GTSM and you MUST use BFD, that
is not the intent of the text.

-- Jeff

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 19 Mar 2021 15:30:12 +0100

The MD5 was in context to auto PE-CE eBGP peering on the RFC4364 VRFs. Not
DC under the same operations.

> The proposal must be able to support GTSM, or no GTSM.

Respectfully I have a different opinion.  That should be part of
provisioning the auto peer template. Nothing to do with auto discovery.

Thx,
R.

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 19 Mar 2021 11:28:32 -0400

> DC under the same operations.

The motivation for the broader analysis of auto configuration was to make
sure we don't have to completely reinvent this stuff a second round. :-)

> provisioning the auto peer template. Nothing to do with auto discovery.

Having it in the discovery protocol doesn't impact that if your
implementation doesn't want to use it.  It simply becomes another piece of
conflicting configuration if it doesn't.

If your configuration template doesn't have security configured, but it is
required by the auto-discovery advertiser, your implementation would try to
open a bgp session and that would fail.  Your debugging would show that you
received a discovery message, but that tcp fails to connect.  The same would
be true for GTSM.  For BFD, the BGP session may come up, and then bounce, or
not proceed into Established.

If the parameters are in the discovery message, you don't end up with such
mismatches unless you want them to be forced to a particular setting.  And
even if you have preferences about how the session comes up (e.g. require no
authentication for NSR considerations), you still have information in the
discovery that permits you to find out why the session may not be
establishing.

-- Jeff

---

From: "Fomin, Sergey (Nokia - US/Mountain View)" <[email-masked]>
Date: Tue, 23 Mar 2021 04:28:17 +0000

Hi Jeff,

> The motivation for the broader analysis of auto configuration was to make sure we don't have to completely reinvent this stuff a second round. :-)

Maybe we should revisit the scope definition then?

I have to agree with Robert on this one.

Either we are trying to solve a DC underlay discovery use case, or something else.

> Having it in the discovery protocol doesn't impact that if your implementation doesn't want to use it.

It may bring unnecessary complexity (depending on protocol design). Especially given the “MUST” requirement.

"The auto-discovery mechanism is designed to be simple.", says section 2.2 of the draft..

> If your configuration template doesn't have security configured, but it is required by the auto-discovery advertiser, your implementation would try to open a bgp session and that would fail.

And that might be just fine? Again, if we talk about a DC under a single administrative entity; and expect that other ZTP/configuration provisioning mechanisms are already in-place.

--

Sergey

-----Original Message-----
From: Idr <[email-masked]> On Behalf Of Jeffrey Haas
Sent: Friday, March 19, 2021 8:29 AM
To: Robert Raszuk <[email-masked]>
Cc: [email-masked]; Acee Lindem (acee) <acee=[email-masked]>
Subject: Re: [Idr] BGP Auto-Discovery Protocol State Requirements

On Fri, Mar 19, 2021 at 03:30:12PM +0100, Robert Raszuk wrote:

> The MD5 was in context to auto PE-CE eBGP peering on the RFC4364 VRFs.

> Not DC under the same operations.

The motivation for the broader analysis of auto configuration was to make sure we don't have to completely reinvent this stuff a second round. :-)

> provisioning the auto peer template. Nothing to do with auto discovery.

Having it in the discovery protocol doesn't impact that if your implementation doesn't want to use it.  It simply becomes another piece of conflicting configuration if it doesn't.

If your configuration template doesn't have security configured, but it is required by the auto-discovery advertiser, your implementation would try to open a bgp session and that would fail.  Your debugging would show that you received a discovery message, but that tcp fails to connect.  The same would be true for GTSM.  For BFD, the BGP session may come up, and then bounce, or not proceed into Established.

If the parameters are in the discovery message, you don't end up with such mismatches unless you want them to be forced to a particular setting.  And even if you have preferences about how the session comes up (e.g. require no authentication for NSR considerations), you still have information in the discovery that permits you to find out why the session may not be establishing.

-- Jeff

Idr mailing list

[email-masked]<mailto:[email-masked]>

---

From: Jeffrey Haas <[email-masked]>
Date: Tue, 23 Mar 2021 08:05:15 -0400

Sergey,

>From the transport requirements for BGP sessions, what would you subtract
that you think isn't DC specific?

> "The auto-discovery mechanism is designed to be simple.", says section 2.2 of the draft..

The writeup for this thread shows a split: Transport requirements to even
get BGP to come up.  Session parameters either in the discovery protocol, or
discovered when you bring up your BGP FSM and you receive an OPEN.

As noted in the thread, the important property a protocol must design for is
how you handle retries.

> And that might be just fine? Again, if we talk about a DC under a single administrative entity; and expect that other ZTP/configuration provisioning mechanisms are already in-place.

This is a valid opinion.

End of the day, the requirements are motivated by how do you make auto
configuration able to be done.  If the working group decides that certain
types of misconfigurations (and thus less "auto") are acceptable, so be it.
But it won't be done blindly because the issue has not been considered.  You
simply get to justify it to your customers.

-- Jeff

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 23 Mar 2021 14:23:32 +0100

Hi Jeff,

> As noted in the thread, the important property a protocol must design for
is

As this point is being brought back again let's zoom on it a bit here.
Clearly the above indicates changes of "handling retries" to current
RFC4271.

So I assume you possess some operational evidence which would lead to
suggest that if peer IP address is learned via auto discovery we need to
apply a different BGP OPEN retry procedure or at least timers then if the
same peer's IP address is pushed by cfg from mgmt station. Can you share
this data ?

If not could you please describe how auto discovered peers would need to
have different BGP FSM then say those configured with mistakes. To me this
difference is not obvious.

Now let's see if the same concerns or observations apply if you limit auto
discovery to *only* same L2 domain.

Last the draft says:

"The auto-discovery mechanism will not replace or conflict with data
exchanged by the BGP FSM, including its OPEN message."

But if you are thinking about modifications of handling BGP session retries
then I guess the above needs to have one exception added.

Then we will see different OPEN behaviour depending if peer got auto
discovered vs provisioned ...

Many thx,
R.

On Tue, Mar 23, 2021 at 12:43 PM Jeffrey Haas <[email-masked]> wrote:

> > > The motivation for the broader analysis of auto configuration was to
> make sure we don't have to completely reinvent this stuff a second round.
> :-)
> >
> implementation doesn't want to use it.
> >
> > > If your configuration template doesn't have security configured, but
> it is required by the auto-discovery advertiser, your implementation would
> try to open a bgp session and that would fail.
> >
> > And that might be just fine? Again, if we talk about a DC under a single
> administrative entity; and expect that other ZTP/configuration provisioning
> mechanisms are already in-place.
>
> This is a valid opinion.
>

---

From: Jeffrey Haas <[email-masked]>
Date: Tue, 23 Mar 2021 11:08:37 -0400

Robert,

On Tue, Mar 23, 2021 at 02:23:32PM +0100, Robert Raszuk wrote:
> > As noted in the thread, the important property a protocol must design for
> > is how you handle retries.
> 

Thanks for this framing, it's exactly the point the WG needs to discuss for
retries.

If a router configures a hundred BGP sessions, it has a hundred that it
expects to deal with.  This includes sessions that fail to connect.  But the
general expectation when you have sessions configured is that you expect
them to connect.

With a discovery protocol, you have at best a rough bound of the number of
interfaces that it is running on.  It may be more.  If the peers aren't
found to be acceptable, they'll continue on indefinitely.

Is that acceptable?  Your argument is roughly that "you asked for it, deal
with the consequences".  

At low scales, it's probably quite fine.  

At high scales, with a central listen socket for incoming sessions, you may
end up exhausting the "backlog" parameter of the socket and result in a
denial of service of legitimate sessions.

> difference is not obvious.

The distinction is when you instantiate a session for a discovered peer and
when you keep it provisioned vs. quiescieng or deleting it.  Remember,
discovery may be a denial of service on the discovering party as well.

If a discovered peering session is unacceptable, why would you keep running
the BGP state machine over and over if it's not going to improve?

The options we have upon discovering a peer that isn't acceptable are:
1. Quiesce the disovered session and require a manual clearing event.
2. Keep the instantiated session running even if it never connects.
   Analogous to explicit configuration of a misconfigured session.
3. The discovery protocol itself tells us that the situation has changed.
   This gives the implementation the option to quiesce without requiring a
   manual clearing event.

> discovery to *only* same L2 domain.

It's a matter of scale, not scope.

> discovered vs provisioned ...

The distinction isn't about what to do about a BGP session that has been
created by the discovery mechanism.  It's the fact that a session is created
by the discovery mechanism and whether it's left running in a broken mode or
not.

-- Jeff

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 23 Mar 2021 16:18:47 +0100

Hi Jeff,

While actually I tend to agree with some of your comments I think they are
only valid if we are really talking about "high scales".

"High" in the context of BGP.  (Which to me means at least 1000s)

So tell me (and all of us here) where in the DC fabric you see this high
scale especially if we are only talking here about underlay (as draft
says).

Moreover as BGP is trying to do the IGP job here it may be perhaps useful
to tak a look on how IGPs deal with establishing adjacencies and if our
worries BGP may fall into would be something one needs to also consider
when planning to use link state in the DC fabric.

And that is why IMHO unless we clearly state the scope of this work (for
example to limit it to peers connected on the same L2 link - same L3 subnet
- no over the top transport, directly connected ebgp, ebgp between
loopbacks, no ibgp etc ... ) then I am afraid we will keep a bit of
ping-pong here as each person's view may be in fact correct in the specific
deployment scenario one has in mind for the new functionality under
discussion. Except that the deployment scenarios are very different.

Many thx,
Robert.

---

From: Jeffrey Haas <[email-masked]>
Date: Tue, 23 Mar 2021 12:02:51 -0400

Robert,

> scale especially if we are only talking here about underlay (as draft

Thanks for confirming that you agree this may be an issue for higher scale.

The scope from the draft remains the same:

: 2.1.  Problem Scope
: 
:    The current target environment is BGP as used for the underlay
:    routing protocol in data center networks.  Other scenarios may be
:    considered as part of the analysis for this work, but work on those
:    environments will be deferred to other efforts.

> discussion. Except that the deployment scenarios are very different.

The position I have been taking is figuring out what is required for BGP
auto-configuration.  Data center is the specific scenario we are solving for
right now.  The working group may take on non-data center cases as future
work.  The analysis discussion is being used to refine what state is
required for each of the scenarios.

-- Jeff

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 23 Mar 2021 16:58:08 +0100

> required for each of the scenarios.

And I think this is where the confusion is coming from.

The document is about DC but it includes lots of additional baggage which
in view of number of people is not required for the DC auto discovery.

I guess in the best intentions you added all additional data there thinking
ahead and planning to use the same mechanism for any BGP auto discovery
scenario which was precisely agreed not to be a goal of this document to
start with.

So my little suggestion is to scope this to DC. If we end up with two
different set of requirements - one for DC and one for WAN/IXP that would
be IMO fine.

Many thx,
R.

---

From: Jeffrey Haas <[email-masked]>
Date: Tue, 23 Mar 2021 12:44:05 -0400

> in view of number of people is not required for the DC auto discovery.

You, and the rest of the working group, are explicitly solicited to provide
your feedback on what *is* required for BGP auto-discovery for a DC context.[1]

The draft provides a framework for how the requirements are analyzed in
terms of:
- What BGP needs for a session to come up.
- How a discovering peer can choose whether it wants to peer with the
  discovered thing.
- What the transport requirements are for the discovery protocol.
- What the security considerations are for your desired properties.

Please provide your requirements with those things addressed.

-- Jeff

[1] As a member of the design team, you were requested to do this prior to
publication.  You didn't.

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 23 Mar 2021 17:35:22 +0100

> [1] As a member of the design team, you were requested to do this prior to

Members of the design team were not given a freedom to carry an eraser :(

---

From: heasley <[email-masked]>
Date: Tue, 23 Mar 2021 18:13:16 +0000

>    This gives the implementation the option to quiesce without requiring a

maybe 4. a RFD-like auto-discovered peer connection suppression?

expect each will have propoents.

I do not run a DC, but appreciate your considering the scaling.

---

From: Jeffrey Haas <[email-masked]>
Date: Tue, 23 Mar 2021 15:40:22 -0400

> maybe 4. a RFD-like auto-discovered peer connection suppression?

The RFC 4271 machinery technically already has this sort of thing.  The two
interacting RFC things are the ConnectRetryTimer and the
DampPeerOscillations feature.

What is partially problematic on the "rely on the existing BGP FSM" is many
people aren't thinking through those timer considerations:

- The recommended value for ConnectRetryTimer is 120s.
- Oscillation damping is intentionally under-specified.  And it's totally
  optional.
- Many implementations (including the one I code for), do not stick to 120s
  values as an initial value.

Operators usually want their sessions to come up promptly.  In a data center
fabric context, how long after discovery are you willing to wait before it
comes up?

What this likely will mean for people's implementations is:
- The initial ManualStart event for the session (since this is coming from
  outside of BGP timer infra) will be shortly after discovery.
- The ConnectRetryTimer is likely to be short.  Oscillations damping is also
  likely to be short, even if it's on a decay cycle that takes it to longer
  values after a small number of rounds.

What this can mean is that the BGP Speaker sending out discovery messages
can expect to be hammered very hard with connection requests.  The listen
socket for a central dispatched listen mechanism will likely overload, which
will cause TCP timers to either go into backpressure modes, or aggressive
implementations that will simply close the socket and trigger the connecting
BGP FSM to back off itself.

Are people going to be happy when their fabric connection takes a few
minutes to Establish?  Probably not.  They'll tweak the timers.

And that's on top of the timers for the auto-discovery mechanism itself and
its reliability mechanisms.  See Section 3.2 in the auto-discovery
considerations draft.

> expect each will have propoents.

It's engineering.  After you've dealt with correctness, you have choices and
you have consequences.

-- Jeff