Subject: draft-ietf-l3vpn-mvpn-considerations-05

From: "NAPIERALA, MARIA H (ATTLABS)" <[email-masked]>
Date: Fri, 11 Dec 2009 12:08:06 -0500

Hi Thomas,

I have several comments on the latest version of the draft:

1. I believe that the draft should contain some analysis on how PIM-SM
differs when handled by BGP signaling.  The difference is significant
enough to warrant at least a theoretical consideration if no significant
field experience is available.  Currently, vast majority of multicast
applications used in the enterprise networks are PIM-SM based. It is
important to analyze the BGP translation of pruning source traffic off
the shared tree because this is the most significant change from PIM-SM.

When BGP is used for PE-PE C-multicast signaling the data-driven PIM-SM
state changes are replaced by additional BGP (Source Active) routes and
timer-driven PIM messages. Let me point out just few aspects related to
those changes that should be considered, starting from timer-driven
Prune(S,G,rpt).

When a single forwarder selection is the procedure used to avoid sending
duplicate packets to MVPN customers during switching from RPT to SPT,
the correct setting of the above mentioned timer to prune source traffic
off the shared tree is important. It is not going to be trivial to tune
such timer in large networks, because it can cause either duplicate
traffic (if timer is set too high) or loss of traffic (if timer is set
too low). This approach to PIM-SM changes the way the PIM-SM multicast
trees are constructed, and it is not known to what extend this can
impact existing multicast applications.
If the solution to this is to make the timer for sending Prune(S,G,rpt)
sufficiently high (to assure that SPT is created) and avoid duplicates
sent to customers by having egress PEs discard data packets received
from the "wrong" upstream PE, this should be included in MVPN
considerations draft. It should be also considered that it may not
always be possible for the egress PE to determine the upstream PE, in
which case the single forwarder selection has to be used. 

Regarding Source Active A-D routes, it was pointed out recently and in
the past that the amount of BGP state they generate should be analyzed.
The Appendix A ignores this important scalability aspect of
BGP-signaling. 
The field experience shows that there are multicast applications used in
enterprise VPNs such that only control messages, but no multicast data,
are sent across MVPN backbone. When BGP-multicast signaling is used, the
Source Active A-D route is originated for every C-(S,G) flow that
registers with C-RP and such that the C-RP joins "S" across MVPN
backbone, even if no inter-site data is sent for this flow. Hence, the
BGP scheme requires not only that for any given C-(S,G) flow that enters
MVPN backbone an SA message is announced and maintained by all PEs in a
given MVPN, but that SA messages are also announced for flows that never
enter the MVPN backbone.

2. The considerations draft should also point out the difference how BGP
vs. PIM (used as PE-PE signaling protocol) handles the upstream PE
selection in case of multi-homed sources/RPs. 

Typically, in a layer 3 VPN deployment it is not known whether or not
all VRFs of a given VPN have the same best route to a given source.  In
other words, VRFs routing policies are provisioned independently of each
other. If all VRFs of a given MVPN have the same best/ installed route
to a multi-homed source/RP then this VPN customer *expects* that the
source/RP will be joined via the installed route. 

When PIM is used as PE-PE C-signaling, the installed route towards the
source is always chosen as UHM. If different VRFs have different best
routes to C-S or C-RP, the Assert process is a tie breaker. No steady
state duplicate traffic is sent across MVPN backbone.

Now let's consider BGP as the PE-PE signaling protocol and let's assume
the UMH policy is to choose the installed/best route towards C-source or
C-RP. If different VRFs have different installed routes to C-S/C-RP then
there will be a steady state duplication of traffic across the MVPN
backbone (to the same set of PEs if MI-PMSI is used). This also requires
that the egress PEs be able to discard data traffic that arrives from
the "wrong" upstream PE. Such duplication of traffic might not be
operationally acceptable to a service provider because of too much
network bandwidth being consumed. The other UMH policy that can be used
with BGP signaling where duplicates are not generated, is a single or
default forwarder selection. However, single forwarder selection might
not choose the installed/best route (to a multi-homed C-S/C-RP) *even*
if all VRFs have the same best route to C-S/C-RP. Hence, in a MVPN where
all VRFs have the same best path to C-S or C-RP, its multicast traffic
might not flow via the best path as is expected by the VPN customer.

3. In section A.1.1.3, why the "each additional PE joining" row for the
"upstream PE" column shows "0"? The same question for "upstream PEs"
column and the "each PE leaving" row.

4. Why Appendix A assumes that the RT-constraint is deployed when BGP
signaling is used?  This is not a mandatory implementation.

5. Why Appendix A analyzes PIM LAN procedure with explicit tracking but
not the explicit tracking using BGP-signaling? 

6. The appendix states that "we propose here a rough estimation of the
load of PEs". And it is in fact a very *rough* estimation and not even a
complete *rough* estimation as it almost dismisses ASM analysis. The
appendix treats different and, in fact, not comparable kinds of
"processing" as *equal* and uses the same unit "m.e" to measure them. 
For example, the first row in the table in section A.1.1.1, says that to
process the first join on the upstream PE takes "1 m.e" and that the
same amount of processing, "1 m.e", is required on every other PE in
MVPN. Clearly, the amount of processing on the upstream PE to which the
joined is destined is not the same as the "processing" on other PEs. Any
other PE than the upstream PE will simply discard this Join because the
neighbor address in the PIM Join is not this PE address. While the
upstream PE has to create the (s,g) state in the control plane, program
the data plane for forwarding, and send Join (s,g) message upstream.
This is by any measure much more processing than discarding the Join
based on the address. Yet, it is assumed that the resources used in both
cases are the same. 
Another example is the row "each PE leaving" (but last) in the same
table in section A.1.1.1. The only processing the upstream PE does is
setting a timer for receiving prune-override PIM Join message, and when
such PIM join is received it cancels the timer. This is estimated as
"2 m.e", yet, this is much less "work" to be done by the upstream PE
than in the previous "first PE joins" example. In addition, the column
"other PEs" in the row "each PE leaving" states that every other PE in
MVPN will also have to do an action worth "2 m.e". Yet, only the PEs
that have interested receivers in (s,g) have to perform a task (i.e.,
randomly generate a join message delay timer and cancel it when they
hear a join from another PE; only one of those PEs will send a PIM Join
message). The PEs that do need to receive (s,g) traffic will simply drop
the (s,g) prune message. Hence, the application of the multiplier
#MVPN_PE in this row/column is also not correct.
This "problem" of treating not equal actions as equal repeats itself
throughout the scaling analysis in the appendix. Such analysis is not
only *rough* but misleading, and because of this may lead to wrong
conclusions. I believe that unless the appendix A is corrected by
removing all not comparable, or even wrong estimations, it should be
removed from the draft. 

Maria

---

From: Thomas Morin <[email-masked]>
Date: Wed, 16 Dec 2009 18:10:48 +0100

Maria, working group,

Find below some replies to some of your comments.

But first of all : correct me if I'm wrong, but it does not seem to me 
that any of them is specific to changes made to the document since the 
WG Last Call in June.... ??

> the shared tree because this is the most significant change from PIM-SM.

"significant enough to warrant" ... but not worth making the suggestion 
six month after the LC ?  ;-)

> considerations draft.

I personnally agree with you it would be reasonable that the draft 
recommends implementing these procedures ( 
http://tools.ietf.org/html/draft-ietf-l3vpn-2547bis-mcast-09#section-9.1.1 )

Let's propose this to the working group (keeping in mind the delay after 
WG LC...).

> It should be also considered that it may not

> The Appendix A ignores this important scalability aspect of

( I'll comment on this one in a later email, in reply to Eric Rosen 
comments)

> given MVPN, but that SA messages are also announced for flows that never

What you state above it true only with one way of using the BGP approach 
to support ASM (the one in section 14 of 
draft-ietf-l3vpn-2547bis-mcast-bgp)?

The other way (section 13) does not have this effect, and happens to 
indirectly be the one recommended by 
draft-ietf-l3vpn-mvpn-considerations (because it is the one of the two 
not requiring an RP function co-located in the PE).

So well, I'm not sure that this point would be very important to cover.

> source is always chosen as UHM.

This is not my understanding, to quote 
draft-ietf-l3vpn-2547bis-mcast-09, section-5.1.3:

    The default procedure, which MUST be implemented, is to select the
    route whose corresponding upstream PE address is numerically highest,
    where a 32-bit IP address is treated as a 32 bit unsigned integer.
    Call this the "default upstream PE selection".

    [...]

    Another procedure, which SHOULD be implemented, is to use the
    Installed UMH Route as the Selected UMH Route.

>   If different VRFs have different best
> C-RP.

Given the above, why is this assumption relevant ?

>   If different VRFs have different installed routes to C-S/C-RP then
> network bandwidth being consumed.

I don't think this is a relevant drawback :
- if a provider chooses the non-default UMH selection you mention,  it 
is likely that he is fine with the bandwidth aspect, and has made sure 
the vendors implement the procedures to discard data traffic that 
arrives from the "wrong" upstream PE .
- see above for the suggestion to recommend implementing the procedures 
required to discard data traffic that arrives from the "wrong" upstream 
PE (making it easier for the provider to make sure that it is ok for him 
to use the non-default UMH selection you mention ).

> The other UMH policy that can be used

See above.

> column and the "each PE leaving" row.

See reply to Eric Rosen "draft-ietf-l3vpn-mvpn-considerations-05 / 
Appendix A / section A.1.1.3".

> signaling is used?  This is not a mandatory implementation.

See reply to Eric Rosen "draft-ietf-l3vpn-mvpn-considerations-05 / 
Appendix A / RT Constraint recommendation".

> not the explicit tracking using BGP-signaling?

I'm not sure as what you call "explicit tracking using BGP-signaling".

There is, AFAIK, no such thing as "explicit tracking" in the BGP 
procedures for C-multicast routing, and Appendix A is only about 
C-multicast routing options.

( Possibly you refer to the case where BGP auto-discovery procedures are 
used to allow a PE to know about which PEs would be leaves of S-PMSI 
tunnels ?)

> complete *rough* estimation as it almost dismisses ASM analysis.

On "almost dismisses ASM analysis", see reply to Eric Rosen 
"draft-ietf-l3vpn-mvpn-considerations-05 / Appendix A / ASM".

> The appendix treats different and, in fact, not comparable kinds of

On "non comparable" I disagree; moreover this is not something having 
been brought up by anyone until now, AFAICT.

> joined is destined is not the same as the "processing" on other PEs.
>    
>   Any other PE than the upstream PE will simply discard this Join because the
> neighbor address in the PIM Join is not this PE address.

See correction by Yakov, which you acknowledged, and let me point out 
that, if you write "simply discard this Join because the neighbor 
address in the PIM Join is not this PE address" while you mean "parse + 
lookup in TIB + discard", then I disagree with you because that "parse + 
lookup in TIB" does not seem to me as insignificant.

>   While the
> based on the address.

(see above, PE that are not the upstream have more to do than merely 
discarding the Join based on the upstream address ).

That said it is true that among the upstream PE for the first join, the 
upstream PE for later joins or  the PEs that have (S,G) state, and the 
other PEs that have no (S,G) state, not all of them have the exact 
amount of processing to do.

Where I'm not sure is about, when some equipment has more to do than 
another one, "how much more" it is ; it seems to me that it is in fact 
likely to be very much vendor specific.

>   Yet, it is assumed that the resources used in both
> cases are the same.
>    

Well the idea is to accept this approximation, and document it.

The introduction of Appendix A already provides such an explanation, and 
your comment would lead to think of improving it.  I see that this is 
what Yakov Rekhter as suggested, and I think that this is a good idea.

> "2 m.e"

Not exactly, the "2 m.e" includes the processing of the Prune *and* the 
processing of the Join send by one PE to override the Prune.

> , yet, this is much less "work" to be done by the upstream PE
> than in the previous "first PE joins" example.

(see above with reference to how we can reasonably take into account 
"how much less" work it is)

>   In addition, the column
> the (s,g) prune message.

This is not exact.
(see the correction brought up by Yakov, for the details)

>   Hence, the application of the multiplier

(it is correct: see the reply to Eric Rosen 
"draft-ietf-l3vpn-mvpn-considerations-05 / Appendix A / PIM LAN procedures")

> removing all not comparable, or even wrong estimations, it should be

...but you only find this suggestion worth enough bring brought forward 
only six month or so after the WG LC... ;-)

I think that the proposed modification to the introduction of Appendix A 
made by Yakov is much more reasonable (copy-pasted below for easier 
follow up) :

    Though the overhead of message processing and the state overhead
    may vary between different approaches, and even within the same
    approach between different equipments (PEs or RRs), we propose
    here a rough estimation of the load of PEs, in terms of number
    of messages processed and number of control plane states maintained.

-Thomas

---

From: "NAPIERALA, MARIA H (ATTLABS)" <[email-masked]>
Date: Wed, 16 Dec 2009 13:57:59 -0500

Thomas,

> > the shared tree because this is the most significant change from
PIM-
> SM.

You must recall that the request to analyze the BGP translation of
pruning source traffic off the shared tree in MVPN-consideration draft
was made several times by Eric Rosen (as well as by me) in the past. I
don't think that this significant change from PIM-SM procedures can be
ignored.

> > When BGP is used for PE-PE C-multicast signaling the data-driven
PIM-
> SM

Good.

> ( I'll comment on this one in a later email, in reply to Eric Rosen

OK

> > The field experience shows that there are multicast applications
used
> in
> > MVPN backbone an SA message is announced and maintained by all PEs
in
> a

> not requiring an RP function co-located in the PE).

No, what I stated above applies to section 13 (I did not even considered
the implications of section 14..). The inter-site shared trees are used.
When a PE receives C-multicast Join route (which triggers Source Active
A-D route) it does not necessarily mean that data was actually sent
across MVPN backbone. We see a lot of applications behaving that way in
the field.
 
> > the UMH policy is to choose the installed/best route towards
C-source
> or
> arrives from the "wrong" upstream PE.

Why not to state it? This does not come for free and requires "strong"
RPF check i.e., checking the upstream PE. Are you aware of any such
implementations? 

> > might not flow via the best path as is expected by the VPN customer.

Same as above for me also.
 
> Appendix A / section A.1.1.3".

OK. I will read and reply there.
 
> Appendix A / RT Constraint recommendation".

I will read it.

> C-multicast routing options.

Well, in that case, PIM LAN procedure with explicit tracking should be
removed also. Why would you do explicit tracking of receivers when using
PIM?
 
> "draft-ietf-l3vpn-mvpn-considerations-05 / Appendix A / ASM".

OK. I will read and reply there.
 
> > For example, the first row in the table in section A.1.1.1, says
that
> to

First of all, this is still not the same amount of work that the
upstream PE has to perform, i.e., to create the (S,G) state, propagate
it upstream, and set up the forwarding state.

Second, this is still just *discarding* the massage based on two things:
"I am not the upstream router" and "I don't have (S,G) state". This is
all the work the "other PE" has to do in this case. You cannot claim
that this is equal with what the upstream PE has to do, right?

Third, regarding the workload on the upstream PE when BGP signaling is
used, how is BGP processing accounted for? All the work that upstream PE
needs to do in PIM signaling case has also to be done in BGP signaling
case. But with BGP signaling, upstream PE has also to perform the BGP
processing (and *parsing*), including the best path selection. I don't
see this being accounted for at all on PEs.
If you worry so much about the work a PE has to do to *discard* a
message, why would you ignore a significant amount of BGP related work a
PE?
 
> discarding the Join based on the upstream address ).

See my comment above also.

> likely to be very much vendor specific.

That is why I believe that attempting such an analysis as in Appendix A
is not a right way to go. And this is why I recommend removing it from
the draft. 

> Well the idea is to accept this approximation, and document it.

This is precisely the problem. Such approximations are quite arbitrary
and the danger is that an analysis made of such approximations might
lead to wrong conclusions. 

> what Yakov Rekhter as suggested, and I think that this is a good idea.

But improving the *description* of the issue will not solve the issue
itself.

> (see the correction brought up by Yakov, for the details)

See my reply above.

> only six month or so after the WG LC... ;-)

The analysis kept changing, so I waited until it is finalized (at least
by the author). I believe that it is more important to make the draft as
accurate as possible than worry about the timing. I don't think anybody
would want that important scaling conclusions in an IETF document be
based on *rough approximations*.

>     of messages processed and number of control plane states
See my comment above on this.

---

From: Thomas Morin <[email-masked]>
Date: Thu, 17 Dec 2009 14:24:46 +0100

Maria,

NAPIERALA, MARIA H (ATTLABS):
>> ...but you only find this suggestion worth enough being brought forward
>>      
> accurate as possible than worry about the timing.

This is not the usual understanding of "Last-Call", to say the least !
I've got no more comment on this except suggest reading the BCPs 
describing the IETF process...

( Oh, in fact, let me also add for the sake of precision that it is just 
totally false to state that
"the analysis kept changing", since it had mostly only editorial changes 
between draft-morin-l3vpn-mvpn-considerations-03, in July 2008, and the 
time the WG LC was made. The particular part you care about here has 
been _strictly_ unchanged in the draft since July 2008. )

-Thomas

---

From: Yakov Rekhter <[email-masked]>
Date: Thu, 17 Dec 2009 05:52:37 -0800

Maria,

Maria> The field experience shows that there are multicast applications used in
Maria> enterprise VPNs such that only control messages, but no multicast data,
Maria> are sent across MVPN backbone. When BGP-multicast signaling is used, the
Maria> Source Active A-D route is originated for every C-(S,G) flow that
Maria> registers with C-RP and such that the C-RP joins "S" across MVPN
Maria> backbone, even if no inter-site data is sent for this flow. Hence, the
Maria> BGP scheme requires not only that for any given C-(S,G) flow that enters
Maria> MVPN backbone an SA message is announced and maintained by all PEs in a
Maria> given MVPN, but that SA messages are also announced for flows that never
Maria> enter the MVPN backbone.

By definition the MVPN service is a Layer 3 service, and as such it
does not distinguish whether customer's IP multicast packets carry
some application specific control information, or whether they are
the data traffic of a particular application. Thus from the MVPN
service point of view there is no different between a customer
multicast flow that carries control traffic of a particular application
and a customer multicast flow that carries data traffic of that
application. 

Given the above, if C-RP joins "S" across the MVPN backbone for a
particular (C-S, C-G), that means that *contrary to your claim above*,
the (C-S, C-G) traffic/flow is going to flow from "S" towards C-RP,
and it is going to flow *across the MVPN backbone*. And as I said
above, from the MVPN service point of view it is just a multicast
flow, irrespective of whether this flow carries some application
control information or a data of this application.

Thomas> What you state above it true only with one way of using the BGP 
Thomas> approach to support ASM (the one in section 14 of
Thomas> draft-ietf-l3vpn-2547bis-mcast-bgp)?
Thomas>
Thomas> The other way (section 13) does not have this effect, and happens to
Thomas> indirectly be the one recommended by
Thomas> draft-ietf-l3vpn-mvpn-considerations (because it is the one of the two
Thomas> not requiring an RP function co-located in the PE).
  
Maria> No, what I stated above applies to section 13 (I did not
Maria> even considered the implications of section 14..). The
Maria> inter-site shared trees are used.  When a PE receives
Maria> C-multicast Join route (which triggers Source Active A-D
Maria> route) it does not necessarily mean that data was actually
Maria> sent across MVPN backbone. 

Incorrect.

This is because for a PE to receive a Source Tree Join C-multicast
route for (C-S, C-G), it would require the PE that originates this
route to first receive PIM Join (C-S, C-G) from one of its CEs, and
that, in turn, would first require (C-S, C-G) to be sent across the
MVPN backbone. Therefore, this (C-S, C-G) has to be sent across the
MVPN backbone prior to the PE receiving the Source Tree Join
C-multicast route.

Yakov.

---

From: Thomas Morin <[email-masked]>
Date: Thu, 17 Dec 2009 18:56:37 +0100

Maria, working group,

Maria:
> >  >  If the solution to this is to make the timer for sending
>    
> >  >  Prune(S,G,rpt)  sufficiently high (to assure that SPT is created) and avoid
>    
> >  >  duplicates  sent to customers by having egress PEs discard data packets received
> Good

Ok.

We can propose this to the working group, supposing noone disagrees 
despite this coming far after WG LC.

> Why not to state it?

At this point in the process the question is "why state it ?" rather 
than "why not" (supposing we have a question to ask)

> This does not come for free and requires "strong" RPF check i.e., checking the upstream PE. Are you aware of any such implementations?

(ask your vendors)

>> PE (making it easier for the provider to make sure that it is ok for
>> him to use the non-default UMH selection you mention ).
>>      

Isn't it sufficient ?

> removed also. Why would you do explicit tracking of receivers when using

Well, see my reply to Eric Rosen 
"draft-ietf-l3vpn-mvpn-considerations-05 / Appendix A / PIM with 
explicit tracking":
- there is perfect rationale to cover it (part of RFC4601 => part of the 
PIM-based C-multicast routing options)
- at the same time, if there is consensus that it is not useful I don't 
mind removing it

-Thomas

---

From: "NAPIERALA, MARIA H (ATTLABS)" <[email-masked]>
Date: Thu, 17 Dec 2009 14:59:02 -0500

Yakov,

> and it is going to flow *across the MVPN backbone*. And as I said

Actually, the multicast traffic does not necessarily flow across the
MVPN backbone. Customers use techniques (like multicast ttl-threshold)
to keep multicast data traffic local to a site. We see this happening
frequently. 

> control information or a data of this application. 

This is not what I meant, i.e., that the flow itself carries some
"control" information.
I meant that the traffic does no flow at all, as I explained above.
 
Maria