Subject: Re: [PWE3] Why we consider the method of "label sharing for fast PE protection"

From: Mingui Zhang <[email-masked]>
Date: Thu, 7 Nov 2013 22:15:10 +0000

Hi Jakob,

There are three ways,

1. Always, this label can be configured.

2. We do have a companion draft to realize this. Please look at http://tools.ietf.org/html/draft-zhang-pwe3-iccp-label-sharing-00 . We already presented this draft in pwe3. (Let me copy this track to the pwe3 mailing list.) With the ICCP connection established, PEs can negotiate this label.

2. The last choice is to use a global label if it was supported.

Thanks,
Mingui
________________________________
From: Jakob Heitz [[email-masked]]
Sent: Friday, November 08, 2013 5:50
To: Mingui Zhang; [email-masked]
Subject: RE: Why we consider the method of "label sharing for fast PE protection"

Several people at the mike asked this question:
How do you make sure that the PEs allocate the same label?

This needs to be part of the document, because it is quite important.
If an external entity allocates the labels, the protocol
between the PEs and that entity needs to be standardized.
Since this is a feature that provides redundancy, the
label allocating entity also needs to be backed up by a
redundant entity. The protocol between the redundant
label allocators needs to be standardized.

--

Jakob Heitz.

________________________________
From: [email-masked] [[email-masked]] on behalf of Mingui Zhang [[email-masked]]
Sent: Thursday, 07 November 2013 11:40 AM
To: [email-masked]
Subject: Why we consider the method of "label sharing for fast PE protection"

Hi,

As a choice of fast PE protection,

1. This solution is simple and light-weight. We need not introduce the complex context label table in PE routers. So label table need not be stored repeatedly on RG members.

2. Also, it’s easy to be deployed. It does not bring any change to P routers (control plane & data plane). It even does not change the data plane of PE routers.

3. In addition, it does not bear the restriction of “no penultimate-hop-popping”.

Thanks,
Mingui

---

From: Robert Raszuk <[email-masked]>
Date: Sat, 9 Nov 2013 07:20:27 +0100

Hi Mingui,

If your implementation configures a same VPN label across N sites
manual or by SDN magic you are free to advertise it along with VPNv4
routes. I am not sure why any draft is needed for that at all. After
all it is your implementation.

To the best of my knowledge L3VPN spec does not prohibit in any RFC to
advertise the same label across N PEs. In fact it does not mandate any
correlation of label value across PEs so you are pretty free to go.

However if you go that path please notice that this is not just about
.. "let's configure same VPN label".

We do have three types of VPN labels:

- per vrf
- per next hop
- per prefix

I am a bit implicitly assuming you are talking about per vrf label
only. Other modes of label allocation would be rather cumbersome to be
done manually or even programmatically as their appearance and
allocation must be coordinated with new CE or new VPN advertisement in
any VPN site and across existence of already provisioned sites.

As Eric mentioned today devil is in details and where you go down to
actually implement robust solution I can bet that you will start very
much to like to concept of context labels for protection and
independent VPN (or for that matter any other application - pwe3/l2vpn
etc) label allocation pardigm :)

Best regards,
R.

On Thu, Nov 7, 2013 at 11:15 PM, Mingui Zhang <[email-masked]> wrote:
> 2. We do have a companion draft to realize this. Please look at
> http://tools.ietf.org/html/draft-zhang-pwe3-iccp-label-sharing-00 . We
> already presented this draft in pwe3. (Let me copy this track to the pwe3
> mailing list.) With the ICCP connection established, PEs can negotiate this
> label.
>
>
>
> 1. This solution is simple and light-weight. We need not introduce the
> complex context label table in PE routers. So label table need not be stored
> repeatedly on RG members.
>
>
> 2. Also, it’s easy to be deployed. It does not bring any change to P routers
> (control plane & data plane). It even does not change the data plane of PE
> routers.
>
>

---

From: Mingui Zhang <[email-masked]>
Date: Sat, 9 Nov 2013 08:32:57 +0000

Hi Robert,

Thanks for your comments!

>all it is your implementation.

[ZMG] The point is that each PE used to allocate the VPN label for each VRF locally without knowing what label has been assigned by other PEs. Now, we need them “happen to” allocate the same label for each VRF. 
>correlation of label value across PEs so you are pretty free to go.

[ZMG] Or you can speak in this way. It’s free for this draft to mandate a REQ to the label allocation that PEs in a Redundant Group always allocate the same label for each VRF.

<snip>
>only. 

[ZMG] Yes, you are right. We are assuming the “per VRF” allocation in our draft for the sake of scalability. 

>Other modes of label allocation would be rather cumbersome 

[ZMG] I agree with you. Other two finer grained label allocations are not recommended. 

<snip>
>etc) label allocation pardigm :)

[ZMG] I believe when Eric made that statement, he assumed the label allocation was done by a “SDN central controller”. I guess that was simply an abstraction caused by the chaos of the “central controlled” conception. 

[ZMG] To make the discussion clear, let’s forget about the “central controlled” like stuff from now on. We proposed the draft leveraging ICCP connection to do the label negotiation. Please focus on that choice.  ICCP has already provided that robustness. Why reinvent the wheel? :)

Regards,
Mingui

________________________________________
From: [email-masked] [[email-masked]] on behalf of Robert Raszuk [[email-masked]]
Sent: Saturday, November 09, 2013 14:20
To: Mingui Zhang
Cc: Jakob Heitz; [email-masked]; [email-masked]
Subject: Re: Why we consider the method of "label sharing for fast PE protection"

Hi Mingui,

- per vrf
- per next hop
- per prefix

Best regards,
R.

---

From: Mingui Zhang <[email-masked]>
Date: Thu, 14 Nov 2013 04:57:23 +0000

Hi Bruno, 

My comments inline below. [Mingui]

>1)      The document does not describe the scaling impact on the core as it
>creates, in the core IGP, states on a per CE (customers'site) basis. For large SP,
>this will represent a significant impact on the IGP and will degrade the IGP
>convergence time. ( o(number of important IGP prefixes to rewrite) )

[Mingui] Here, I need to point out that the vNH is not created per CE. If multiple CEs are connected to an RG, one vNH is enough. When you read the first sentence of Section 2.1, you will be clear.
[Mingui] "A vNH router is created in IGP to represent the set of CEs which are dual-homed to the same egress PEs in the Service Provider's backbone".
[Mingui] Exactly, it's per RG basis. In a real network, the numbers of RGs is comparable to the number of PEs, so it's similar to the per PE basis choice (the best choice) in draft-minto.

>2)  > A master PE is elected in the same way the DR is selected (see section 7.3
>of [RFC2328] <http://tools.ietf.org/html/rfc2328#section-7.3> ), or the DIS is
>selected [ISIS
><http://tools.ietf.org/html/draft-zhang-l3vpn-label-sharing-01#ref-ISIS> ].
>I don't think that this is directly applicable. You may need to elaborate on this
>and specify the corner cases (e.g. CE starts and advertise its routes to both PE
>at the same time. Hence both PE believes they are the first one and hence are
>the master.) Also in BGP VPN or IGP there is no advertisement of "Router
>Priority". I guess that this can always be specified, but this remains to be done.

[Mingui] That's a valid point. We did define the "Priority" in the "vNH TLV" in Section 2.3.3 of the companion draft (http://tools.ietf.org/html/draft-zhang-pwe3-iccp-label-sharing-00). We can change the words to fix this inconsistence.

<snip>
>IMHO additional elaboration may be required as the nominal PE
>may be the nominal because it has advertised a more specific route. Hence a
>look up in the VRF will match the more specific route from the nominal/failed PE.
>Not sure how a vanilia PE implementation would react. The issue is that when
>receiving the packet, the backup PE does not know whether the nominal is dead
>or not. In contrast to draft-minto for which the PE know whether it's used as
>backup or not, thanks to the contextual label.

[Mingui] So you propose the scenario that the backup PE directs packets to the primary PE since the primary PE advertised a more specific route. This makes the primary PE be an inevitable point on all forwarding paths to the CE. Eric ever used "BGP LOCAL_PREF" to compose a similar scenario (http://www.ietf.org/mail-archive/web/l3vpn/current/msg03691.html).

[Mingui] For this subtle case, the redundant connection is not utilized in the first place. If customers prefer resilience to more specific routes, the backup PE should not use the primary PE as the next hop. 

[Mingui] All in all, if customers really want the resilience, those routes via the primary PE SHOULD NOT be installed in the FIBs of backup PEs. 

[Mingui] By the way, the scenario you designed may suffer from transient loop issue. Imagine that the more specific routes fails on the primary PE, the primary PE will resort to the backup PE. At that time, it's probably that the backup PE is unaware of the failure. It will send packet back to the primary PE. Oops, loop occurs. Following the above "SHOULD NOT be installed" requirement, such kind of loops can be avoided.

<snip>
>More seriously:
>- Draft minto does not need to define a new protocol (ICCP) in order to learn the
>labels of the backup PE. BGP already provides this. While ICCP is a 79 pages
>draft, plus 11 for the extension.
>- Draft zhang relies on the having the same label on redundant PE. While this is
>doable, this is a significant change compared to the existing deployed model.
>- regarding context label tabel
>                  - From a standardization standpoint, context-specific label
>space is already defined in RFC 5331, so nothing new.
>                  - From an implementation standpoint, I agree that this
>requires some additional resources in the forwarding plane. But draft minto
>does not require that all PE support this. This can be performed by a protector
>PE if some PE do not support it.

[Mingui] If nothing new for protocols, this becomes an implementation issue. Then, should this be considered in IETF? :-)

[Mingui] We do not define ICCP, we make use of it. If we can have a solution happen in purely the control plane, as IETFers, we should be happy. If customers want this feature, I promise you will see this feature supported by products in several months. For vendors who have already implemented ICCP, it can be faster. More importantly, this is a "once for all" effort compared to the redundant forwarding tables. 

[Mingui] As Robert and I discussed previously, existing deploy model does not prohibit PE to advertise the same label. I don't think this changes the existing deploy model.

>So label table need not be stored repeatedly on RG members.
>
>#Bruno: agreed.
>
>
>2. Also, it's easy to be deployed. It does not bring any change to P routers
>#Bruno: It's not easy to deploy as it mandates to update all PE routers, both
>nominal and backup. While with draft-minto-2547-egress-node-fast-protection
><http://tools.ietf.org/html/draft-minto-2547-egress-node-fast-protection-02>
>the deployment of a single protector PE can fast protect all PE of the AS with no
>change on existing PE.

[Mingui] As I explained above, the update to the control plane should be OK.

[Mingui] OK. Let me give an analysis of the draft-minto-2547-egress-node-fast-protection. I hope this is constructive when the draft is updated.

[Mingui] a.	The co-located and hybrid models require configuration, which is error prone and increases a management burden to operators.
[Mingui] b.	The centralized model is not robust. How do you handle the failure of the centralized protector itself? This problem had been proposed for the "central controller" stuff. 
[Mingui] c.	The delegation of protector is topology dependent. I guess the number of possible alternative paths held by the protector is a much smaller set of all possible alternative paths. 
[Mingui] d.	The centralized protector has to manage the context label assignment and maintain multiple forwarding tables for all RGs. I am afraid this model is not scalable. 
[Mingui] e.	There is a brief description on Fast ReRoute in Section 5.3. For the "RSVP" choice, it depends on another individual draft [http://tools.ietf.org/html/draft-minto-2547-egress-node-fast-protection-02]. Does this draft mandates update on P and PE routers? For the "LDP" choice, it's not clear how IGP can find alternative paths (I find the "beyond the scope" words.). I think the coverage of alternative paths cannot be very good, especially when the centralized protector is used.

>3. In addition, it does not bear the restriction of "no penultimate-hop-popping".
>
>#Bruno: ultimate hop poping is only required for the label associated with the
>context table of the backup PE. This is required to identify the context-specific
>label space. This is not an additional argument compared to 1 (context label
>table).

[Mingui] I agree. 

Thanks,
Mingui

---

From: <[email-masked]>
Date: Thu, 14 Nov 2013 10:12:21 +0000

Hi Mingui,

Thanks for the answers.
You have added pwe in cc. I'm not sure why as I'm following up on your presentation during L3 VPN WG on L3 VPN PE protection.

My comments inline below [Bruno]

>From: >Mingui Zhang >Sent: Thursday, November 14, 2013 5:57 AM
>
>Hi Bruno,
>
>[Mingui] Here, I need to point out that the vNH is not created per CE. If
>multiple CEs are connected to an RG, one vNH is enough. When you read the
>first sentence of Section 2.1, you will be clear.
>[Mingui] "A vNH router is created in IGP to represent the set of CEs which
>are dual-homed to the same egress PEs in the Service Provider's backbone".

[Bruno] Agreed. 

>[Mingui] Exactly, it's per RG basis. In a real network, the numbers of RGs
>is comparable to the number of PEs, so it's similar to the per PE basis
>choice (the best choice) in draft-minto.

[Bruno] First, consider that in a "POP" there may be multiple PE, i.e. more than the 2 required for the dual homing. Let's say 5 PE.
The number of RG is flexible, but this also comes with drawbacks.
One option is to have a single RG. In this case, you indeed need a single vNH but you require global label allocation for the perimeter of those 5 PE. Which roughly divide the available label space by 5. Then if this represents a scaling issue, I guess you can call for bigger labels, e.g. by stacking 2 labels. But this requires more lookup work in the forwarding plane. Roughly equivalent to the label context space that you did not want in the first time...
Another option is to have one RG per couple of PE. In this case you need 10 vNH (IINM). That's roughly the number of PE but that's still *3 the number of states in the IGP so I think the document should talk about this.

>[Mingui] That's a valid point. We did define the "Priority" in the "vNH TLV"
>in Section 2.3.3 of the companion draft (http://tools.ietf.org/html/draft-
>zhang-pwe3-iccp-label-sharing-00). We can change the words to fix this
>inconsistence.

[Bruno] Ack. 

>[Mingui] So you propose the scenario that the backup PE directs packets to
>the primary PE since the primary PE advertised a more specific route. This
>makes the primary PE be an inevitable point on all forwarding paths to the
>CE. Eric ever used "BGP LOCAL_PREF" to compose a similar scenario
>(http://www.ietf.org/mail-archive/web/l3vpn/current/msg03691.html).
>
>[Mingui] For this subtle case, the redundant connection is not utilized in
>the first place. If customers prefer resilience to more specific routes, the
>backup PE should not use the primary PE as the next hop.

[Bruno] IMO this is a normal routing behavior which needs to be supported. It doesn't mean that the redundant is not utilized because another more specific prefix could be "attached" to this redundant connection. It probably means that the CE/customer is using TE to better load balance the flows across both connections.

>[Mingui] All in all, if customers really want the resilience, those routes
>via the primary PE SHOULD NOT be installed in the FIBs of backup PEs.

[Bruno] Good point. So you probably should add this to the document.

>[Mingui] By the way, the scenario you designed may suffer from transient
>loop issue. Imagine that the more specific routes fails on the primary PE,
>the primary PE will resort to the backup PE. At that time, it's probably
>that the backup PE is unaware of the failure. It will send packet back to
>the primary PE. Oops, loop occurs. Following the above "SHOULD NOT be
>installed" requirement, such kind of loops can be avoided.

[Bruno] You are right. I suspect that RFC 4364 implies that a PE should not reflect back to the core, a VPN packet received from the core. But I haven't checked.

>[Mingui] If nothing new for protocols, this becomes an implementation issue.
>Then, should this be considered in IETF? :-)

[Bruno] That's not a local implementation issue as this is a global behavior/feature at the network level. And to allow this behavior we need all implementations to implement this feature.
Note that all IETF documents do not describe change of the bits sent on the wire.
Finally, one protocol changed will be required.

>[Mingui] We do not define ICCP, we make use of it.

[Bruno] idem for context label :-)

> If we can have a solution happen in purely the control plane, as IETFers, we should be happy. 

[Bruno] Sure. But the solution relies on global VPN labels which is not what is deployed today. So it all depends on whether you will be able to deploy such change before. Which will requires all implementations to do it while draft minto has no such dependency.

> If customers want this feature, I promise you will see this feature supported
>by products in several months.

[Bruno] Ok. But I guess you mean a subset of the products currently deployed. There may be multiple vendors/implementations in a given network. And some implementations are sometimes even considered as legacy (by the vendor).

> For vendors who have already implemented
>ICCP, it can be faster. More importantly, this is a "once for all" effort
>compared to the redundant forwarding tables.
>
>[Mingui] As Robert and I discussed previously, existing deploy model does
>not prohibit PE to advertise the same label. 

[Bruno] Agreed.

> I don't think this changes the existing deploy model.

[Bruno] I think it does.

>[Mingui] OK. Let me give an analysis of the draft-minto-2547-egress-node-
>fast-protection. I hope this is constructive when the draft is updated.
>
>[Mingui] a.	The co-located and hybrid models require configuration, which is
>error prone and increases a management burden to operators.

#Bruno: You need to configure the pairs of PE. But this is a one time configuration on a per PE basis when the PE is provisioned. I don't think per VPN configuration is required as by listening to BGP routes, the backup PE can discover that it needs to back up the nominal one. Your proposition probably need the same configuration effort (configuring RG and vNH)

>[Mingui] b.	The centralized model is not robust. How do you handle the
>failure of the centralized protector itself? This problem had been proposed
>for the "central controller" stuff.

#Bruno: If one wants to protect the protection one may use a second (redundant) protector. 
However, as the protector is expected to be in use only few seconds per day at most, I'm not sure people would consider such specific case of double failure.

>[Mingui] c.	The delegation of protector is topology dependent. I guess the
>number of possible alternative paths held by the protector is a much smaller
>set of all possible alternative paths.

#Bruno: Indeed, currently the location of the protector is topology independent. Simply because this relies on the existing FRR mechanism used in the core and this FRR may have coverage limitations (e.g. LFA). There are plans to remove this limitation.
I'm not sure to understand your second point (second sentence).

>[Mingui] d.	The centralized protector has to manage the context label
>assignment and maintain multiple forwarding tables for all RGs. I am afraid
>this model is not scalable.

#Bruno: Just like a PE does (maintain multiple forwarding tables) so I think this is just as scalable as 4364 VPNs are. Besides, we are talking about dedicated protector PE so it can be choosen to scale in that direction (just like for VPN BGP RR)

>[Mingui] e.	There is a brief description on Fast ReRoute in Section 5.3. For
>the "RSVP" choice, it depends on another individual draft
>[http://tools.ietf.org/html/draft-minto-2547-egress-node-fast-protection-
>02]. Does this draft mandates update on P and PE routers? For the "LDP"
>choice, it's not clear how IGP can find alternative paths (I find the
>"beyond the scope" words.). I think the coverage of alternative paths cannot
>be very good, especially when the centralized protector is used.

#Bruno: For LDP, cf 2 points above.

>>label space. This is not an additional argument compared to 1 (context

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: Mingui Zhang <[email-masked]>
Date: Mon, 18 Nov 2013 09:31:41 +0000

Hi Bruno,

Please find my comments inline below [Mingui].

>You have added pwe in cc. I'm not sure why as I'm following up on your
>presentation during L3 VPN WG on L3 VPN PE protection.

[Mingui] Sorry that I didn't explain the reason every time. There was a companion draft presented in pwe3 for this l3vpn-PE-protection draft. Pwe3 WG determined they also need know the progress of this l3vpn draft.

>[Bruno] First, consider that in a "POP" there may be multiple PE, i.e. more than
>the 2 required for the dual homing. Let's say 5 PE.
>One option is to have a single RG. In this case, you indeed need a single vNH but
>you require global label allocation for the perimeter of those 5 PE. Which roughly
>divide the available label space by 5. 

[Mingui] That is a misinterpretation. Actually, all 5 PEs share the same label range (e.g., 1~1100) rather than dividing the label space. There is no scaling issue.

>Then if this represents a scaling issue, I
>guess you can call for bigger labels, e.g. by stacking 2 labels. But this requires
>more lookup work in the forwarding plane. 

[Mingui] As explained above, it does not need multiple lookup. Remember, we do not want to change the forwarding plane.

>Roughly equivalent to the label
>context space that you did not want in the first time...
>Another option is to have one RG per couple of PE. In this case you need 10 vNH
>(IINM). That's roughly the number of PE but that's still *3 the number of states
>in the IGP so I think the document should talk about this.

[Mingui] On one hand, the above extreme case happens only in theory. The reason we scope the document to talk about 2 PEs is that duel homing is the BCP. Think about the real deployment, I believe most POPs will use two PEs for the redundant connection. On the other hand, even if you have 5 PEs in one POP, just put them into one RG. Therefore, the increase of the number of states in the IGP is not so big.

[Mingui] But I agree with your advice that we can spend some words to address this scaling issue in the document.

<snip>
>>[Mingui] All in all, if customers really want the resilience, those
>>routes via the primary PE SHOULD NOT be installed in the FIBs of backup PEs.
>

[Mingui] Sure.

>>[Mingui] By the way, the scenario you designed may suffer from
>>transient loop issue. Imagine that the more specific routes fails on
>>the primary PE, the primary PE will resort to the backup PE. At that
>>time, it's probably that the backup PE is unaware of the failure. It
>>will send packet back to the primary PE. Oops, loop occurs. Following
>>the above "SHOULD NOT be installed" requirement, such kind of loops can be
>avoided.
>
>[Bruno] You are right. I suspect that RFC 4364 implies that a PE should not
>reflect back to the core, a VPN packet received from the core. But I haven't
>checked.

[Mingui] OK. How about the local-repair and label-retention technique? With that technique, the primary PE may cache the route advertised by the backup PE and use it when the route it learns from the CE fails. I guess this issue is kind of off the track. We can talk more about it between us two.

>[Bruno] Sure. But the solution relies on global VPN labels which is not what is
>deployed today. So it all depends on whether you will be able to deploy such
>change before. Which will requires all implementations to do it while draft minto
>has no such dependency.

[Mingui] The solution does not rely on *global* VPN labels. They are still locally assigned in the scope of the RG.

>> If customers want this feature, I promise you will see this feature
>>supported by products in several months.
>
>[Bruno] Ok. But I guess you mean a subset of the products currently deployed.
>There may be multiple vendors/implementations in a given network. And some
>implementations are sometimes even considered as legacy (by the vendor).

[Mingui] For those legacy products, operators can update their software. If bare legacy PEs and label-sharing PEs are interconnected, ICCP connection will not be established. It allows PEs falls back to the non-redundant connection. For those label-sharing PEs from different vendors, if they implement the same RFC, they are supposed to be interoperable.

>>number of possible alternative paths held by the protector is a much
>>smaller set of all possible alternative paths.
>
>#Bruno: Indeed, currently the location of the protector is topology independent.
>Simply because this relies on the existing FRR mechanism used in the core and
>this FRR may have coverage limitations (e.g. LFA). There are plans to remove
>this limitation.

[Mingui] The central protector is delegated as the PLR, right? This further reduces the coverage of LFA. As we know, a free LFA can have any router as the PLR.

Thanks,
Mingui