Subject: composite link - candidate for respin, maybe

From: Curtis Villamizar <[email-masked]>
Date: Fri, 26 Mar 2010 03:01:54 -0400

This email is to the authors of the CL drafts and the WG as a whole.

I started out writing a quick few bullets on what we talked about in
today's WG meeting that we seemed to agree on and that made some sense
to me.  The latter part may be a filter that introduces artifacts to
the signal (or noise).

That didn't seem to stand alone so I added some terms up front and
then some introductory text, trying to keep it concise.

What I would like to ask the WG is whether I'm on a reasonable track
here.  [hopefully we won't have continued dead silence on the list.]

What I would like to ask the authors is whether they would consider
this a reasonable restart point.

I've tried to keep in mind the chair's instructions and what I sensed
to be the feelings in the audience (though we didn't formally hum - is
formal WG hum an oxymoron?).

  keep it short - 5 pages including boilerplate if possible

  include clear requirements

  do not mandate implementation details

I'm not sure if the usage scenarios at the end are needed or if this
could stand alone.

There was a lot of chatter in the room so I appologize if I missed
something.  This is based on my recollection which is known to have a
fairly high bit error rate even on better days.

Curtis

btw - this is obviously not an I-D as is for more reasons than missing
boilerplate.  Its also late and I'm tired so I made no attempt to get
citation right.

Key terms:

  flow - A flow in the context of this document is a aggregate of
    traffic for which packets should not be reordered.  A flow is
    similart to a microflow or ordered aggregate as defined in
    [diffserv framework].  The term "flow" is used here for brevity.
    This definition of flow should not be interpreted to have broader
    scope than this document.

  flow identification - The means of identifying a flow or a group of
    flows may be specific to a type of payload.

  top label entry - In MPLS the top label entry contains the label on
    which an intitial forwarding decision is made.  This label may be
    popped and the forwarding decision may involve further labels but
    that is immeterial to this discussion.

  label stack - In MPLS the label stack includes all of the MPLS
    labels from the top of the stack to the label marked with the
    S-bit (Bottom of Stack bit) set.

  outer and inner LSP - The LSP associated with labels in the outer
    encapsulation are called outer LSP.  Those LSP which are
    associated with inner encapsulation (closer to the label entry
    containing the S-bit) are called inner LSP.  These are not called
    top and bottom LSP since MPLS and PWE draw the label stack in
    opposite directions with PWE putting the outermost label on the
    bottom of diagrams (and confusing people in doing so).

  component link - See RFC4201.

  composite link - [pull from existing I-D]

Introduction:

  There is often a need to provide large aggregates of bandwidth that
  is best provided using parallel links between routers or MPLS LSR.
  In core networks there is often no alternative since the aggregate
  capacities of core networks today far exceed the capacity of a
  single physical link or single packet processing element.

  Today this requirement can be handled by Ethernet Link Aggregation
  [IEEE802.1X], link bundling [RFC4201], or other aggregation
  techniques some of which may be vendor specific.  Each has strengths
  and weaknesses.

  The term composite link is more general than terms such as link
  aggregate which is generally considered to be specific to Ethernet
  and its use here is consistent with the broad definition in [ITU
  8xxx].

  Large aggregates of IP traffic do not provide explicit signaling to
  indicate the expected traffic loads.  Large aggregates of MPLS
  traffic are carried in MPLS tunnels supported by MPLS LSP.  LSP
  which are signaled using RSVP-TE extensions do provide explicit
  signaling which includes the expected traffic load for the
  aggregate.  LSP which are signaled using LDP do not provide an
  expected traffic load.

  MPLS LSP may contain other MPLS LSP arranged hierarchically.  When
  an MPLS LSR serves as a midpoint LSR in an LSP carrying other LSP as
  payload, there is no signaling associated with these inner LSP.
  Therefore even when using RSVP-TE signaling there may be
  insufficient information provided by signaling to adequately
  distribute load across a composite link.

  Generally a set of label stack entries that is unique across the
  ordered set of label numbers can safely be assumed to contain a
  group of flows.  The reordering of traffic can therefore be
  considered to be acceptable unless reordering occurs within traffic
  containing a common unique set of label stack entries.  Existing
  load splitting techniques take advantage of this property in
  addition to looking beyond the bottom of the label stack and
  determining if the payload is IPv4 or IPv6 to load balance traffic
  accordingly.

  For example a large aggregate of IP traffic may be subdivided into a
  large number of groups of flows using a hash on the IP source and
  destination addresses.  This is as described in [diffserv
  framework].  For MPLS traffic carrying IP, a similar hash can be
  performed on the set of labels in the label stack.  These techniques
  are both examples of means to subdivide traffic into groups of flows
  for the purpose of load balancing traffic across aggregated link
  capacity.  The means of identifying a flow should not be confused
  with the definition of a flow.

  Discussion of whether a hash based approach provides a sufficiently
  even load balance using any particular hashing algorithm or method
  of distributing traffic across a set of component links is outside
  of the scope of this document.

  The use of three hash based approaches are defined in RFCxxxx.  The
  use of hash based approaches is mentioned as an example of an
  existing set of techniques to distribute traffic over a set of
  component links.  Other techniques are not precluded.

Requirements:

  These requirements refer to link bundling solely to provide a frame
  of reference.  This requirements document does not intend to
  constrain a solution to build upon link bundling.  Meeting these
  requirements useing extensions to link bundling is not precluded, if
  doing so is determined by later IETF work to be the best solution.

  The first few requirements listed here are met or partially met by
  existing link bundling behavior including common behaviour that is
  implemented when the all ones address (for example 0xFFFFFFFF for
  IPv4) is used.  This common behaviour today makes use of a hashing
  technique as described in the introduction, though other behaviours
  are not precluded.

  1.  Aggregated control information which summarizes multiple
      parallel links into a single advertisement is required to reduce
      information load and improve scaleability.

  2.  A means to support very large LSP is needed, including LSP whose
      total bandwidth exceeds the size of a single component link but
      whose traffic has no single flow greater the component links.
      In link bundling this is supported by many implementations using
      the all ones address component addressing and hash based
      techniques.

      Note: some implementations impose further restrictions regarding
      the distribution of traffic across the set of identifiers used
      in flow identification.  Discussion of algorithms and
      limitations of existing implementations is out of scope for this
      requirements document.

  The remaining requirements are not met by existing link bundling.

  3.  In some more than one set of metrics is needed to accommodate a
      mix of capacity with different characteristics, particularly a
      bundle where a subset of component links have shorter delay.

  4.  A mechansism is needed to signal an LSP such that a component
      link with specific characteristics are chosen, if a preference
      exists.  For example, the shortest delay may be required for
      some LSP, but not required for others.

  5.  LSP signaling is needed to indicate a preference for placement
      on a single component link and to specifically forbid spreading
      that LSP over multiple component links based on flow
      identification beyond the outermost label entry.

  6.  A means to support non-disruptive reallocation of an existing
      LSP to another component link is needed.

  7.  A means to populate the TE-LSDB with information regarding which
      links (per end) can support distribution of large LSP across
      multiple component links based on the component flows and the
      characteristics of this capability.  Key characteristics are:

	a.  The largest single flow that can be supported.  This may
	    or may not be related to the size of component links.

	b.  Characteristics of the flow identification method.  [These
	    can be enumberated in this document or a later document. ]

	c.  Other characteristics?  [ Not sure if I got everything
	    mentioned in the WG meeting. ]

  8.  Some means is needed for an LSP which allows distribution of
      flows across member links to indicate characteristics of flow
      distribution.  These characteristics include:

	a.  The largest flow expected.

	b.  Other? [ Did we identify any other?  There was some
	    chatter about distribution of flows but no specific
	    characteristics was called for - AFAIK ]

  9.  In some cases it may be useful to measure link parameters
      and reflect these in metrics.  Link delay is an example.

  10. Some uses require an ability to bound the sum of delay metrics
      along a path while otherwise taking the shorted path related to
      another metric.  [This was mentioned but seems a bit orthogonal
      to all but #3.]

Purpose:

  [ A set of example scenarios were discussed.  We may want to capture
    them here (and maybe refine the examples). ]

---

From: "Mcdysan, David E" <[email-masked]>
Date: Fri, 26 Mar 2010 14:12:16 -0400

Hi Curtis,

Thanks for taking the initiative to write this up.

Also thanks to the rtgwg participants who provided many insightful
questions and suggestions at the mike as well as the helpful direction
from the chairs.

The definition of "flow" is not agreed to. The current scope is MPLS
only and not the Diffserv definition you propose unless the WG agrees to
extend the scope. I had asked the wg chairs to call for a resolution to
this decision on the list, and hopefully we can accomplish this soon.

A few detailed comments in line below.

If the other folks who took notes could publish their lists of things we
identified as requirements and/or problem statements.

Regards,

Dave 

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] 
> On Behalf Of Curtis Villamizar
> Sent: Friday, March 26, 2010 3:02 AM
> To: [email-masked]
> Subject: composite link - candidate for respin, maybe
> 
> 
> I started out writing a quick few bullets on what we talked 
> about in today's WG meeting that we seemed to agree on and 
> that made some sense to me.  The latter part may be a filter 
> that introduces artifacts to the signal (or noise).
> 
> That didn't seem to stand alone so I added some terms up 
> front and then some introductory text, trying to keep it concise.
> 
> What I would like to ask the WG is whether I'm on a 
> reasonable track here.  [hopefully we won't have continued 
> dead silence on the list.]
> 
> What I would like to ask the authors is whether they would 
> consider this a reasonable restart point.
> 
> I've tried to keep in mind the chair's instructions and what 
> I sensed to be the feelings in the audience (though we didn't 
> formally hum - is formal WG hum an oxymoron?).
> 

I heard 7 pages, including boilerplate, and Alex's response to my
question regarding "Acknowledgement of Prior work" material was that it
could be moved to an Appendix and not count against this quota. Alex,
please confirm.

> I'm not sure if the usage scenarios at the end are needed or 
> if this could stand alone.
> 
> There was a lot of chatter in the room so I appologize if I 
> missed something.  This is based on my recollection which is 
> known to have a fairly high bit error rate even on better days.
> 
> Curtis
> 
> 
> 
> btw - this is obviously not an I-D as is for more reasons 
> than missing boilerplate.  Its also late and I'm tired so I 
> made no attempt to get citation right.
> 
> 
> 
> 
> Key terms:
> 

Note that Diffserv microflow includes source address, source port,
destination address, destination port and protocol id. See scope
decision.

>     bottom of diagrams (and confusing people in doing so).

Need to cover case with more than two LSPs. Would using inner LSPs. (For
brevity is last sentence necessary?
>   component link - See RFC4201.
Not clear, component link  is not the bundled link, use existing I-D?

>   single physical link or single packet processing element.

Move following to Appendix on summary of Existing approaches per
direction from Alex and merge with existing material and your comments,
inputs to the list in the thread "Acknowledgement of Prior work" .

>   component links.  Other techniques are not precluded.

End of Material Moved to Appendix. 

I had envisioned a section summarizing network operator problems to be
solved preceding the requriements. I plan to draft some text and send it
out for comment on the list.  

>   IPv4) is used.  This common behaviour today makes use of a hashing
                                  ^Appendix^
>       information load and improve scaleability.

Based upon Tony's comments, I think that the objective is to convey the
same information as a set of parallel links with different
characteristics more efficiently and in a more scalable manner than
making an advertisement for each parallel link. As discussed,
summarization (e.g., more than one value for latency) is a specific
solution. Comments, other recollections?

>       total bandwidth exceeds the size of a single component link but
						           than
largest
>       whose traffic has no single flow greater ^ the   ^ component
link.

Above is an important requirement you mentioned. 

Move following to Appendix
>       limitations of existing implementations is out of scope for this
End Move to Appendix.

>       bundle where a subset of component links have shorter delay.

I would avoid use of metric to avoid confusion with current solutions.
Proposed rewording as follows:

A means in control and data plane protocols is needed to accomodate a
composite link composed of component links with different
characteristics, including at least: capacity, current latency,
indication of whether latency can change, ... others?

>       some LSP, but not required for others.

As discussed in the meeting, picking the shortest delay per composite
link is one requirement as you state above. 

We need to add the other service provider requirement described in the
meeting where certain LSPs have a latency that is less than a specified
end-end value. 

In my view, these are separate requirements, which may have different
solutions.

>       identification beyond the outermost label entry.

Need to clarify whether this applies to outer LSP and/or Inner LSP(s).
As discussed, we need to add description that Composite Link end point
routers participate in outer LSP signaling, may "snoop" signaling for
inner LSPs(), or may be able to determine that a label may be used for
component link assignment decisions (e.g., entropy label). 

>       LSP to another component link is needed.

Need to include the control of change frequency from 4.1.1.2.3 the
existing I-D to this requirement. 

In the current draft, the use of MPLS TC (aka EXP) and DSCP bits are
only specified for this purpose. Do we want to describe use in other
cases? Also should describe fact that most operators do not modify DSCP
and map this to EXP bits so that DSCP is transparent to customers. 

> 	    can be enumberated in this document or a later document. ]

Would this be a place where MPLS and IP requirements would be
differentiated?

> 	    mentioned in the WG meeting. ]

We ran out of time in the wg, but support for LDP is an important
operator requirement. The vast majority of L3VPNs run over LDP
"tunnels." I think that stating the why instead of how for the material
on this subject from section 4.2.3 is something we need to do. 

Also need to cover case mentioned by Ning that both LDP and RSVP-TE will
be present on the same composite link. I think you are proposing to add
unlabeled IP traffic to this (data plane) set as well. 

Also, traffic measurement based support at the composite link is
important.

Need to describe the operator requirements on what needs to happen in
the event of "Bandwidth Shortage Events"  See section 4.1.4.1. 

I was going to describe "auto-bandwidth" for RSVP-TE as a current method
used by operators that the composite link should still support. 

How control (routing, signaling) and management (OAM) packets are
directed to component links and how this needs to be done so that no
impacts to liveliness, adjacency and/or OAM occurs needs to be stated.

Add dynamic signaling and advertisement of lower layer component links,
and feedback from the lower layer regarding latency.  See 4.2.1.1.

Backward compatibility as you proposed on previous thread (merge with
text in 4.2.1.3)

Automatic derivation of routing metrics based upon signaled (or
measured) latency changes (4.2.1.3).

Would look to others who were keeping notes to add to the above list.

> 	a.  The largest flow expected.

Please say more, not sure I understand this point. 

> 	    characteristics was called for - AFAIK ]

See above; change control frequency,

Also, "pinning" in some way (see 4.1.1.1) based upon snooped  signaling
for inner LSPs and configuration (e.g., of FEC (ranges)). 

>       and reflect these in metrics.  Link delay is an example.

I think this is one of the most important requirements. Editorially, it
should be toward the top of the list.

As commented by Dimitri; need to state requirements on the frequency of
latency measurements and the precision required.

>       another metric.  [This was mentioned but seems a bit orthogonal

See 4 above as well. We think the characteristics of 4.2.1.3 and 4.2.2.1
are important, but need to focus the text on the semantics of why these
are important instead of the current descriptions which are a how
assuming crankback style signaling as the solution approach. A
discussion of the how alternatives view could be moved to the framework
document.

Also as discussed in the meeting, a means to implement Diffserv for
MPLS-TE when composite links are present in the network is also
required. 

>     them here (and maybe refine the examples). ]

IMO, example scenarios,or solution sketches may be more appropriate for
a framework document. 

> 
> _______________________________________________
> rtgwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtgwg
>

---

From: Tony Li <[email-masked]>
Date: Fri, 26 Mar 2010 15:55:01 -0700

Hi Dave,

> destination address, destination port and protocol id. See scope

I think that where we're getting in trouble here is the analogy to Diffserv.
I'm not certain that Curtis intended to invoke all of the differentiators
here that you're objecting to.  Could we simply drop the analogy and delete
the second and third sentences?

> brevity is last sentence necessary?

How about innermost?

> solution. Comments, other recollections?

Agreed.  Perhaps it would be useful here to again back up and define our
requirements.  How many total components are we willing to advertise from a
single router?  How many routers do we see in a particular IGP area/level?

Obviously we can't carry additional information in zero bits, but we can
afford some bits per link if we're willing to carry a larger link state
database (LSDB).  If we can define what we're targeting, then we can do the
computations to see if everything would fit.

Note that asking for infinite of everything is by definition not helpful.
While that precludes carrying things in the IGP, that also effectively
precludes using crankback as a solution as well, as crankback setup times
will explode.

As always, we need to make an intelligent time/space tradeoff.

> indication of whether latency can change, ... others?

I don't find 'metric' as ambiguous, as we've dealt with multi-metric based
IGP's before.  

> In my view, these are separate requirements, which may have different

And again, this wording seems to imply that we're doing path selection via
signaling and not routing.  How about s/signal/compute/?  Same below...

> Need to clarify whether this applies to outer LSP and/or Inner LSP(s).

I think the point here was that we should provide a knob.  I concur.

'Nuf for now,
Tony

---

From: Curtis Villamizar <[email-masked]>
Date: Sun, 28 Mar 2010 01:33:56 -0400

In message <[email-masked]>
> this decision on the list, and hopefully we can accomplish this soon.

As Tony's email pointed out, we may be in agreement.  I might have
been unclear.

> identified as requirements and/or problem statements.

Further comments inline.

Regards,

Curtis

> could be moved to an Appendix and not count against this quota. Alex,

The request was 5 pages.  The compromise was 7 pages.  The real point
was to keep it concise.  If clarity is improved with a little more,
then I'm sure it would be OK.  If the document rambles (as I am doing
now), clarity is reduced with length.

> destination address, destination port and protocol id. See scope

Tony suggested (in email) that I remove all but the first sentence.
I'd like to keep the example but improve clarity as follows.

  flow - A flow in the context of this document is a aggregate of
    traffic for which packets should not be reordered.  A flow in this
    context should not be confused with a microflow or ordered
    aggregate as defined in [diffserv framework] which share the
    similarity of requiring that reordering be avoided but is specific
    to IP.  The term "flow" is used here for brevity.  This definition
    of flow should not be interpreted to have broader scope than this
    document.

> brevity is last sentence necessary?

Outer and inner are directions.  Outermost and innermost are top and
the one marked with BOS.  This allows us to say "top label" and inner
labels (plural).  Outer are the label(s) used for forwarding.  If PHP
is used, numerous outer labels may be popped, requiring the next label
to be examined before the packet is sent on its merry way.

According to his email Tony was also confused and thought inner meant
innermost.  Let me try again.

  outer LSP(s) and inner LSP(s) - The LSP(s) associated with labels in
    the outer encapsulation are called outer LSP.  The outer label
    stack entries are used for forwarding.  The remaining LSP(s) which
    are associated with inner encapsulation (closer to the label entry
    containing the S-bit) are called inner LSP(s).  There is a single
    outermost LSP and innermost LSP, but may be multiple outer and
    inner LSP.  These are not called top and bottom LSP since MPLS and
    PWE draw the label stack in opposite directions with PWE putting
    the outermost label on the bottom of diagrams (and confusing
    people in doing so).

That is getting rather long.

> Not clear, component link  is not the bundled link, use existing I-D?

A component link is a single link in the link bundle.  We can pull a
definition from elsewhere if you prefer.  Ethernet LAG uses "member".
Link bundle uses "component".

> >   composite link - [pull from existing I-D]

We can pull component from there too if there is a clear definition.

> End of Material Moved to Appendix. 

I think this belong up front.  We need to define the problem and it
doesn't hurt to spend a page describing what exists today.  The extent
to which some requirements are already met *is* relevant.

> out for comment on the list.  

I was going to put story telling time at the end.  :-)

> solution. Comments, other recollections?

I think you mean Tony's comment at the mic.

This requirement #1 is to reduce the size of the TE-LSDB.  This is
what link bundle already does get right.  You seem to be commenting on
item 3 after reading only item 1.

> Above is an important requirement you mentioned. 

Again, link bundling does this but there is no way for the ingress to
pick which behaviour it wants.  That is item #5.

> >       limitations of existing implementations is out of scope for this

I do think this is relevant and belongs here.  Items #1 and #2 are
requirements met by link bundling.  The reset is requirements not met.

> Proposed rewording as follows:

Metric is used in the LS routing protocol sense.

> indication of whether latency can change, ... others?

That is item #4.  Please read to the end before citing omissions.

> link is one requirement as you state above. 

OK.

> meeting where certain LSPs have a latency that is less than a specified

Item #10.  Again, read to the end before citing omissions.  It wasn't
*that* long.

> In my view, these are separate requirements, which may have different

That is what one is item #4 and the other is item #10.

> component link assignment decisions (e.g., entropy label). 

What I had in mind was two choices, outer only or all.  See revised
definition of outer and inner above.

I really don't want to get into counting a certain depth into the
stack as no hardware today does that.

If we did add that, we'd need a capability indication for 1) can use
outer only, 2) can also use entire stack up to N, 3) can use lesser of
M (if M is speicified in LSP) or entire stack up to N.  Specifying
that is not a problem.  No existing equipment or emerging merchant
Ethernet silicon does this but we're not cast in silicon yet so OK
with me.  :-)  Others might object.

> existing I-D to this requirement. 

OK.  That belongs with new LSP parameters (Item #8).

> and map this to EXP bits so that DSCP is transparent to customers. 

Are you suggesting that we move some TC values and not others?  That
could be done today by creating two LSP one that is more "moveable"
than the other among the set of component links.

No hardware today routes based on both label and EXP at a midpoint.
However some hardware today will allow yout to use DSCP or TC to
direct traffic into different LSP.

> Would this be a place where MPLS and IP requirements would be

Yes.  IP would typically be just src/dst hash.  MPLS would be how the
label stack is used, and whether the implementation looks past the
label stack to see if traffic is IP.  This is not the place for
protocol details but each of these characteristics need to be encoded
in a compact encoding.

> on this subject from section 4.2.3 is something we need to do. 

This (Item #7) is a discussion relevante link LSA or LSPDU TLVs.  The
above comment is relevant to iten #8 but I'll respond here.

The esisting load balance implementations are control plane agnostic.
The reserved bandwidth is used for admission control to the aggregate
capacity when the load balance is enabled.

Of the bag of tricks the link has to offer the control plane picks
what the LSP wants.  We can certainly extend LDP as well as RSVP-TE as
described in item #8.

> unlabeled IP traffic to this (data plane) set as well. 

We can just as easily include RSVP-TE, LDP, and IP.  Both LDP and IP
traffic have to be measured.  The means to identify groups of flows
differs and IP bahavior will need to be configured (ie: management
plane) but otherwise techniques are the same for IP and LDP.

> Also, traffic measurement based support at the composite link is

Item #9 covered this.  The sum of LDP and IP can be used to reduce the
reservable bandwidth.  There is no way to "preempt" IP or LDP without
causing oscillations so only RSVP-TE can be preempted if measurement
is done.  I pointed this out in an earlier email.

> the event of "Bandwidth Shortage Events"  See section 4.1.4.1. 

Preemption works.  LDP and IP can't be preempted.  Maybe MUST NOT
result in oscillations can be added and put an end to this.

> used by operators that the composite link should still support. 

See above mention of "he sum of LDP and IP can be used to reduce the
reservable bandwidth."  The existing "auto-bandwidth" is an ingress
function where the requested resrvation is changed.

> impacts to liveliness, adjacency and/or OAM occurs needs to be stated.

For control "nothing breaks" would be a sufficient requirement.

We can mention that LM measurement is lost on any LSP that is not kept
atomic (is allowed to be load balanced to multiple components) despite
the characteristic of having no reordering on flows.  Otherwise OAM
works.

A requirement would be "impacts on OAM must be documented".  A
dissertation on the topic is not needed in the requirements document.

> and feedback from the lower layer regarding latency.  See 4.2.1.1.

Are we aggregating information or advertising individual link?  This
is another case of "which is it".  I thought we agreed to aggregating
information except for circumstances where some subset of the links
had different delay.

> Backward compatibility as you proposed on previous thread (merge with

I thought the WG meeting concensus was to toss the original.  BTW- Its
interesting for a link to advertise its desired maximum latency.

Advertising delay is one thing.  We don't want to get carried away and
specifify mean, stddev, N-th norm, or percentile histograms.  Lets
stick with a single number for delay.

> measured) latency changes (4.2.1.3).

If we add a delay metric, then the delay metric can depend on measured
delay.  Adjusting general routing metrics according to load takes us
down the road of provably stable mechanisms.  "MUST NOT oscillate" is
a strong requirement.

> Please say more, not sure I understand this point. 

If CL component links are 10G each, the CL can't handle a single flow
of 20G that came in on a 100G link.  If the LSP claims to have up to
20G flows, the LSP has to be rejected at setup time.

This is not all that hard to do.  If the largest customer attachment
is a 10G rate limited to 4G, then the largest possible flow is 4G.  If
a customer has a 100G attachment then the largest possible flow can be
a negociated (ie: at contract time) parameter.  If it is a content
provider traffic is going to a million eyeballs and the largest flow
may be much smaller than the attachment circuit.

> See above; change control frequency,

If we base this on hash, then the control frequency can't easily be
held constant over the entire set of LSP.  My assumption is that LSP
that were put on a single component would only be moved if link bundle
style LSP allocation had created a "bin packing" problem that had to
be resolved by moving something and the remaining "spread spectrum"
LSP would fill in the remaining capacity (which in practice would most
likely be most of the bandwidth).

> for inner LSPs and configuration (e.g., of FEC (ranges)). 

Pinning has to be on the outer labels or not at all.  The LSR is not
going to made parapsychic by these requirements.

> should be toward the top of the list.

Order is not intended to indicate importance.  It was supposed to be a
logical grouping with capabilities building on prior ones.

> latency measurements and the precision required.

We can save that for the later.  Requirements are just that the
frequency of sampling be configurable.

> discussion of the how alternatives view could be moved to the framework

I think we agreed at the meeting to toss the existing draft.  For
example, how does an LSP signal "delay variation".  A link would
signal jitter if anything didi.  The ingress can try to predict
cummulative jitter base on link jitter along the path.  I don't think
there is concensus on adding jitter though.

> MPLS-TE when composite links are present in the network is also

Before we add a requirement, you need to explain how support for
diffserv would ever be lost by load balancing.

> IMO, example scenarios,or solution sketches may be more appropriate for

If they are properly specified as a set of cases, then yes.

Lets get past the requirments first.

---

From: Curtis Villamizar <[email-masked]>
Date: Sun, 28 Mar 2010 01:57:04 -0400

Hi Tony,

> the second and third sentences?

Let me know if the modified version in the 

> > brevity is last sentence necessary?

Again see response to Dave.

> As always, we need to make an intelligent time/space tradeoff.

What I had was an item #1 which was to retain aggregation of
information and iten #3 was to partially deaggregate.

For example, if there is a microwave link on mountaintops across a
body of water woth 4 x OC12 (0.65) at a shorter delay and a fiber path
with longer delay but 12 x 10G, then the advertisement could have a
set of link bundle like TLVs on the link advertisement with:

    Max BW: 122.5G
    Bax LSP: 10G
    Group 1:
      Max BW: 2.5G (nominal)
      Max LSP: 0.65G (nominal)
      delay: 2 msec (big puddle)
    Group 2:
      Max BW: 120G
      Max LSP: 10G
      delay: 5 msec
    load balance: full stack to N=8

Maybe not the best example, but the point is the first two would be
for backward compatibility with link bundle and the rest would be for
the more enlightenned ingress LER.

> I don't find 'metric' as ambiguous, as we've dealt with multi-metric based

Agreed.

> signaling and not routing.  How about s/signal/compute/?  Same below...

Is RSVP-TE LSP setup signaling?  If I change signal to setup in the
above sentence are we OK?

> I think the point here was that we should provide a knob.  I concur.

My response to Dave was the we should specify only one of two options.
Either use the outer labels (usually means just top unless a POP
occurs) and don't load balance the LSP or use the whole label stack
and spread the traffic.

We could also defer discussion of the shape of the knob to later.

> Tony

Plenty for now.

Thanks,

Curtis

---

From: Tony Li <[email-masked]>
Date: Sun, 28 Mar 2010 00:25:59 -0700

Hi Curtis,

> Let me know if the modified version in the

Saw that.  I can live with it, but would prefer more brevity.  The inverse
definition (this is not XYZ ...) holds little value for me.

> the more enlightenned ingress LER.

Or, we go down the path of advertising vectors of more explicit information
on a per-component basis.  This may be necessary when we start talking about
allocated capacity.

It also occurs to me that we need to resolve whether or not our goals
include the capability of doing globally optimal calculations.  To do so
requires that we be able to do centralized computations and establish paths
down to the component level.

> above sentence are we OK?

Yes, I consider RSVP-TE LSP setup to be signaling.  No, changing this to
setup doesn't work for me.  Again, are we fulfilling the requirements at
path computation time or setup time?  At this point, I don't think we want
to bind to one approach.

> We could also defer discussion of the shape of the knob to later.

I don't see that we're going to be able to pick and choose here.  Some
networks will not need diversity for their LSPs.  Others will require it.  I
think we have to be more granular.

Regards,
Tony

---

From: Curtis Villamizar <[email-masked]>
Date: Sun, 28 Mar 2010 23:11:30 -0400

> definition (this is not XYZ ...) holds little value for me.

In general that is true.  In this case there is a significant tendency
to equate CL flow with diffserv microflow.  It might be a necessary
clarification.

I'll look at it again and try to be more concise.

> down to the component level.

What we don't want to do (IMHO) is completely remove aggregation of
the bundle (aka CL) advertisement.  If there are a dozen identical
links and four that are differnt there is no need to pick exactly one,
just one from either set, or don't care.

If we don't put a bound on the number of sets, in the limit each
component could be advertised with the added semantic that there is a
sharing option.  In that case the LSP may need to specify the set of
"groups" (as I have it above in the absense of a term) that are
acceptable.  This could make for a very big ERO.  It should be
adequate to indicate in the RRO that the bundle is being used with a
field indicating which member was currently in use (if nailed in
place).  This is beyond requirements, so if we are OK with the stated
requirement, let defer the details to later.

> path computation time or setup time?  At this point, I don't think we want

The ingress has to indicate which group of links to use, in the
example, the lower delay group, or the fatter pipe, or don't care.
The ingress decides this in RSVP-TE or in MPLS-TP without a control
plane the management plane provides this information.

If you consider RSVP-TE to be signaling, this is correct.  In the
absense of a control plane it is not needed but in the presence of a
control plane it is needed.

> think we have to be more granular.

OK.  So what is your objection?  Are you suggesting that we need to
restrict the load balance to look down N labels from the top and no
further?

> Tony

Good to be working with you again.

Regards,

Curtis

---

From: Tony Li <[email-masked]>
Date: Sun, 28 Mar 2010 22:38:25 -0700

Hi Curtis,

> just one from either set, or don't care.

My concern is that down the road, as resources get used, the links will no
longer be identical.   Each will have its own unique usage levels.  Global
planning and optimization can use this information to determine exactly how
many LSPs and at what sizes can fit down the link.

> requirement, let defer the details to later.

I'm fine with that, but we may also want to specify the component in the
ERO.

> restrict the load balance to look down N labels from the top and no

No, I'm suggesting that we need a bit in the RESV to say that this LSP
should be demuxed by inner labels or kept atomic.

> Good to be working with you again.

Likewise.  While we have a history of disagreeing, I find it far preferable
to disagree intelligently with someone than some of the other discussions
that I've been having lately.  ;-)

Tony

---

From: Curtis Villamizar <[email-masked]>
Date: Mon, 29 Mar 2010 02:49:38 -0400

> many LSPs and at what sizes can fit down the link.

My response included the following:

   If we don't put a bound on the number of sets, in the limit each
   component could be advertised with the added semantic that there is
   a sharing option.  In that case the LSP may need to specify the set
   of "groups" (as I have it above in the absense of a term) that are
   acceptable.  This could make for a very big ERO.  It should be
   adequate to indicate in the RRO that the bundle is being used with
   a field indicating which member was currently in use (if nailed in
   place).  This is beyond requirements, so if we are OK with the
   stated requirement, let defer the details to later.

So what I'm saying is you could do that.  I think it is a real bad
idea, but you could do it.

> I'm fine with that, but we may also want to specify the component in the

Again, if you specificy the group, and there is only one component in
the group, then you have that behavior.  There is also the option of
putting more than one component in the group.  We are getting past
requirements.  This would be fine fodder for a framework (aliteration
was unitentional, but amusing nonetheless).

> should be demuxed by inner labels or kept atomic.

That is in there but whether it is one bit is not defined:

It doesn't specify "a bit".  I think this would be in the PATH (from
ingress) and maybe in the ERO if at some hops the load split options
were unacceptable but at other hops a suitable method was found.
Anyway that is beyond the requirments stage.

We can rearrange for clarity if we have general agreement on content.
If this were combined with the other LSP parameters it might be more
clear.

> that I've been having lately.  ;-)

I've been told that we generally disagree in a very entertaining
fashion.  That has to count for something.  :-)  [what I don't know]

Regards,

Curtis

---

From: Tony Li <[email-masked]>
Date: Mon, 29 Mar 2010 00:03:20 -0700

> idea, but you could do it.

I think we're miscommunicating.  Rather than have a big ERO, I'm suggesting
that this info go into the IGP and that the ERO specify the specific member.
I'm not so scared about adding to the IGP as we can start by advertising
about 275KB per router without a challenge.

> was unitentional, but amusing nonetheless).

Concur, but we seem to be characterizing the requirements around a
particular approach.  I'm trying to avoid doing that.

>       identification beyond the outermost label entry.

Do you meant to imply that there would also be a way to indicate a
preference for diverse placement of a hierarchical LSP?

Tony

---

From: "So, Ning" <[email-masked]>
Date: Mon, 29 Mar 2010 15:09:06 +0000

Tony and Curtis,

Thanks a lot for the great input and comments.  I am trying to take care
of my family and my day job which pays my salary at Verizon first since
I was gone from both last week :)  I will try to catch up on this e-mail
thread later this week and get back to you.  I am slow, but hopefully
steady.  

Regards,

Ning So
Lead Engineer
Global Data Network Traffic Planning
972-729-7905

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Tony Li
Sent: Monday, March 29, 2010 2:03 AM
To: [email-masked]
Cc: [email-masked]
Subject: Re: composite link - candidate for respin, maybe

> idea, but you could do it.

I think we're miscommunicating.  Rather than have a big ERO, I'm
suggesting
that this info go into the IGP and that the ERO specify the specific
member.
I'm not so scared about adding to the IGP as we can start by advertising
about 275KB per router without a challenge.

Tony

---

From: John E Drake <[email-masked]>
Date: Mon, 29 Mar 2010 08:56:35 -0700

Comments inline

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Tony Li
> Sent: Sunday, March 28, 2010 10:38 PM
> To: [email-masked]
> Cc: [email-masked]
[JD] 

Then why bother with link bundling?  This level of control has been part of TE since the beginning.  Link bundling was introduced to trade information hiding for scalability.

> I'm fine with that, but we may also want to specify the component in the
[JD] 

See http://datatracker.ietf.org/doc/draft-ietf-mpls-explicit-resource-control-bundle/

> should be demuxed by inner labels or kept atomic.
[JD] 

In current link bundling an LSP is either assigned to a specific component link or split across all of the component links.  In the latter case this effectively means all or a subset of the component links.

If an LSP is assigned to a specific component link, RFC 5150 (http://datatracker.ietf.org/doc/rfc5150/) (a one hop LSP on the assigned component link) and MBB can be used to move the LSP non-disruptively to another component link

---

From: Tony Li <[email-masked]>
Date: Mon, 29 Mar 2010 10:59:19 -0700

> Then why bother with link bundling?  This level of control has been part of TE
> since the beginning.  Link bundling was introduced to trade information hiding
> for scalability.

Because we can gain a great deal of scalability by advertising vectors of
metrics rather than explicit metrics per link.  So, rather than:

    link 1
        bandwidth 10gbps
    link 2
        bandwidth 40gbps
    link 3
        bandwidth 100gbps

We have:

    composite links
        bandwidth 10, 40, 100

This saves an enormous amount of overhead in TLV encoding.  If we can
further only vectorize a subset of the metrics for the composite link (e.g.,
IGP metric and link coloring might be common), we get further scaling
advantages.  And on top of that, little or no loss of useful information.

> See 
> http://datatracker.ietf.org/doc/draft-ietf-mpls-explicit-resource-control-bund
> le/

Perfect, thanks.

Tony

---

From: "Mcdysan, David E" <[email-masked]>
Date: Mon, 29 Mar 2010 14:22:38 -0400

Comments in linbe below.

Dave 

> From: Tony Li [mailto:[email-masked]] 
> Sent: Monday, March 29, 2010 3:03 AM
> To: [email-masked]
> Cc: Mcdysan, David E; [email-masked]
> I think we're miscommunicating.  Rather than have a big ERO, 
> I'm suggesting that this info go into the IGP and that the 
> ERO specify the specific member.
> I'm not so scared about adding to the IGP as we can start by 
> advertising about 275KB per router without a challenge.
> 
> 

This is how I had interpreted Tony's comment at the mike. If all of the
"useful" characateristics component links are carried in the composite
link advertisement, then optimization can be done on a specified metric
(e.g., IGP "cost", TE-metric, or a "latency" related metric), then SPF
can be done at the source subject to constraints. For example, if
"latency" and available capacity is advertised, then a solution to
requirement 10 that Curtis wanted to make out of scope seems like a
straightforward extension, without using crankback signaling. 

> >> I'm fine with that, but we may also want to specify the 
> component in 
> >> the ERO.
> > 
> > the group, then you have that behavior. 

I think this is a valid construct. Propose that the definition of
composite link allow one or more component links. 

There is also the 
> option of 
> Concur, but we seem to be characterizing the requirements 
> around a particular approach.  I'm trying to avoid doing that.
> 
> 
> >> No, I'm suggesting that we need a bit in the RESV to say that this 
> >> LSP should be demuxed by inner labels or kept atomic.
> > 
> Do you meant to imply that there would also be a way to 
> indicate a preference for diverse placement of a hierarchical LSP?
> 
> 
> Tony
> 
> 
>

---

From: "Mcdysan, David E" <[email-masked]>
Date: Mon, 29 Mar 2010 14:20:52 -0400

Hi John,

Comments in line.

Thanks,

Dave 

> On Behalf Of John E Drake
> Sent: Monday, March 29, 2010 11:57 AM
> To: Tony Li; [email-masked]
> Cc: [email-masked]
> Subject: RE: composite link - candidate for respin, maybe 
> 
> Comments inline
> 
> > From: [email-masked] 
> [mailto:[email-masked]] On Behalf 
> > Of Tony Li
> > > links and four that are differnt there is no need to pick exactly 
> > > one, just one from either set, or don't care.
> > 
> > planning and optimization can use this information to determine 
> > exactly how many LSPs and at what sizes can fit down the link.
> [JD] 
> 
> Then why bother with link bundling?  This level of control 
> has been part of TE since the beginning.  Link bundling was 
> introduced to trade information hiding for scalability.

This is a key point. 

As mentioned in the current draft and as one of the first operator
stories that I told in the meeting is the current practice is to keep
the links separate since link bundling aggregation loses too much
information. If latency is important, then setting the TE-metric
proportional to the latency meets some of the requirements. However, if
one wants to set the TE-metric as a funciton of something other than
latency, then the status quo does not allow some customer or internal
use requirements in terms of latency to be supported. 

Some form of composition (or bundling) in routing must give some benefit
in terms of scalability/ stability as compared with separate links. 

> component in the
http://datatracker.ietf.org/doc/draft-ietf-mpls-explicit-resource-contro
l-bundle/

Being able to identify the component link on which an LSP is placed (or
spread across) is I believe a requirement which we need to capture,
which it appears this draft is proposing a specific solution.

> In current link bundling an LSP is either assigned to a 
> specific component link or split across all of the component 
> links.  In the latter case this effectively means all or a 
> subset of the component links.
> 
> If an LSP is assigned to a specific component link, RFC 5150 
> (http://datatracker.ietf.org/doc/rfc5150/) (a one hop LSP on 
> the assigned component link) and MBB can be used to move the 
> LSP non-disruptively to another component link  

minimal disruption. Some disruption (e.g., packet reordering) could
occur if the links have different latencies.

---

From: John E Drake <[email-masked]>
Date: Mon, 29 Mar 2010 11:27:10 -0700

Tony,

The statement I was responding to seemed to be implying per component  
link advertisements.

The difference between advertising one bundled link with multiple sets  
of parameters or multiple bundled links each with one set of  
parameters doesn't seem so significant.

Thanks,

John

Sent from my iPhone

---

From: Greg Mirsky <[email-masked]>
Date: Mon, 29 Mar 2010 11:55:31 -0700

Dear Tony,
a link might have B/W per TC. Thus encoding

   - composite links

bandwidth 10, 40, 100
will become

   - composite links
      - P0 a, b, c
      - P1 ...

That will same some encoding but I wonder if such saving is real motivation
for this work.

Regards,
Greg

---

From: John E Drake <[email-masked]>
Date: Mon, 29 Mar 2010 11:54:59 -0700

Dave,

If, when an LSP is being moved from one component link to another with  
MBB, packets of given flow are all placed on the same component link,  
there shouldn't be an issue with packet re-ordering even if the two  
links have different delays.

Thanks,

John

Sent from my iPhone

On Mar 29, 2010, at 11:23 AM, "Mcdysan, David E" <[email-masked] 
 > wrote:

---

From: "Mcdysan, David E" <[email-masked]>
Date: Mon, 29 Mar 2010 15:28:19 -0400

Hi John,

The scenario I had in mind was as follows:

1. An LSP is currently placed on Component Link 1 (CL.1)
2. A request is made to move this LSP to Component Link 2 (CL.2)
3. An LSP (segment) is Made on CL.2 Before Breaking the LSP (segment) on
CL.1 
4. At some point in time (t1) the router stops sending packets from the
flow from the LSP (segment) on CL.1 and begins sending these packets on
the LSP on CL.2

Define the difference between the latency of CL.2 and latency of CL.1 as
Delta. 

Define amount of time that the last packet of the flow is sent on CL.1
as t0 and the time that the first packet is sent on CL.2 as t2.  

By construction t0 <= t1 <=t2.

If Delta>0 and Delta>(t2-t0) then at least one packet on CL.1 will
arrive before the first packet arrives from CL.2, and at least these two
packets will be out of order.

If Delta=0 then there is no problem and there truly is no disruption. 

If Delta< 0 then two packets that were shaped so as to be spaced out in
a regular way will become clumped together, increasing short-term delay
variation as perceived by the receiver.

Only when Delta=0 is this truly "non-disruptive," but I would consider
the above scenario "minimally disruptive" in the sense that there is no
loss. However, as was commented by several people controlling the
frequency of such MBB LSP "movements" is an operational requirement.

Thanks,

Dave 

> Sent: Monday, March 29, 2010 2:55 PM
> To: Mcdysan, David E
> Cc: [email-masked]
> If, when an LSP is being moved from one component link to 
> another with MBB, packets of given flow are all placed on the 
> same component link, there shouldn't be an issue with packet 
> re-ordering even if the two links have different delays.
> 
> Thanks,
> 
> John
> 
> Sent from my iPhone
> 
> On Mar 29, 2010, at 11:23 AM, "Mcdysan, David E" 
> <[email-masked]  > wrote:
> 
> > Hi John,
> >
> > Comments in line.
> >
> > Thanks,
> >
> > Dave
> >
> >> From: [email-masked] [mailto:[email-masked]] On 
> >> Behalf Of John E Drake
> >> Then why bother with link bundling? This level of control has been 
> >> part of TE since the beginning.  Link bundling was introduced to 
> >> trade information hiding for scalability.
> >
> > This is a key point.
> >
> > proportional to the latency meets some of the requirements. 
> However, 
> > if one wants to set the TE-metric as a funciton of something other 
> > than latency, then the status quo does not allow some customer or 
> > internal use requirements in terms of latency to be supported.
> >
> > Some form of composition (or bundling) in routing must give some 
> > benefit in terms of scalability/ stability as compared with 
> separate 
> > links.
> >
> >>
> >>>
> >>>
> > Being able to identify the component link on which an LSP is placed 
> > (or spread across) is I believe a requirement which we need to 
> > capture, which it appears this draft is proposing a 
> specific solution.
> >
> >>
> >>>
> >> In current link bundling an LSP is either assigned to a specific 
> >> component link or split across all of the component links.  In the 
> >> latter case this effectively means all or a subset of the 
> component 
> >> links.
> >>
> >> (http://datatracker.ietf.org/doc/rfc5150/) (a one hop LSP on the 
> >> assigned component link) and MBB can be used to move the LSP 
> >> non-disruptively to another component link
> >

---

From: John E Drake <[email-masked]>
Date: Mon, 29 Mar 2010 12:45:44 -0700

Dave,

The e-mail to which I was replying stated that there might be packet  
re-ordering, and I was just pointing out that a reasonable  
implementation can avoid either packet re-ordering or loss.

I seriously doubt that per flow shaping is done on links between core  
routers.

Thanks,

John

Sent from my iPhone

On Mar 29, 2010, at 12:28 PM, "Mcdysan, David E" <[email-masked] 

>> http://datatracker.ietf.org/doc/draft-ietf-mpls-explicit-resource- 
>> cont
>>> ro
>>> l-bundle/
>>>

---

From: Tony Li <[email-masked]>
Date: Mon, 29 Mar 2010 13:37:01 -0700

Hi John,

> implementation can avoid either packet re-ordering or loss.

I have to respectfully disagree.

The case that Dave points out is exactly right: if a router is transitioning
to a link of lower delay, then has no choice but to risk mis-ordering
packets.  The only other implementation alternative is to actually buffer
them, which has some horrible side-effects for congestion control.

Alternatively, if you have some magic that you're willing to share with us
wherein you can get some packets to move faster than others, we'd love to
know about it.  ;-)

Tony

---

From: Tony Li <[email-masked]>
Date: Mon, 29 Mar 2010 14:12:13 -0700

Hi John,

> parameters doesn't seem so significant.

Well, that depends entirely on our scalability goals, which is why I would
like to see our requirements quantified.

As an example, if we use IS-IS with TE right now, a link consists of:
      2 octets of TLV overhead
      7 octets of system Id and pseudonode number
      3 octets of default metric
      1 octet of length of sub-TLVs
      6 octets of Administrative group (color)
      6 octets of IPv4 interface address
      6 octets of IPv4 neighbor address
      6 octets of Maximum link bandwidth
      6 octets of Reservable link bandwidth
      34 octets of Unreserved bandwidth (for 8 traffic classes)
      5 octets of TE Default metric

If we want to track latency, then I'd propose that we need 3 bytes to track
worst case latency on the link  (2 bytes overhead, 1 byte for ms granularity
latency).

If I've done my math right, that works out to 85 bytes.

If we then vectorize this, we would need, per component:
    4 octets of maximum link bandwidth
    4 octets of reservable link bandwidth
    32 octets of unreserved bandwidth
    1 octet of latency

That's 41 bytes.

If we're willing to dispense with pre-reserved bandwidth per traffic class,
then we can further drop 32 octets.

So we're down to 9 bytes per link, or almost a 10x improvement in
scalability.

IMHO, that's significant.

If we now have ~16 components per composite link, then we can advertise the
composite link in about 200 bytes.  That works out to well over 1000
composite links (and therefore 16,000 components) per system.  And that's
before we break a sweat and do funky things to get more LSP space (e.g.,
expanded MTUs, zero-cost pseduonodes for more fragments).

So, what are the real requirements?

Tony

---

From: John E Drake <[email-masked]>
Date: Mon, 29 Mar 2010 17:46:09 -0700

Tony,

I was looking at this situation as a special case of placing different flows on different component links.  So, you are correct in saying that flows may be re-ordered which is business as usual, but within the flows the packets are not mis-ordered.

I.e., packets within a given flow would be placed on either the new component link or the old component link.  After some period of time, packets of all flows would be placed on the new component link and the one hop LSP on the old component link would be torn.

Thanks,

John 

> Sent: Monday, March 29, 2010 1:37 PM
> To: John E Drake; Mcdysan, David E
> Cc: [email-masked]

---

From: John E Drake <[email-masked]>
Date: Mon, 29 Mar 2010 18:01:23 -0700

Tony,

Clearly there is extra overhead (roughly 2*(N - 1)* 40 bytes?) with N homogeneous bundled links versus one heterogeneous bundled link.  The concern I have is that having heterogeneous bundled links may complicate the data plane packet forwarding.

A clear statement as to what we are trying to accomplish wrt IGP scaling by using bundled links would be good.

Thanks,

John

> Sent: Monday, March 29, 2010 2:12 PM
> To: John E Drake
> Cc: [email-masked]; [email-masked]

---

From: Curtis Villamizar <[email-masked]>
Date: Mon, 29 Mar 2010 21:53:47 -0400

> particular approach.  I'm trying to avoid doing that.

I'm giving an example in email.  There is no indication of such an
approach in the revised text I sent.

Aggregating information in the IGP is still a requirement.  It is at
least desireable.  I think highly desireable.  Specifying more than
one metric is covered.  I gave the example of one component per group
if someone really fealt a need to pick the component in the ERO rather
than a group with known characteristics.

> preference for diverse placement of a hierarchical LSP?

That is definitely one of the choices.  Just as is done today.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Mon, 29 Mar 2010 22:18:53 -0400

> Comments inline

John,

Another top posting windoze user?  I'm disappointed.  :-)

> part of TE since the beginning.  Link bundling was introduced to trade
> information hiding for scalability.

You are missing a few things.

One is the abilit to move LSP from one component to another.  In
Tony's view that would not be possible because the LSP is nailed to
one component, not a group with common attributes and a specification
of maximum dynamic.

Another is the ability to allow an LSP to be load split across a set
of components.  The top LSP may (if signaling indicates this is OK) be
split among component links.  This is the only way to support LSP of
greater than 100G in the near future.

> http://datatracker.ietf.org/doc/draft-ietf-mpls-explicit-resource-control-bundle/

The functionality I mentioned above is missing.

> component link or split across all of the component links.  In
> the latter case this effectively means all or a subset of the
> component links. 
>  

Creating every LSP by stitching one hop LSPs together is a very poor
solution IMO.  The requirement though do not specify a solution.

The point is to allow a midpoint or pair of midpoints on either end of
a CL to perform such tasks as rearranging LSP to fix a bad bin packing
outcome with minimal changes to the way an LSP is set up.  If the
commitment of the RRO is to use any LSP within a group that has a
specific set of characteristics, and to stay within some parameters
related to anchoring the LSP (not an absolute commitemtn to never move
the LSP but a commitment to move it only within some time limits or
under specific conditions).

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Mon, 29 Mar 2010 22:43:15 -0400

> advantages.  And on top of that, little or no loss of useful information.

In this case it shouldn't matter which component link is chosen if the
LSP is under 10G.  If the delay on one link is shorter then it would
matter to some LSP but not others.

It save a lot if there are dozens of links of each type.  Then we
have:

      composite link (singular)
          J @ bandwidth 10 delay X
	  K @ bandwidth 40 delay Y
	  L @ bandwidth 100 delay Z

> > http://datatracker.ietf.org/doc/draft-ietf-mpls-explicit-resource-control-bund

Quite a bit less than perfect.  A small step toward the requirements.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 30 Mar 2010 00:26:17 -0400

> links have different delays.

If moving from a longer delay to a shorter delay, interleaving and
therefore reordering can occur.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 30 Mar 2010 00:44:10 -0400

Tony,

Its not just about whether it fits into the available set of LSPDU.
Every time a new LSP is signaled reservable bandwidth changes and it
could potentially mean another advertisedment of a bloated LSPDU
fragment (or a much smaller but still bloated opaque LSA).

This can be a big problem after a fault has occurred, particularly a
fault involving more than one SRLG where protection also failed.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 30 Mar 2010 00:49:13 -0400

In message <[email-masked]>
John E Drake writes:
>  
> Tony,
>  
> Clearly there is extra overhead (roughly 2*(N - 1)* 40 bytes?) with N
> homogeneous bundled links versus one heterogeneous bundled link.  The
> concern I have is that having heterogeneous bundled links may
> complicate the data plane packet forwarding.

How the control plane information is presented does not "complicate
the data plane packet forwarding".

> A clear statement as to what we are trying to accomplish wrt IGP
> scaling by using bundled links would be good.

Please indicate what in the requirements we are discussing is not
clear rather than a blacket statement that the goals are not clear.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 30 Mar 2010 01:36:09 -0400

In message <[email-masked]>
John E Drake writes:
>  
> Tony,
>  
> I was looking at this situation as a special case of placing
> different flows on different component links.  So, you are correct in
> saying that flows may be re-ordered which is business as usual, but
> within the flows the packets are not mis-ordered. 
>  
> I.e., packets within a given flow would be placed on either the new
> component link or the old component link.  After some period of time,
> packets of all flows would be placed on the new component link and the
> one hop LSP on the old component link would be torn. 
>  
> Thanks,
>  
> John 

John,

When nothing is being moved nothing is reordered.  If a flow or many
flows is moved from a link with longer delay to one with shorter
delay, unless the flow is stopped and bufferered for at least the
delay difference and then drained, then reordering will occur.

So I agree with Tony.  It is not practical to avoid reordering when
traffic is moved.  It is therefore minimally disruptive.  The products
of your new employer do the same thing.

Curtis

---

From: Tony Li <[email-masked]>
Date: Tue, 30 Mar 2010 00:17:42 -0700

> clear rather than a blacket statement that the goals are not clear.

Once again: what are the scalability goals that we're trying to achieve,
both on a per-system and network-wide basis?  This should be a quantitative
requirement.

Tony

---

From: Tony Li <[email-masked]>
Date: Tue, 30 Mar 2010 00:21:48 -0700

Hi Curtis,

> fragment (or a much smaller but still bloated opaque LSA).

? 1200 bytes is hardly bloated.  And I think we've established that the
frequency of churn is more important than the size.

And we've have the issue about LSPDU updates since we started the TE work.
I believe that we've demonstrated that it can be successfully resolved
without causing undue hardship.

> fault involving more than one SRLG where protection also failed.

Not sure I see this as a "big problem".  Yes, there will be a spike in
processing, but we get that anytime there's a topological change.  Doesn't
seem like an insurmountable issue.

Tony

---

From: Tony Li <[email-masked]>
Date: Tue, 30 Mar 2010 00:27:43 -0700

> That is definitely one of the choices.  Just as is done today.

Could we please explicitly call out all of the alternatives here, please?
Just to be crystal clear...

Thanks,
Tony

---

From: Tony Li <[email-masked]>
Date: Tue, 30 Mar 2010 00:30:21 -0700

Hi Curtis,

> one component, not a group with common attributes and a specification

Actually, I'm rather into flexibility.  Some LSPs might be pinned.  Some
might not.  

Tony

---

From: John E Drake <[email-masked]>
Date: Tue, 30 Mar 2010 08:05:53 -0700

[JD] 

Moving a section of an LSP from one path to another does not require link bundling.

[JD] 

The discussion was about IGP scaling.  Both examples you describe deal with signaling and could be decoupled from any IGP enhancements that are made.

> a
> stated
> control-bundle/
[JD] 

Then you need to re-phrase your requirement (or re-read the I-D)

[JD] 

It depends upon how many LSPs need this behavior.  LSP Hierarchy would also work.  So the requirement needs to be more precise.

---

From: John E Drake <[email-masked]>
Date: Tue, 30 Mar 2010 08:23:28 -0700

> From: [email-masked] [mailto:[email-masked]]
> Sent: Monday, March 29, 2010 9:49 PM
> To: John E Drake
> Cc: Tony Li; [email-masked]; [email-masked]
[JD] 

If we are placing flows on the component links of a heterogeneous link bundle, we may have to look at additional fields within packets and/or maintain additional state, relative to what is required to place flows on the component links of a homogeneous link bundle.  The example that has been given is flows that require low latency.

The requirements document needs to be clear as to which fields in the packet header are to be used in packet classification.  E.g., do we deal strictly with the MPLS header or can the packet header also be used, and if so, do we have to deal with encrypted as well as unencrypted packets? 

[JD] 

I was agreeing with Tony, so please direct this question to him.  (But, I don't see any real discussion of IGP scaling requirements.)

---

From: John E Drake <[email-masked]>
Date: Tue, 30 Mar 2010 08:27:59 -0700

[JD] 

One last time:  Flows may be re-ordered, but packets within flows are not re-ordered.  This is no different than placing flows of a single LSP on multiple component links, which is a required behavior.  The only difference is that when an LSP is pinned to a specific component link, this only occurs when the LSP is being pinned to a new component link using MBB.

---

From: Tony Li <[email-masked]>
Date: Tue, 30 Mar 2010 09:26:34 -0700

Hi John,

> If we are placing flows on the component links of a heterogeneous link bundle,
> we may have to look at additional fields within packets and/or maintain
> additional state, relative to what is required to place flows on the component
> links of a homogeneous link bundle.  The example that has been given is flows
> that require low latency.

If we go by the definition that we're working with, a "flow" is atomic and
therefore should never be demuxed, just to ensure ordering.

> I was agreeing with Tony, so please direct this question to him.  (But, I
> don't see any real discussion of IGP scaling requirements.)

I think that this has to come from Dave.  If I get to set the requirements,
I promise a Very Low Bar.   ;-) ;-) ;-)

Tony

---

From: "Mcdysan, David E" <[email-masked]>
Date: Wed, 31 Mar 2010 07:19:28 -0400

I had a general statement about scaling in the wg slides, and I recall
Alex objecting that in general the IETF avoids quantitative values. 

This is an important issue, and is one we need to address to better
understand the cost/benefit of approaches that differ from just
extending the set of advertised parameters and/or their useage in
signaled LSPs.

I propose that we discuss a parameteric model, e.g., Number of nodes
(N), total number of Links (L), some range of Component links per
Composite (C(i)), frequency of latency changes (L), frequency of
capacity changes, etc. and try to compare the "scalability" and
"response time" in a parametric way. Then, outside of the IETF,
quantitative numbers could be inserted.

Dave

> On Behalf Of Tony Li
> Sent: Tuesday, March 30, 2010 3:18 AM
> To: [email-masked]; John E Drake
> Cc: [email-masked]
> Once again: what are the scalability goals that we're trying 
> to achieve, both on a per-system and network-wide basis?  
> This should be a quantitative requirement.
> 
> Tony
> 
>

---

From: "Mcdysan, David E" <[email-masked]>
Date: Wed, 31 Mar 2010 07:19:28 -0400

"Pinning" an LSP to a specific component link was an intended use case
from the prior draft not in the current list. 

Do need to think more about the means (configuration, signaling) of how
this could be done. I will draft some text as to why an operator would
want to do this.

Dave

> Sent: Tuesday, March 30, 2010 3:30 AM
> Actually, I'm rather into flexibility.  Some LSPs might be 
> pinned.  Some might not.  
> 
> 
> Tony
> 
>

---

From: "Mcdysan, David E" <[email-masked]>
Date: Wed, 31 Mar 2010 07:19:28 -0400

> Sent: Tuesday, March 30, 2010 12:27 PM
> To: John E Drake; [email-masked]
> Cc: [email-masked]
> > If we are placing flows on the component links of a 
> heterogeneous link 
> > bundle, we may have to look at additional fields within 
> packets and/or 
> > maintain additional state, relative to what is required to 
> place flows 
> > on the component links of a homogeneous link bundle.  The 
> example that 
> > has been given is flows that require low latency.

The current draft combines change frequency and low latency. I think we
need to separate these out as separate characteristics.

> 
> 
> If we go by the definition that we're working with, a "flow" 
> is atomic and therefore should never be demuxed, just to 
> ensure ordering.
> 
> 
> > I was agreeing with Tony, so please direct this question to him.  
> > (But, I don't see any real discussion of IGP scaling requirements.)
> 
> 

I agree that we need to have scaling requirements. Also, I think that
convergence time (or response time to a transient) is something we need
to specify. See my earlier note.

---

From: Tony Li <[email-masked]>
Date: Wed, 31 Mar 2010 09:22:37 -0700

Hi Dave,

> Alex objecting that in general the IETF avoids quantitative values.

I object to the objection.  ;-)

Without a quantitative goal, we're left arguing in a vacuum about what the
real requirements are and whether they can be met.  To understand the
solution space, we need to understand the problem, and since this clearly
includes scaling, we need to understand our scaling goals.

I'll point out that there is precedent for quantitative work.  For example,
when we were discussing IPng, we definitely set goals for the number of
hosts and number of networks to be supported.

Alex, could we please get some reconsideration here?

> quantitative numbers could be inserted.

I agree with the general approach.  There are lots of different link
characteristics that we carry and I'm wondering if we can generalize and
just carry the frequency of change of link characteristics.  This is so we
don't drown in our own model.

Tony

---

From: "So, Ning" <[email-masked]>
Date: Wed, 31 Mar 2010 19:51:23 +0000

All,

I am starting another thread of the discussion on the topic for a
moment.  I took some notes during the meeting, but I was only tying done
the key words the speaker was trying to communicate.  I expanded them
according to my memory and understanding here.   Lucy also took some
notes.  And the meeting minutes should come out soon.  We need to
assemble a complete set of notes/recollections from the meeting, so that
we can ensure all the issues raised during the meeting are addressed on
the mailing list.  

My meeting notes:  

1.  Upper layer and lower layer (outer and inner label) awareness.  Top
label too large, how to distribute lower label.

2.  Change make-before-break to "un-obstructive"

3.  draft to RFP like

4.  signaling to indicated if LSP can/want to be split or not

5.  combination of latency and BW in signaling

6.  end-to-end latency vs. local latency (advertisement and requirement)

7.  Can latency be categorized and use affinities to indicate

8.  Measurement (latency) frequency.  Other things to be measure such as
burst?  Specify what to measure

9.  How often to move LSP?

10.  Label stack lookup flexibility

11.  Spell out IPSec tunnel so no hashing 

12.  After traffic assignment, traffic behavior (BW and so on) may
change

13.  give priority of requirements

14.  limit the draft to 5 pages (7 pages)?  Put additional material into
appendix?

Ning So
Lead Engineer

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 01 Apr 2010 00:52:33 -0400

> both on a per-system and network-wide basis?  This should be a quantitative

When BGP specified the implied goal to pack more than one NLRI into a
BGP update there was no need to justify the efficiency improvement
with target size of the global routing table and routing information
capacity.

We are shooting for very significantly better packing by allowing CL
with very many components to be expressed in a more compact way.

I don't care much about the CL with two components.  Its the one with
many 10s of components that matter.  These exist today.

That is all the justification we need.

If you really insist I'll add the bits for a 30 component CL using the
existing bundling vs as individual links as an example and stick it in
the draft.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 01 Apr 2010 01:13:20 -0400

> without causing undue hardship.

If there is a lot of churn, a lot of information is sent rather than a
summarization then there is a lot of processing.  The less individual
entities to manange in the TE-LSDB the better.

> seem like an insurmountable issue.

I'd have to say that if you don't see a big problem you are just
oblivious to the problem.

There have been numerous presentations in NANOG on order of LSA/LSPDU
processing, fixed origination holdoffs vs variable origination
holdoffs vs immendiate origination, readvertise before SPF, SPC
holdoffs, CSPF timer reductions, incremental SPF and CSPF, CSPF
results caching, achieving real-world sub-second IGP convergence,
achieving faster real world reroute times (for mutliple fault
situations when protection also fails), etc.  These presentation have
come from both vendors and providers with providers mostly reiterating
requirements or reporting on experiences.

A lot happens when a multiple fault occurs.  Some alternate paths fill
and the feedback doesn't reach some ingress before they initiate
signaling attempting to use the no longer available resource.  The
feedback is important but never instantaneous and there is a tradeoff
between advertising every small change and loading CPUs and getting
infornation out fast.

If this doesn't all go smoothly and convergence is over the SLA
threshhold then it is clearly a big deal and it carries a big
financial penalty.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 01 Apr 2010 01:26:49 -0400

OK.

   5.  LSP signaling is needed to indicate a preference among the
       following two behaviours.

         a.  Placement of the LSP on a single component link.  This
             option specifically forbids spreading that LSP over
             multiple component links based on flow identification
             beyond the outermost label entry.

         b.  Load balancing using flow identification information
             beyond the outermost label entry.  The preferred flow
             identification method and flow distribution method must
             be a method whose support is indicated in the link
             advertisement.

I was under the impression that the exact set of mechanisms is out of
scope for the requirements document.  If we were to put in any load
balancing mechanism at this time we would document simple hash based
load balancing by citing RFC2991 and RFC2992.  We would also leave
codepoints to allow other methods to be specified.  We might even set
aside a set of experimental codepoint available for "unspecified
vendor magic" to be assigned real code points later.

For the requirements document the only requirement is "extensible".

This is supposed to fit in 5 pages after all.  The point was *not* to
constrain the requirments by specifying a particular implementation.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 01 Apr 2010 01:59:48 -0400

> Moving a section of an LSP from one path to another does not require

Yes entirely true but being able to move a pinned LSP and pin it
somewhere else is not the only requirement.

> The discussion was about IGP scaling.  Both examples you describe
> deal with signaling and could be decoupled from any IGP
> enhancements that are made. 

You've deleted enough context (the examples) that I'm not sure if I'm
adequately addressing your point here.

In order for an ingress to specify that some sort of load split of an
LSP cound occur it needs to know that that capability is available on
that link.  So for example, if an LSP is 180G and there are two 100G
interfaces in parallel, the ingress needs to know that this is a CL
and that it can handle this LSP.  The IGP provides that information.

> Then you need to re-phrase your requirement (or re-read the I-D)

Again you deleted context.  By "re-phrase your requirement" I take it
that you mean just the following requirement:

There is nothing here that prohibits using stitching or any other
method.  We are not defining *how* to do anything, just what the
requirements are.

The very first paragraph reads:

What about "solely to provide a frame of reference" don't you
understand?

I don't see any point in adding that "One of these requirements might
be met by LSP stitching.  This requirements document does not intend
to constrain a solution to build upon LSP stitching.  Meeting these
requirements useing extensions to LSP stitching is not precluded, if
doing so is determined by later IETF work to be the best solution."

Do you get it now?

> It depends upon how many LSPs need this behavior.  LSP Hierarchy
> would also work.  So the requirement needs to be more precise. 

I also do not see the point of adding "One of these requirements might
be met by LSP hierarchy.  This requirements document does not intend
to constrain a solution to build upon LSP hierarchy.  Meeting these
requirements useing extensions to LSP hierarchy is not precluded, if
doing so is determined by later IETF work to be the best solution."

We are not specifying solutions at this point.  We are specifying
requirements.  Got it?

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 01 Apr 2010 02:47:48 -0400

In message <[email-masked]>
John E Drake writes:
> If we are placing flows on the component links of a heterogeneous link
> bundle, we may have to look at additional fields within packets and/or
> maintain additional state, relative to what is required to place flows
> on the component links of a homogeneous link bundle.  The example that

If the LSP requires low latency, then the LSP is signaled for low
latency.  If it asked to be pinned, all the flows in that LSP are
pinned to one component link.  That is easy, just map the label using
the ILM to one link.

If the LSP requires a particular flavor of low latency, but it doesn't
mind being load split then it is load split across only those
components that meet that latency requirement.  The top label
identifies the subset of the component links that are "fair game" and
some algorithm that the LSP has signaled it is OK to use splits the
flows among those links.

If you think about how this is implemented in routers today this is
all doable now, just not with a very good load split.  If you don't
have the slightest clue how it is done today RFC2991 and RFC2992 will
provide the slightest clue but not much more.

> The requirements document needs to be clear as to which fields in the
> packet header are to be used in packet classification.  E.g., do we
> deal strictly with the MPLS header or can the packet header also be
> used, and if so, do we have to deal with encrypted as well as
> unencrypted packets?

No it doesn't!  We are specifying requirements, not solutions.  When
we get around to specifying solutions then it does.

There is 10-15 years of history on this that says it is doable,
because it is being done so we are well beyond the point where we need
example to establish that a feasible technique exists.

OK.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 01 Apr 2010 02:55:44 -0400

In message <[email-masked]>
John E Drake writes:
> One last time: Flows may be re-ordered, but packets within flows are
> not re-ordered.  This is no different than placing flows of a single
> LSP on multiple component links, which is a required behavior.  The
> only difference is that when an LSP is pinned to a specific component
> link, this only occurs when the LSP is being pinned to a new component
> link using MBB.

John,

You are making no sense at all.

Consider the case where only one flow exists.  Two path exist, A and
B.  Path A is used.  It has a longer delay than B.  The difference in
delay is N packet times.  When switching from A to B, the first packet
to arrive on path B arrives N-1 packets before the last packet to go
into link B.  Packets from that one flow are interleaved for about N
packets.

The case that we are talking about is moving a flow or group of flows
from one link to another.  It is minimally disruptive.  Do you have a
problem with that wording?

Curtis