Subject: Acknowledgement of Prior Work (Was: Composite Link Requirements)

From: "Mcdysan, David E" <[email-masked]>
Date: Tue, 2 Mar 2010 09:25:12 -0500

Hi Curtis,

The co-authors of this draft reviewed your comments and decided to
respond with three separate messages, to separate the threads as follows
so that all of the issues you raise can be resolved efficiently.

	#1 Composite Link Trademark Issue (Was: Composite Link
Requirements)
	#2. Acknowledgement of Prior Work (Was: Composite Link
Requirements)
	#3. Proposed Resolution of Comments (Was: Composite Link
Requirements)

This is thread #2.

Dave  

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] 
> On Behalf Of Curtis Villamizar
> Sent: Saturday, February 27, 2010 4:00 AM
> To: [email-masked]
> Subject: Composite Link Requirements
> 
> 
> Hi there good people of RTGWG,
> 
> This is in regards to the goals that are embodied in the 
> RTGWG acceptance of a draft to deal with requirements for 
> composite link, currently named draft-ietf-rtgwg-cl-requirement-00.txt
> 
> 
> I'm bringing up two issues in this email.  One is prior 
> composite link work and the 

This is the subject of this thread: 
> other is prior methods of 
> handling composite link, which should be acknowledged.  

Snipped

> 
> Note that ITU's G.800 does not define what a composite link 
> is and only mentions composite four times in the document, 
> including use of composite link and composite trail.  The 
> figure indicates that a composite link is "inverse 
> multiplexing".  For this reason, I don't think G.800 should 
> be referenced because its a big load of **** with only slight 
> mention of CL.
> 

Part of the rtgwg acceptance of this a wg draft was to include a
reference to G.800. Lucy pointed out that the current text in section
2.2 is a paraphrase. We could replace with the following quote from
section 6.9.2 of G.800, if that is wg consensus:

"Multiple parallel links between the same subnetworks can be bundled
together into a single composite link. Each component link of the
composite link is independent in the sense that each component link is
supported by a separated service layer trail. The composite link conveys
communication information using different server layer trails thus the
sequence of symbols cross these links may not be preserved."

The text related to Inverse multiplexing is one of three cases in
section 6.9.2. The text above is the first case. G.800 states that these
are separate cases.

The text above Figure 16 in G.800 related to concatenated server trails
may also be relevant (at least in the framework).

So, the choices are replace current text with the direct quote, remove
the reference to G.800 and replace it with some other text.

WG comments?

Text Snipped

> 
> Second issue is how CL has been handed in the past.
> 
> Whether it was two links to two places that took completely 
> different paths (trails in ITU speak but this is IETF where 
> we say path), or two parallel links, this has been called 
> ECMP in IETF (and elsewhere) for two decades or more.  Both 
> ISIS and OSPF use the term ECMP.  The techniques used for 
> ECMP load balance was discussed on IETF lists quite a bit in 
> the early to mid-1990s.  The three techniques applied to IP 
> networks (in the terminology of that time) were:
> 
>   1.  per packet load balance
>   2.  per bit or byte load balance aka bit striping or inverse-mux
>   3.  IP src/dst hash
> 
> The second is applicable only to parallel links.  Using 
> larger chunks it is also the technique used in MPPP 
> (multilink PPP).  MPPP is also sometimes abbreviated PPP-ML, 
> though not in the RFC.  MPPP is no longer of much interest as 
> it was only applied to low speed links.
> 
> The per packet load balance caused packet reorder and a great 
> deal of grief for service providers, hence the abundance of 
> discussion within IETF at the time.  The use of IP src/dst 
> hash, while widespread and widely discussed, did not get 
> documented in an RFC until Chris Hopps and Dave Thaler wrote 
> RFC 2991 "Multipath Issues in Unicast and Multicast Next-Hop 
> Selection" and RFC 2992 "Analysis of an Equal-Cost Multi-Path 
> Algorithm" in November 2000.  (at least AFAIK).
> 
> The IP src/dst technique itself is beleived to have 
> originated in the T1-NSFNET, which puts its use back to circa 1987.
> 
> The OMP work predates RFC 2991 and RFC 2992 but never made it 
> past the internet-draft stage.  In that work the use of 
> src/dst hash and the use of adaptive algorithms with src/dst 
> hash is discussed.  On the IETF mailing lists even methods of 
> implementation were discussed, table based and parallel sets 
> of comparator pairs (TCAM like).
> 
> Circa 2000 there was a lot of discussion of the use of the 
> MPLS label stack to provide the entropy for ECMP vs looking 
> past the label stack at the IP payload.  Today's PW control 
> word acknowledges this common practice and avoids it for PW, 
> but the fat-pw aka entropy label puts better entropy back into PW.
> 
> In practice today, all core hardware uses the same IP src/dst 
> hash to provide a load balance for ECMP and LAG.
> 
> 
> The existing internet-draft acknowledges link bundling, but 
> does not accurately characterize ECMP and LAG and the src/dst 
> hash technicque used by both, nor does it acknowledge the 
> prior OMP work.
> 

The existing draft states some of these points already, but there is
certainly more background information that you provide. It seems that
you have more specific suggestions on the text in the draft and that is
where we propose specific changes to address your comments. Another
approach could be to add more text on IP-related load balancing as
compared with the MPLS-based load balancing which is the focus of the
draft.

Could you provide a URL for the prior OMP work that we can add as
informative reference.

Also, the scope of the draft is MPLS, which is not covered specifically
in the points above. 

Remainder of orignal message snipped.

---

From: "Mcdysan, David E" <[email-masked]>
Date: Tue, 2 Mar 2010 09:25:16 -0500

Hi Curtis,

This is thread #3.

Dave   

> composite link work and the other is prior methods of 

This is the basis for thread#3.

> After 
> that I just have some comments and questions on the draft.

Text Snipped

> 
> Comments on the draft:
> 
> The following statements may be inaccurate:
> 
>    The Link Bundle concept is somewhat limited because of the
>    requirement that all component links must have identical
>    capabilities, and because it applies only to TE links.
> 
>      This may be inaccurate.  I don't think there is a requirement
>      that a link bundle use identical links.
> 
>      In any case, both Avici composite links and many LAG
>      implementations allow a mix of member speeds and neither was
>      applicable to TE links only.

Not clear what specific text change is being proposed. Does the
following address your comment?

EXISTING 3.1 TEXT 
   o  Advertisement of each component link into the IGP. Although this
      would address the problem, it has a scaling impact on IGP routing,
      and was an important motivation for the specification of link
      bundling [RFC4201]. However, there are two gaps in link bundling:

         1.  It only supports RSVP-TE, not LDP.

   	   2.  It does not support a set of component links with
different
      characteristics (e.g., different bandwidth and/or latency).

      For example, in practice carriers commonly use link bandwidth and
      link latency to set link TE metrics for RSVP-TE.  For RSVP-TE,
      limiting the component links to same TE metric has the practical
      effect of dis-allowing component links with different link
      bandwidth and latencies.

PROPOSED TEXT

   o  Advertisement of each component link into the IGP. Although this
      would address the problem, it has a scaling impact on IGP routing,
      and was an important motivation for the specification of link
      bundling [RFC4201]. However, there are two gaps in link bundling:

         1.  It only supports RSVP-TE, not LDP.

   	   2.  It only supports advertisement of a single TE metric for 
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
	a set of component links, 

	However, the component links in this set may have different
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
	characteristics (e.g., different bandwidth and/or latency).
      For example, in practice carriers commonly use link bandwidth and
      link latency to set link TE metrics for RSVP-TE.  For RSVP-TE,
      limiting the component links to same TE metric has the practical
      effect of dis-allowing component links with different link
      bandwidth and latencies.

> 
> The following should be replaced:
> 
>    Traffic Flow: A set of packets that with common identifier
>    characteristics that the composite link is able to use to aggregate
>    traffic into Connections.  Identifiers can be an MPLS label stack
>    or any combination of IP addresses and protocol types for routing,
>    signaling and management packets.
> 
> Diffserv already defined a microflow to be the same thing.  
> We should not invent new terms to mean the same thing as 
> existing terms.  We can just point out that labels can also 
> be used to identify a microflow.

A clear definition is certainly the goal. Can you provide a specific
Diffserv reference for review by the Working Group. 

> 
> This statement is definitely inaccurate for a number of reasons:
> 
>    o  ECMP/Hashing/LAG: IP traffic composed of a large number of flows
>       with bandwidth that is small with respect to the individual link
>       capacity can be handled relatively well using ECMP/LAG
>       approaches.  However, ?many/some of? these approaches do not
make use of MPLS
                              ^^^^^^^^^^^^^^
>       control plane information nor traffic volume
>       information. Distribution techniques applied only within the
>       data plane can result in less than ideal load balancing across
>       component links of a composite link.
> 
>   Avici used feedback from the egress port to the ingress port on
>   traffic volume and queue occupancy to influence the distribution of
>   the hash.  There is nothing in the definition of ECMP to prohibit
>   this and the OMP technique explicitly called for doing so and
>   proposed protocol extension to be able to go beyond just a decision
>   within a single NE as Avici did.

Does above proposed change address (part of your comment)? If so, should
this state many or some? Other parts of your comment seem to be solution
oriented and should certainly be considered to meet issues raised in the
requirements or included in the proposed framework draft.

> 
> This is inaccurate:
> 
>   o 2.  It does not support a set of component links with different
>         characteristics (e.g., different bandwidth and/or latency).
> 
>       For example, in practice carriers commonly use link bandwidth
>       and link latency to set link TE metrics for RSVP-TE.  For
>       RSVP-TE, limiting the component links to same TE metric has the
>       practical effect of dis-allowing component links with different
>       link bandwidth and latencies.
> 
> There is no formal meaning to the link metric in ISIS or OSPF.

See proposed rewording above.

> 
> Under inverse-mux: the real problem with inverse-mux is the 
> amount of bandwidth that needs to be multiplexed greatly 
> exceeds the fastest single packet processing element and 
> therefore doesn't work.  

This is true (we may want to state this as the bandwidth of any
"connection" as defined in the draft instead of "packet processing
element"). We recommend adding this to the Inverse Multiplexing bullet
in the motivation section.

> The latency argument is not really valid.

Please elaborate.

> 
> I think that the ability of an LSR to measure latency on and 
> LSP and report a latency figure or route based on lowest 
> latency is almost orthogonal to the problem of composite 
> link.  If latency and bandwidth at each holding priority is 
> advertised, then we have a cross product of advertisements.  
> For example, you can have 1 Gb/s at 10msec at pri#1, but if 
> you can live with 12msec you can have 2Gb/s, or at 14 msec 
> 3Gb/s, but at pri#2 you only get ... and so on for 8 priorities.
> Is this what we're aiming for?

Advertising each link would solve the problem (see second bullet in
motivation section), but creates more advertisements in the IGP and may
not work well with LDP.

> 
> The table at the beginning of seciton 4 is meaningless.

Do you believe that the text above the table is sufficient to define
these terms, which are used in the outline? Amongst the co-authors
agreeing on the meaning of these terms used to structure the outline
based upon prior comments was viewed as useful. Do other members of the
rtgwg want to delete this table, refine the definitions, or propose
specific changes to the requirements outline? 

> 
> In this section:
> 
>   4.1.1.1. Traffic Flow and Connection Mapping
> 
>    The solution SHALL support operator assignment of traffic flows to
>    specific connections.
> 
>    The solution SHALL support operator assignment of connections to
>    specific component links.
> 
> How is this supposed to work for signaled LSP where the 
> component links are not idendified in control signaling?  Is 
> this scalable from a configuration standpoint or only 
> applicable to staticly configured MPLS cross connect?

A future solution could identify component links. The wg direction for
this draft was to focus on requirements. These comments may be
applicable to the framework draft or a specific proposed solution.

> 
>    In order to prevent packet loss, the solution must employ make-
>    before-break when a change in the mapping of a connection to a
>    component link mapping change has to occur.
> 
> Only the ingress of an LSP can initiate make-before-break and 
> the ingress doesn't know about the component links.  In 
> RFC3209, make-before-break involves a new LSP using the same 
> tunnel-id.
> Are you using a different meaning for make-before-break?
> 
Good point. This is more like MPLS FRR at an intermediate point where
the bypass tunnel is first established before switching. Will clarify
this point and provide appropriate reference(s).

> Regarding this statelent:
> 
>    The solution SHALL support management plane controlled parameters
>    that define at least a minimum bandwidth, maximum bandwidth,
>    preemption priority, and holding priority for each connection
>    without TE information (i.e., LDP signaled LSP that does not
>    contain the same information as an RSVP-TE signaled LSP).
> 
> Could you explain how preemption would work for LDP?  Do you 
> plane to withdraw the FEC?  If so, for how long?  Forever?  
> If not forever would the traffic periodically come back, get 
> remeasured and withdrawn again?

Good questions, but they seem solution oriented. The wg direction for
this draft was to focus on requirements. These comments may be
applicable to the framework draft or a specific proposed solution. Would
you propose a rewording, clarification of the requirement, and/or
additional requirements? 

> 
> In 4.2.2.1 what does this mean?
> 
>    o  Bandwidth of the highest and lowest speed

We discussed this and propose the following replacement:

o Maximum and minimum acceptable bandwidth of the LSP

> 
> Overall I find many of the stated requirements to be unclear. 
>  Perhaps some discussion and improvements to the wording will 
> bring clarity.

Achieving clarity is certainly our shared wg goal. Specific questions
and comments like those Curtis has provided would be most appreciated by
the co-authors.

> Or maybe I'm just dense.
> 
> Curtis
> _______________________________________________
> rtgwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtgwg
>

---

From: "Mcdysan, David E" <[email-masked]>
Date: Tue, 2 Mar 2010 09:25:08 -0500

Hi Curtis,

Andy will be sending #1 and I will be sending #2 and #3.

Thanks,

Dave 

> handling composite link, which should be acknowledged.  After 
> First issue is prior composite link work, both under the name 
> "composite link" and under other names.
> 
> A search of US trademarks yields:
> 
>   Word Mark    COMPOSITE LINKS
>   Goods and Services     (ABANDONED) IC 009. US 021 023 026 
> 036 038. G &
>   S: Computer software, namely, routing software for use in enabling
>   multiple links or fiber interfaces between routers to be 
> grouped into
>   a single logical connection
>   Standard Characters Claimed	
>   Mark Drawing Code   (4) STANDARD CHARACTER MARK
>   Serial Number	    78363042
>   Filing Date	    February 5, 2004
>   Current Filing Basis	     1B
>   Original Filing Basis	     1B
>   Published for Opposition     June 21, 2005
>   Owner	  (APPLICANT) Avici Systems, Inc. CORPORATION 
> DELAWARE 101
>   Billerica Avenue North Billerica MASSACHUSETTS 018621256
>   Attorney of Record     John L. DuPre'
>   Disclaimer  NO CLAIM IS MADE TO THE EXCLUSIVE RIGHT TO USE "LINKS"
>   APART FROM THE MARK AS SHOWN
>   Type of Mark   TRADEMARK
>   Register       PRINCIPAL
>   Live/Dead Indicator	DEAD
>   Abandonment Date	September 14, 2007
> 
> Since this may predate (and certainly overlaps) the ITU use 
> of the term "composite link", I think Avici should be 
> acknowledged if we continue to use the term composite link.
> 
> The definition is nice:
> 
>   Computer software, namely, routing software for use in enabling
>   multiple links or fiber interfaces between routers to be grouped
>   into a single logical connection.
> 
> Prior to the registered tradement, composite link was used by 
> Avici as a plain old trademark.  Although the registered 
> trademark does not predate Ethernet Link Aggregation, the 
> product and technique to which the registered trademark 
> applies does predate it (1998 vs circa 2000 for IEEE).  The 
> techniques are the same expect IEEE focused solely on 
> Ethernet, while Avici made use of extensions to PPP (as in POS).
> 
> In any case the aspect of the composite link dealt with in 
> IEEE 802.1x or Avici's composite link was the protocol to 
> negociate the binding.
> 
> 
>       approaches.  However, these approaches do not make use of MPLS
> therefore doesn't work.  The latency argument is not really valid.
>

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 03 Mar 2010 00:15:41 -0500

In message <[email-masked]>
> Andy will be sending #1 and I will be sending #2 and #3.

Sounds like a good plan.

Thanks,

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 03 Mar 2010 01:26:45 -0500

In message <[email-masked]>
> the reference to G.800 and replace it with some other text.

If I understand this definition of CL encompasses all existing LAG (or
non-Ethernet LAG like aggregation) and existing ECMP and existing
unequal mulltipath techniques.

If so, then the requirements here define yet another instance of CL as
defined by ITU.

You might want to acknowledge that there was a very similar prior
registered tradement meaning of CL that is now abandonned if for no
other reason to say "that's not what we mean".

> compared with the MPLS-based load balancing which is the focus of the

There are two things I'd like to see changed though I was not clear in
that I didn't provider suggested changes to text.  If you agree in
principle, then I can provide some text.

The two are:

  1.  Accurately characterize what exists today, what existing CL
      techniques have come before this, in use or not, and accurately
      characterize the common use cases of existing CL.

  2.  State as a requirement (we are at the requirement stage) that to
      the exent possible new CL capability will:

      1.  Continue to accommodate common use cases today, including an
          ability to carry IP traffic which MAY BE omitted in an
          implementation but MUST be accommodated, at least as an
          option, by any proposed solution.

      2.  Retain backward compatibility with existing MPLS/GMPLS LSR
          with no loss of existing capability, but possibly no gain in
          functionality if the legacy LSR is anywhere on the LSP path
          include as an LER.

If the characterization of existing CL gets too long it could be a
separate informational internet-draft that is referenced but I don't
think it will get that long.

> Could you provide a URL for the prior OMP work that we can add as

Did you want one that still works?  :-)

BTW- Its a part of your current employer that very abruptly shut down
the web site that had this and some other work on it (UUNET after I
left shut down engr.ans.net).  Some content was never recovered, but
this is just an aside.

Data tracker is probably the best reference.

https://datatracker.ietf.org/doc/draft-ietf-ospf-omp/
https://datatracker.ietf.org/doc/draft-ietf-isis-omp/
https://datatracker.ietf.org/doc/draft-villamizar-mpls-omp/

There are also IETF WG meeting minutes and lots of mailing list
archive discussion but no need to reference them.

> Remainder of orignal message snipped.

The scope of the draft being only MPLS is one of the big issues.  The
existing use cases for LAG/ECMP are:

  IP traffic:
    provider core networks:
      hash based on IP source and IP destination address
    provider non-core:
      hash based on IP source and IP destination address
      and sometimes UDP and TCP ports (but rarely)
    enterprise:
      hash based on IP source and IP destination and UDP or TCP port

  MPLS traffic:
    hash based on a subset of the MPLS stack
    size of the subset varies with vendors
    below some total stack depth (typically 8) the BOS label is included.
    number of labels included typically varies from 3-8
    less than 3 labels doesn't work well due to inadequate diversity
    using the top label very rarely (if ever) works well
    (VPN label, LDP label typically are not diverse enough)

What an implementation does with the hashed value varies:

  At one extreme is the simplicity of just doing a modulo.  This
  doesn't work if component links are not all the same bandwidth.

  A little better is a table lookup based on the hash which allows a
  more even split across component links that are not the same.

  Still better is a table lookup (or other implementation) that uses
  feedback to adjust load balance.  This include two subcategories:

    Feedback is internal to a single NE (Avici's CL is an example), is
    transparent to other NE, and requires no signaling changes.  Avici
    calls this CL but I've also heard the term Adaptive LAG somewhere.

    Feedback is external and requires signaling extensions (OMP is an
    example but was never implemented by an equipment vendor).

The existing use of Link Bundling for MPLS is:

  Maximum LSP Bandwidth is advertised.

  Accounting is on a per component basis.

  The RRO identifies both the component link and label and therefore
  cannot be changed.

This is very rough text (an outline really).  If you agree in
principle to add something I can clean it up.

The CL requirements should specify a delta based on what exists.  We
should also acknowledge a problem with LAG that there is no way to
signal characteristics of the LAG.  Having this a requirement makes it
easier to propose a document to fix this (though not necessarily the
same document that proposes a new, possible MPLS only CL).  All that
is really needed for LAG is a maximum allowed microflow size (size of
component link minus epsilon for adaptive LAG, a configured fraction
of component link size for simple LAG).  An LSP can then signal a
largest expected microflow.  For example, an LSP carrying nothing but
a set of 1GbE PW is not going to have a microflow larger than 1G.

The requirement document need not preclude proposing a CL type that is
MPLS only, but it should definitely not mandate it.

Do you agree in principle to these sort of changes?  If you at least
partially agree I'll write something more concrete that you could
consider.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 03 Mar 2010 03:08:47 -0500

In message <[email-masked]>
> 	#3. Proposed Resolution of Comments (Was: Composite Link

Hi Dave

Replies are inline.

>    	   2.  It only supports advertisement of a single TE metric for 

That is not the text.  The exact text is:

   The Link Bundle concept is somewhat limited because of the
   requirement that all component links must have identical
   capabilities, and because it applies only to TE links.  This document
   sets out a more generic set of requirements for grouping together a
   set of parallel data links that may have different characteristics,
   and for advertising and operating them as a single TE or non-TE link
   called a Composite Link.

There is no requirement "that all component links must have identical
capabilities" imposed by link bundling.

The introduction of new TLVs to carry metrics that allow multiple
combinations of delay and other parameters mixed in with admin color,
priority, etc is completely orthogonal to the use of link bundling.

What I'm asking is that you drop the link bundle bashing here (not
that I am at all a fan of link bundling) and just stick with the
requirement.

    This document proposes that a link metrics allow for grouping
    together a set of parallel data links that may have different
    characteristics, and for advertising and operating them as a
    single link.

Not that if link bundling wanted to advertise more than one delay the
definition of should allow it to make use of such a metric.

> Diffserv reference for review by the Working Group. 

Its a pain to find some of the relevant references but nothing grep
can't handle.

  rfc2474.txt:

   Microflow: a single instance of an application-to-application flow of
   packets which is identified by source address, destination address,
   protocol id, and source port, destination port (where applicable).

  rfc2475.txt:

   Microflow                 a single instance of an application-to-
                             application flow of packets which is
                             identified by source address, source port,
                             destination address, destination port and
                             protocol id.

   Traffic stream            an administratively significant set of one
                             or more microflows which traverse a path
                             segment.  A traffic stream may consist of
                             the set of active microflows which are
                             selected by a particular classifier.

An MPLS microflow is properly identified by its entire lable stack and
if the payload is IP, by the IP source and destination address.  If
not IP as indicated by a PW CW, then only the label stack is used.

You can also reference the "Avoiding ECMP" section in rfc4385.txt
which explains the reason to avoid certain values in the first nibble
of payload and rfc4928 which provides the following terms

   IP ECMP     A forwarding behavior in which the selection of the
               next-hop between equal cost routes is based on the
               header(s) of an IP packet

   Label ECMP  A forwarding behavior in which the selection of the
               next-hop between equal cost routes is based on the label
               stack of an MPLS packet

Note that with label ECMP the top label may be the same but diversity
elsewhere in the label stack allows microflows (or set of microflows)
to be identified.

> >       approaches.  However, ?many/some of? these approaches do not

How abot this change:

   o  ECMP/Hashing/LAG: IP traffic composed of a large number of flows
      with bandwidth that is small with respect to the individual link
      capacity can be handled relatively well using ECMP/LAG approaches.
-     However, these approaches do not make use of MPLS control plane
-     information nor traffic volume information.
+     While nothing precludes using traffic volume information, and
+     some implementations have done so, in practice few if any
+     implementations today make use of MPLS control plane information
+     or traffic volume information.  Implementations commonly use the
+     entire MPLS label stack for non-IP MPLS traffic.
      Distribution
      techniques applied only within the data plane can result in less
      than ideal load balancing across component links of a composite
      link.

It makes it more clear that no ECMP/Hashing/LAG can and has at least
in the past (and possibly will in the future) use traffic volume
information and also commonly use the entire IP stack.  There is
precedence for use of the whole IP stack and traffic volume
information (at least Avici).

> See proposed rewording above.

See my comment above.  Introduction of a new metric should be
orthogonal to CL and applicable to link bundling if done right.

> in the motivation section.

No they are different.  In principle a monster IPSEC GW or other
appliance could stuff a huge amount of traffic into the network which
appears as a single microflow.

The limitation is that at any hop in the network, there must be a
packet processing element big enough to swallow the whole flow and
spit it out striped over more than one component.

In practical terms, today there are 100Gb/s packet processor
elements.  There might be 200G or even 500G PP chips in the future,
but by that time there will already be a lot of Tb/s pipes and maybe
even Tb/s LSPs.

Network growth is outpacing Moore's Law.

> > The latency argument is not really valid.

Ths latency as a metric is a rough convention and the thinking usually
runs along the lines that the longer the fiber run the more dollars it
will cost so use delay rather than hop count for metric.  In practice
metric settings are used as a form of TE rather than an attempt to get
a minimum delay on traffic.

If a metric is invented that specifically means delay, it can be
invented with the semantics needed to handle link bundle, LAG, or CL.

> motivation section), but creates more advertisements in the IGP and may

So what exactly is the requirement, stated only as a requirement void
of any additional bashing about what there is today?

> specific changes to the requirements outline? 

I don't find it at all what the table was supposed to represent.  What
dow a Y mean at the intersection of "With TE Info" and "IGP"?

There is no apparent (to me at least) meaning to the "Y" or "N"
assigned at the intersection of row and colum heading.

> applicable to the framework draft or a specific proposed solution.

Isn't that the "unbundled" version of link bundling, or put another
way, the common practice on a set of parallel links where LAG is not
used and before link bunding existed?

If so, there is nothing new about that "future solution".

As to the existing requirement it appears then that this requirement
implies that there cannot be a control plane.  Is that the case?

> this point and provide appropriate reference(s).

FRR is only temporary and requires a resignal from the ingress to make
it permanent.  The "hint" to the ingress is that protection-inuse is
set in the RESV (and the IGP says the link is down).  There is nothing
to prevent that behaviour with FRR using another member except that
the set of SRLG might notice if it was another wave on the same fiber.

> you propose a rewording, clarification of the requirement, and/or

They are feasibility oriented questions.  If there is no feasible
solution, there shouldn't be a requirement.  I propose that the
requirement be removed until there is evidence that a feasible
solution exists.

> o Maximum and minimum acceptable bandwidth of the LSP

This is sneaking in a requirement with no semantics.  How does the
"Maximum and minimum acceptable bandwidth of the LSP" relate to the
reserved BW at a given priority level and what is it used for?

A requirement document should document what is to be accomplished.  By
including what parameters are to be added you are clearly in the realm
of how, and you haven't fully defined what the purpose of the
parameters are.

I think all of section 4.2 does not belong in a requirement document,
at least in its current form.

> and comments like those Curtis has provided would be most appreciated by

Its getting late so I'll start another thread.

> > Or maybe I'm just dense.

You didn't respond to that.  :-)

---

From: "Mcdysan, David E" <[email-masked]>
Date: Thu, 4 Mar 2010 07:40:47 -0500

Hi Curtis,

When Ning, Andy, Lucy and I first submitted this draft in October 2008,
there was discussion about including IP traffic. The discussion resulted
in a decision to address the MPLS focused case first. 

You are asking the wg to re-evaluate this decision. No one else has
responded to this thread and I would like to wait and hear from others
before making more detailed replies.

A few responses in line below.

Dave 

> Sent: Wednesday, March 03, 2010 1:27 AM
> To: Mcdysan, David E
> Cc: [email-masked]
> Subject: Re: Acknowledgement of Prior Work (Was: Composite 
> Link Requirements)
> 
> 
> In message 
> <[email-masked]
> .verizon.com>
> > respond with three separate messages, to separate the threads as 
> > follows so that all of the issues you raise can be resolved 
> efficiently.
> >  
> > > From: [email-masked] [mailto:[email-masked]] On 
> > > Behalf Of Curtis Villamizar
> > > This is in regards to the goals that are embodied in the RTGWG 
> > > acceptance of a draft to deal with requirements for 
> composite link, 
> > > currently named draft-ietf-rtgwg-cl-requirement-00.txt
> > > 
> > > 
> > > I'm bringing up two issues in this email.  One is prior composite 
> > > link work and the
> >  
> > > Note that ITU's G.800 does not define what a composite 
> link is and 
> > > only mentions composite four times in the document, 
> including use of 
> > > composite link and composite trail.  The figure indicates that a 
> > > composite link is "inverse multiplexing".  For this 
> reason, I don't 
> > > think G.800 should be referenced because its a big load 
> of **** with 
> > > only slight mention of CL.
> > > 
> >  
> > supported by a separated service layer trail. The composite link 
> > conveys communication ion using different server 
> layer trails 
> > thus the sequence of symbols cross these links may not be 
> preserved."
> >  
> > section 6.9.2. The text above is the first case. G.800 states that 
> > these are separate cases.
> >  
> > The text above Figure 16 in G.800 related to concatenated server 
> > trails may also be relevant (at least in the framework).
> >  
> If I understand this definition of CL encompasses all 
> existing LAG (or non-Ethernet LAG like aggregation) and 
> existing ECMP and existing unequal mulltipath techniques.

In G.800 yes, in terms of the draft the intent was to limit the scope to
MPLS-labelled flows.

> 
> If so, then the requirements here define yet another instance 
> of CL as defined by ITU.
> 
> You might want to acknowledge that there was a very similar 
> prior registered tradement meaning of CL that is now 
> abandonned if for no other reason to say "that's not what we mean".
> 
> > Text Snipped
> >  
> > > 
> > > different paths (trails in ITU speak but this is IETF 
> where we say 
> > > path), or two parallel links, this has been called ECMP 
> in IETF (and 
> > > elsewhere) for two decades or more.  Both ISIS and OSPF 
> use the term 
> > > ECMP.  The techniques used for ECMP load balance was discussed on 
> > > IETF lists quite a bit in the early to mid-1990s.  The three 
> > > techniques applied to IP networks (in the terminology of 
> that time) 
> > > were:
> > > 
> > > The second is applicable only to parallel links.  Using larger 
> > > chunks it is also the technique used in MPPP (multilink 
> PPP).  MPPP 
> > > is also sometimes abbreviated PPP-ML, though not in the 
> RFC.  MPPP 
> > > is no longer of much interest as it was only applied to low speed 
> > > links.
> > > 
> > > The per packet load balance caused packet reorder and a 
> great deal 
> > > of grief for service providers, hence the abundance of discussion 
> > > within IETF at the time.  The use of IP src/dst hash, while 
> > > widespread and widely discussed, did not get documented in an RFC 
> > > until Chris Hopps and Dave Thaler wrote RFC 2991 
> "Multipath Issues 
> > > in Unicast and Multicast Next-Hop Selection" and RFC 2992 
> "Analysis 
> > > of an Equal-Cost Multi-Path Algorithm" in November 2000.  
> (at least 
> > > AFAIK).
> > > 
> > > The IP src/dst technique itself is beleived to have originated in 
> > > the T1-NSFNET, which puts its use back to circa 1987.
> > > 
> > > The OMP work predates RFC 2991 and RFC 2992 but never 
> made it past 
> > > the internet-draft stage.  In that work the use of 
> src/dst hash and 
> > > the use of adaptive algorithms with src/dst hash is 
> discussed.  On 
> > > the IETF mailing lists even methods of implementation were 
> > > discussed, table based and parallel sets of comparator 
> pairs (TCAM 
> > > like).
> > > 
> > > Circa 2000 there was a lot of discussion of the use of the MPLS 
> > > label stack to provide the entropy for ECMP vs looking past the 
> > > label stack at the IP payload.  Today's PW control word 
> acknowledges 
> > > this common practice and avoids it for PW, but the fat-pw aka 
> > > entropy label puts better entropy back into PW.
> > > 
> > > In practice today, all core hardware uses the same IP 
> src/dst hash 
> > > to provide a load balance for ECMP and LAG.
> > > 
> > > 
> > > The existing internet-draft acknowledges link bundling, 
> but does not 
> > > accurately characterize ECMP and LAG and the src/dst hash 
> technicque 
> > > used by both, nor does it acknowledge the prior OMP work.
> > > 
> >  
> > you have more specific suggestions on the text in the draft 
> and that 
> > is where we propose specific changes to address your 
> comments. Another 
> There are two things I'd like to see changed though I was not 
> clear in that I didn't provider suggested changes to text.  
> If you agree in principle, then I can provide some text.
> 
> The two are:
> 

The amount of detail here needed depends on the scope decision. Accuracy
is of course important.

>           option, by any proposed solution.

Dependent upon a wg agreement to increase scope to include IP.

>           functionality if the legacy LSR is anywhere on the LSP path

IMHO this is desirable (SHOULD) but not mandatory (SHALL).

> 
> If the characterization of existing CL gets too long it could 
> be a separate informational internet-draft that is referenced 
> but I don't think it will get that long.

Editorially, I like this suggestion. The current draft is on the verge
of being too long already. If an agreement to increase the scope is made
by the wg then I think this should be done.

> BTW- Its a part of your current employer that very abruptly 
> shut down the web site that had this and some other work on 
> it (UUNET after I left shut down engr.ans.net).  Some content 
> was never recovered, but this is just an aside.

The world is ever changing. Many things are not as they once were. :)

> There are also IETF WG meeting minutes and lots of mailing 
> list archive discussion but no need to reference them.

Thanks. For the draft could you also please provide a few sentences as
to why this was never standardized by the IETF.

> 
> > Also, the scope of the draft is MPLS, which is not covered 
> > specifically in the points above.
> >  
> The scope of the draft being only MPLS is one of the big 
> issues.  The existing use cases for LAG/ECMP are:
> 
>   IP traffic:
>     provider core networks:

The above seems to be text that would better suited for a separate
informational draft that summarizes prior approaches.

> 
> The CL requirements should specify a delta based on what 
> exists.  We should also acknowledge a problem with LAG that 
> there is no way to signal characteristics of the LAG.  Having 
> this a requirement makes it easier to propose a document to 
> fix this (though not necessarily the same document that 
> proposes a new, possible MPLS only CL).  All that is really 
> needed for LAG is a maximum allowed microflow size (size of 
> component link minus epsilon for adaptive LAG, a configured 
> fraction of component link size for simple LAG).  An LSP can 
> then signal a largest expected microflow.  For example, an 
> LSP carrying nothing but a set of 1GbE PW is not going to 
> have a microflow larger than 1G.

This appears to be a good example of where there could be commonality of
solution between MPLS and IP. 

> 
> The requirement document need not preclude proposing a CL 
> type that is MPLS only, but it should definitely not mandate it.

This is related to the scope decision.

> 
> Do you agree in principle to these sort of changes?  If you 
> at least partially agree I'll write something more concrete 
> that you could consider.
> 
> Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 05 Mar 2010 01:14:32 -0500

In message <[email-masked]>
> A few responses in line below.

Dave,

You can propose a solution that supports MPLS only, but the
requirements should reflect IP as well as MPLS.  It would not be hard
to put wording in the requirements that allows for solutions that
support only MPLS or solutions that support both.

It is clear that supporting both would be better with allowance for
some network equipment that supports only MPLS.  It is also clear that
backwards compatibility is desireable.  Therefore these things should
be in the requirements.

Someone else might propose a solution that support IP and MPLS
perfectly well and is only a minor change to what is already deployed
and meets all other requirements.

Curtis

BTW- it only makes it harder if you try to put a preconcieved solution
into the requirements and doing so increases the chance of not getting
the best solution out of the process.