Subject: Re: [OPSAWG] WG last call for "Mechanisms for Optimal LAG/ECMP Component Link Utilization in Networks"

From: "Andrew G. Malis" <[email-masked]>
Date: Tue, 3 Sep 2013 13:28:12 -0400

I have provided comments on previous revisions of this draft and believe
that it is now ready for publication.

Thanks,
Andy

On Sat, Aug 24, 2013 at 4:07 PM, Melinda Shore <[email-masked]>wrote:

> This is to announce the start of working group last call on:
>
>     Mechanisms for Optimal LAG/ECMP Component Link Utilization in
>         Networks
>
>
> http://datatracker.ietf.org/doc/draft-ietf-opsawg-large-flow-load-balancing/
>
> It is intended for publication as an informational RFC.
>
> Please give it a careful read and provide any feedback to this
> mailing list by September 9, 2013
>
> Thanks,
>
> opsawg chairs
> _______________________________________________
> OPSAWG mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/opsawg
>

---

From: pete <[email-masked]>
Date: Tue, 3 Sep 2013 10:34:37 -0700

I support this document, as these mechanisms are needed in SP and Research
and Education Networks (RENs).

I have a few minor comments:

1-

4.2. Overview of the mechanism

...

   Step 2) The egress component links are periodically scanned for
link utilization. If the egress component link utilization exceeds a
pre-programmed threshold, an operator alert is generated. The large
flows mapped to the congested egress component link are exported to a
central management entity

I believe you mean to say that the information about the large flows
is exported, rather than the large flows themselves being exported.

2 -

Providing large flow information to a central management entity
provides the capability to further optimize flow distribution at with
multi-node visibility.

The last part of that sentence needs clarification.

Thanks.

pete

---

From: ramki Krishnan <[email-masked]>
Date: Tue, 3 Sep 2013 10:49:21 -0700

Hi Pete,

Thanks a lot for your comments. We will address these.

Thanks,
Ramki

From: [email-masked] [mailto:[email-masked]] On Behalf Of pete
Sent: Tuesday, September 03, 2013 10:35 AM
To: Melinda Shore
Cc: [email-masked]; [email-masked]
Subject: Re: [OPSAWG] WG last call for "Mechanisms for Optimal LAG/ECMP Component Link Utilization in Networks"

I support this document, as these mechanisms are needed in SP and Research and Education Networks (RENs).

1-

...

   Step 2) The egress component links are periodically scanned for link utilization. If the egress component link utilization exceeds a pre-programmed threshold, an operator alert is generated. The large flows mapped to the congested egress component link are exported to a central management entity

I believe you mean to say that the information about the large flows is exported, rather than the large flows themselves being exported.

2 -

Providing large flow information to a central management entity provides the capability to further optimize flow distribution at with multi-node visibility.

Thanks.

pete

On Sat, Aug 24, 2013 at 1:07 PM, Melinda Shore <[email-masked]<mailto:[email-masked]>> wrote:
This is to announce the start of working group last call on:

Thanks,

opsawg chairs
_______________________________________________
OPSAWG mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/opsawg

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Tue, 10 Sep 2013 10:13:20 +0000

Hi,

I apologize for the one day delay in sending this review. Vacation interfered. 

I read the document and I have a basic unclarity. Hopefully it is just me, and maybe some clarification text and references to be added. 

What does this document exactly refer to when saying 'LAG/ECMP techniques'? Is this either LAG (layer 2) or ECMP, or a combination of the two, or all of these? There is no reference at all in the document for LAG, no solid reference for ECMP (2992 only refers to 'an ECMP algorithm'). 

I also had a hard time finding the sFlow references. In desperation I went to wiki http://en.wikipedia.org/wiki/SFlow which kindly informed me that 

> sFlow is a technology for monitoring network,[1] wireless[2] and host[3] devices. The sFlow.org consortium[4] is the authoritative source for the sFlow protocol specifications.[5] sFlow version 5 is the current version of sFlow. Previous versions of sFlow, including RFC 3176, have been deprecated.[6]

This seems problematic. RFC 3176 has no mark of being 'deprecated' in IETF terms anyway. 

Regards,

Dan

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Melinda Shore
> Sent: Saturday, August 24, 2013 11:07 PM
> To: [email-masked]
> Cc: [email-masked]
> Subject: [OPSAWG] WG last call for "Mechanisms for Optimal LAG/ECMP
> Component Link Utilization in Networks"
> 
> Please give it a careful read and provide any feedback to this mailing
> list by September 9, 2013
> 
> Thanks,
> 
> opsawg chairs

---

From: Anoop Ghanwani <[email-masked]>
Date: Tue, 10 Sep 2013 08:30:41 -0700

Hi Dan,

Thanks for the careful review.  Please see inline.

On Tue, Sep 10, 2013 at 3:13 AM, Romascanu, Dan (Dan) <[email-masked]>wrote:

> I read the document and I have a basic unclarity. Hopefully it is just me,
> and maybe some clarification text and references to be added.
>
> What does this document exactly refer to when saying 'LAG/ECMP
> techniques'? Is this either LAG (layer 2) or ECMP, or a combination of the
> two, or all of these? There is no reference at all in the document for LAG,
> no solid reference for ECMP (2992 only refers to 'an ECMP algorithm').
>

It refers to all of them, i.e. LAG, ECMP, and even the (hierarchical)
combination.  We can try and provide some clarification around that.  We
will add 802.1AX as a reference for LAG.  What would be a reasonable
reference for ECMP?  Would RFC 2991 suffice?

>
> I also had a hard time finding the sFlow references. In desperation I went
> to wiki http://en.wikipedia.org/wiki/SFlow which kindly informed me that
>
> > sFlow is a technology for monitoring network,[1] wireless[2] and host[3]
> devices. The sFlow.org consortium[4] is the authoritative source for the
> sFlow protocol specifications.[5] sFlow version 5 is the current version of
> sFlow. Previous versions of sFlow, including RFC 3176, have been
> deprecated.[6]
>

The reference that we have for sflow in the references section is the
"sflow Version 5" specification which is located at
http://sflow.org/sflow_version_5.txt.  We can update the reference to
include the URL.

Anoop

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Tue, 10 Sep 2013 15:39:26 +0000

Hi Anoop,

Thanks for the quick response and for the clarification. Clarification text and better references as you suggest would be welcome.

I am not sure RFC 2991 is a solid reference for ECMP. Maybe you should ask Dave Thaler.

There is another problem that this document prompts to. If the protocol described by RFC 3176 is indeed 'deprecated' the RFC should be made Historical. That's an issue for the IESG to consider.

Regards,

Dan

From: [email-masked] [mailto:[email-masked]] On Behalf Of Anoop Ghanwani
Sent: Tuesday, September 10, 2013 6:31 PM
To: Romascanu, Dan (Dan)
Cc: Melinda Shore; [email-masked]; [email-masked]
Subject: Re: [OPSAWG] WG last call for "Mechanisms for Optimal LAG/ECMP Component Link Utilization in Networks"

Hi Dan,

On Tue, Sep 10, 2013 at 3:13 AM, Romascanu, Dan (Dan) <[email-masked]<mailto:[email-masked]>> wrote:
Hi,

It refers to all of them, i.e. LAG, ECMP, and even the (hierarchical) combination.  We can try and provide some clarification around that.  We will add 802.1AX as a reference for LAG.  What would be a reasonable reference for ECMP?  Would RFC 2991 suffice?

The reference that we have for sflow in the references section is the "sflow Version 5" specification which is located at http://sflow.org/sflow_version_5.txt.  We can update the reference to include the URL.

Anoop

> From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf
> Of Melinda Shore
> To: [email-masked]<mailto:[email-masked]>
> Cc: [email-masked]<mailto:[email-masked]>

---

From: Melinda Shore <[email-masked]>
Date: Tue, 10 Sep 2013 09:46:16 -0800

One issue that's been raised off-list is that there may not be
sufficient visibility into the per-circuit queues for this to
be implementable.  Does one of the authors have any feedback
on this?

Melinda

---

From: Anoop Ghanwani <[email-masked]>
Date: Tue, 10 Sep 2013 10:56:11 -0700

Melinda,

I would need a bit more clarity on what is "per circuit".  The mechanism
works by looking at the utilization on individual component links within a
LAG/ECMP.  The port-level queues and packet/byte counters are always
visible to an implementation.

It is possible that some implementations may not be able to control the
flow of traffic in the way that is proposed in the draft (i.e. mapping
specific large flows to an individual component link), but I'm not aware of
any in recent memory (5+ years).

Anoop

---

From: Randy Bush <[email-masked]>
Date: Tue, 10 Sep 2013 13:58:53 -1000

> I would need a bit more clarity on what is "per circuit".  The
> mechanism works by looking at the utilization on individual component
> links within a LAG/ECMP.  The port-level queues and packet/byte
> counters are always visible to an implementation.

for some definitions of 'always'.  a few of us would greatly appreciate
if you could tell us the commands on junos and cisco XR to look at a
lag's per-physical-circuit queues.

randy

---

From: Anoop Ghanwani <[email-masked]>
Date: Tue, 10 Sep 2013 17:08:25 -0700

Randy,

Please clarify what you mean by per-physical circuit queues.  There is no
mention of that term in this draft.

We deal with per physical port counters.  Finding those should be pretty
straightforward on any router.  Please see section 5.6.1:
http://tools.ietf.org/html/draft-ietf-opsawg-large-flow-load-balancing-05#section-5.6.1
for what the draft needs.  There are standard MIB objects for things like
these.

Anoop

---

From: joel jaeggli <[email-masked]>
Date: Tue, 10 Sep 2013 20:54:21 -0700

> lag's per-physical-circuit queues.

In a distributed forwarding architecture, individual linecards or indeed
routers in the case of various MCLAG implementations may have little
timely insight into the port level queues on all lag members.

---

From: Anoop Ghanwani <[email-masked]>
Date: Tue, 10 Sep 2013 22:34:19 -0700

On Tue, Sep 10, 2013 at 8:54 PM, joel jaeggli <[email-masked]> wrote:

> Joel,

One of the tools we discuss in the draft is sflow.  sflow is capable of
providing network-wide statistics within the time intervals we are talking
about (on the order of seconds).  And this is sufficient for the problem we
are trying to solve.  In fact, trying to solve it on fine timescales may be
counterproductive.

But you do bring up a good issue with MCLAG.  We have not tried to address
that in this document.  Lets take a simple example.  If we have a 2-chassis
system and we have an MCLAG to these two routers, typically whichever
router receives the traffic that is destined for the MCLAG will place it on
its local link.  So really, this becomes a fairly complex global network
optimization issue where one would have to make sure that the equal amounts
of traffic hits each of the routers. It may be possible to solve it by
identifying large flows globally and using PBR entries to ensure they are
routed to the desired router so that it exits on the desired component
link.  If you think it is worth adding some text, we can do that.

Anoop

---

From: Warren Kumari <[email-masked]>
Date: Wed, 11 Sep 2013 10:04:07 -0400

> lag's per-physical-circuit queues.

Yes, yes we would.
A  large number of implementations provide almost no visibility into the interface queues (even if you throw your toys out the cot, threaten to change vendors, etc), regardless of if the interface is part of LAG or not…

However, I don't think that the draft needs queue level information -- interface counters is sufficient. Or am I missing something?

W

--
"When it comes to glittering objects, wizards have all the taste and self-control of a deranged magpie."
-- Terry Pratchett

---

From: ramki Krishnan <[email-masked]>
Date: Wed, 11 Sep 2013 07:53:45 -0700

>> However, I don't think that the draft needs queue level information -- interface counters is sufficient. Or am I missing something?

Thanks Warren -- you said it right. For this draft, we need only interface counters.

Thanks,
Ramki

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Warren Kumari
Sent: Wednesday, September 11, 2013 7:04 AM
To: Randy Bush
Cc: [email-masked]; [email-masked]
Subject: Re: [OPSAWG] WG last call for "Mechanisms for Optimal LAG/ECMP Component Link Utilization in Networks"

> lag's per-physical-circuit queues.

Yes, yes we would.
A  large number of implementations provide almost no visibility into the interface queues (even if you throw your toys out the cot, threaten to change vendors, etc), regardless of if the interface is part of LAG or not...

W

---

From: "George, Wes" <[email-masked]>
Date: Mon, 16 Sep 2013 17:01:54 -0400

My apologies for missing the deadline on this, but I got my wires crossed and reviewed this doc in response to something I saw on IETF LC (which was actually on a different draft), so I figured the least I can do is provide my feedback on the *right* draft/thread/list, and then swear off email and go home for the day before I can do any more damage... sigh.
--------
one substantive comment:

I think within the operational considerations (and possibly the info model), you need some discussion of diagnostics and troubleshooting, both for on-box and off-box implementations. How do I see that it's working properly, and how do I diagnose problems when it's not?
One of the problems with the existing hashing algorithms is that they are often opaque, such that it's not clear what the device is doing, whether the hashing is working properly and the flows are of the sort that create imbalanced distribution, or whether hashing has broken somehow -- occasionally you can get info, but it's usually hidden commands, with difficult-to-interpret responses, and it's not like most vendors publish their "secret sauce" optimizations of hashing so that it's easy to predict what will happen given a certain set of flows.
In order for this to be operationally manageable, especially in the case of on-router processing of this rebalancing, there has to be an easy way for the operator to access the information about what's happening - what the result would be if the flows were balanced according to the hash vs what is happening as a result of rebalancing, so that they can chase down things like rebalancing misses or situations where this local optimization is creating a problem elsewhere in the path because that device did something different in its attempts to balance better, etc. It may also be that this info is necessary to properly tune the frequency of sampling, the thresholds for things like long-lived vs short-lived flows, etc. to the specific network where it is being used.
I realize that in the model you've proposed, we're somewhat limited because this is using sampled flow data instead of the realtime packet hash. It may be that this drives a requirement for the granularity of data being brought into the system in the external mode, and some requirements about the level of information available via the UI (or SNMP or XML or whatever) in the automatic hardware-based mode.

Thanks
Wes George

> Sent: Saturday, August 24, 2013 4:07 PM
> To: [email-masked]

This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.

---

From: ramki Krishnan <[email-masked]>
Date: Tue, 17 Sep 2013 18:47:24 -0700

Hi Wes,

Thanks for your comments. Please find responses below.

>> In order for this to be operationally manageable, especially in the case of on-router processing of this rebalancing, there has to be an easy way for the operator to access the information about w…
 
For determining component link imbalance, we are using interface counters.  For tracking the effectiveness of the scheme, we also have other monitoring information such as 1) Number of times rebalancing was done and 2) Time since the last rebalancing event. Is there something else that you think would be helpful?
 
>> I realize that in the model you've proposed, we're somewhat limited because this is using sampled flow data instead of the realtime packet hash. It may be that this drives a requirement for the gr…
 
For large flow detection, we propose two modes 1) Sampling Techniques 2) Automatic Hardware Recognition. The primary difference between the two modes in terms of performance is the time taken to recognize the long-lived large flows; there is no loss of accuracy in either mode. In the automatic hardware based mode for detecting the large flows, it is best that the load balancing happen in the router. Are you looking for anything else ?

Thanks,
Ramki

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of George, Wes
Sent: Monday, September 16, 2013 2:02 PM
To: Melinda Shore; [email-masked]
Cc: [email-masked]
Subject: Re: [OPSAWG] WG last call for "Mechanisms for Optimal LAG/ECMP Component Link Utilization in Networks"

Thanks
Wes George

---

From: "George, Wes" <[email-masked]>
Date: Wed, 18 Sep 2013 10:43:00 -0400

> From: ramki Krishnan [mailto:[email-masked]]
>
>
> For determining component link imbalance, we are using interface
> counters.  For tracking the effectiveness of the scheme, we also have
> other monitoring information such as 1) Number of times rebalancing was
> done and 2) Time since the last rebalancing event. Is there something
> else that you think would be helpful?
[WEG] yes. You need info about the number of flows being tracked on each interface and their size, possibly also the category (long-lived vs short, large vs small) as well as which flows have been moved and when. You're already building all of this state to provide input to your algorithm, I'm just saying that you need to make it clear in the draft that operators may want to be able to see that data. Like I said, in order to evaluate this as actually being better than the existing hash, to tune the parameters, and to troubleshoot balancing problems, I need to be able to see what it's actually doing, not just the interface counters.

>
> >> I realize that in the model you've proposed, we're somewhat limited
> because this is using sampled flow data instead of the realtime packet
> hash. It may be that this drives a requirement for the granularity of
> data being brought into the system in the external mode, and some
> requirements about the level of information available via the UI (or
> SNMP or XML or whatever) in the automatic hardware-based mode.
>
> For large flow detection, we propose two modes 1) Sampling Techniques 2)
> Automatic Hardware Recognition. The primary difference between the two
> modes in terms of performance is the time taken to recognize the long-
> lived large flows; there is no loss of accuracy in either mode. In the
> automatic hardware based mode for detecting the large flows, it is best
> that the load balancing happen in the router. Are you looking for
> anything else ?
[WEG] Yes, I think maybe I was unclear. I'm looking for diagnostic tools to make this less opaque for an operator. I was noting that it may be difficult to give specific, granular information by flow about the size of each flow and when/if it was moved by the loadbalancing algorithm if you were reliant on sampled flow data, because it wouldn't be complete/granular enough -- it would only show a subset of the flows, not all flows.

Thanks
Wes

---

From: ramki Krishnan <[email-masked]>
Date: Fri, 20 Sep 2013 15:51:32 -0700

Thanks Wes, please see inline.

Thanks,
Ramki
-----Original Message-----
From: George, Wes [mailto:[email-masked]] 
Sent: Wednesday, September 18, 2013 7:43 AM
To: ramki Krishnan; [email-masked]
Subject: RE: [OPSAWG] WG last call for "Mechanisms for Optimal LAG/ECMP Component Link Utilization in Networks"

> other monitoring information such as 1) Number of times rebalancing 
> was done and 2) Time since the last rebalancing event. Is there 
> something else that you think would be helpful?
[ramki] will do.

> For large flow detection, we propose two modes 1) Sampling Techniques 
> 2) Automatic Hardware Recognition. The primary difference between the 
> two modes in terms of performance is the time taken to recognize the 
> long- lived large flows; there is no loss of accuracy in either mode. 
> In the automatic hardware based mode for detecting the large flows, it 
> is best that the load balancing happen in the router. Are you looking 
> for anything else ?
[WEG] Yes, I think maybe I was unclear. I'm looking for diagnostic tools to make this less opaque for an operator. I was noting that it may be difficult to give specific, granular information by flow about the size of each flow and when/if it was moved by the loadbalancing algorithm if you were reliant on sampled flow data, because it wouldn't be complete/granular enough -- it would only show a subset of the flows, not all flows.
[ramki] In this case, large flows are a significant percentage of the link bandwidth. We don't see any issue in tracking such large flows through sampling technologies like sFlow.

Thanks
Wes