Subject: Re: FW: I-D Action: draft-retana-rtgwg-eacp-01.txt

From: Curtis Villamizar <[email-masked]>
Date: Thu, 21 Mar 2013 21:50:24 -0400

Alvaro,

I'm not sure what it means to put a OTN interface in a "low power
state".  OTN and SONET interfaces are a continuous stream of bits at a
constant rate dictated by the ITU-T.  The rate is not adjustable.  It
doesn't save any power by sending GFP null frames over OTN links.

Since most long haul router to router links are Ethenet across the
room, followed by long stretches of OTN, there may be no significant
benefit to putting the short Ethernet into lower power mode.  Plus as
discussed at IETF-86, the Ethernet low power mode applies only to
copper Ethernet (10/100 Mb/s, 1 Gb/s copper) and not to the fiber
forms of Ethernt used by providers.

So before anyone writes any more problem statements, requirements,
frameworks, or complete solutions for energy aware networks, we need
to have some research and some hard facts about where energy goes and
where there are opportunities to save energy.  We also should be
asking whether providers and high end network equipment vendors see a
need for any protocol work in this area.  So far the answer is a
resounding "no".

It is also worth pointing out that research so far has indicated that
the greatest total savings is the furthest down the food chain at or
near the customer network.  So the low power Ethernet applied to the
10/100/1000 copper used in homes and small business makes a lot of
sense.  IETF may not be the right place to attack the next layer up
the food chain, things like CMTS to CM (cable modem termination
systems to cable modem) or DSL efficiencies.

Curtis

In message <[email-masked]>
"Alvaro Retana (aretana)" writes:

[Chair hat off.]

Hi!

We just published an update (a refresh really) of the draft below.

The draft is NOT a set of requirements to be solved in the realm of power
aware/energy efficient networking.  We are not trying to define
requirements on how to save energy, how much energy should be saved, how,
when or where.  Nor are we recommending any specific solution.

So, what is it? ;-)   The intent of the draft is to "provide the tools and
knowledge necessary for protocol designers to modify network protocols to
best balance efficiency against performance, and to provide the background
   information network operators will need to intelligently deploy and use
protocol modifications to network protocols."

In other words..  We recognize that saving energy may be in many people's
minds, but we also recognize that networks have been designed/deployed
with a set of business (availability, ability to satisfy customer SLAs,
for example) and application (min bw, jitter/delay budgets, for instance)
requirements in mind, which need to still be satisfied even while saving
energy.

The draft goes on to evaluate considerations derived from the
business/application requirements that may be affected by applying common
methods of reducing energy (reducing the topology or reducing the link
speed, for example):  bw reduction, stretch, fast recovery, introducing
jitter and other operational aspects.

As an example, we mention (section 5.1) that eliminating links from the
topology (shutting them down or putting them in a "low power state")
should result in lower energy utilization..but that it will also affect
the total available bw in the network.  The requirement then comes in in
the form of "Modifications to control plane protocols to achieve network
energy efficiency SHOULD provide the ability to set the minimal bandwidth,
   jitter, and delay through the network".  I'll leave you to read some of
the other examples/considerations and requirements.

The end result of the draft is the definition of a framework against which
we can evaluate "the tradeoffs between modifications made to network
protocols to conserve energy and network performance metrics and
requirements".

Thanks!

Alvaro.

On 2/13/13 10:18 AM, "[email-masked]" <[email-masked]>
wrote:

>
>A New Internet-Draft is available from the on-line Internet-Drafts
>directories.
>
>
>	Title           : A Framework and Requirements for Energy Aware Control
>Planes
>	Author(s)       : Alvaro Retana
>                          Russ White
>                          Manuel Paul
>	Filename        : draft-retana-rtgwg-eacp-01.txt
>	Pages           : 16
>	Date            : 2013-02-13
>
>Abstract:
>   There has been, for several years, a rising concern over the energy
>   usage of large scale networks.  This concern is strongly focused on
>   campus, data center, and other highly concentrated deployments of
>   network infrastructure.  Given the steadily increasing demand for
>   higher network speeds, always-on service models, and ubiquitous
>   network coverage, it is also of growing importance for
>   telecommunication networks both local and wide area in scope.  One of
>   the issues in moving forward to reduce energy usage is to ensure that
>   the network can still meet the performance specifications required to
>   support the applications running over it.
>
>   This document provides an overview of the various areas of concern in
>   the interaction between network performance and efforts at energy
>   aware control planes, as a guide for those working on modifying
>   current control planes or designing new control planes to improve the
>   energy efficiency of high density, highly complex, network
>   deployments.
>
>
>The IETF datatracker status page for this draft is:
>https://datatracker.ietf.org/doc/draft-retana-rtgwg-eacp
>
>There's also a htmlized version available at:
>http://tools.ietf.org/html/draft-retana-rtgwg-eacp-01
>
>A diff from the previous version is available at:
>http://www.ietf.org/rfcdiff?url2=draft-retana-rtgwg-eacp-01
>
>
>Internet-Drafts are also available by anonymous FTP at:
>ftp://ftp.ietf.org/internet-drafts/
>
>_______________________________________________
>I-D-Announce mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/i-d-announce
>Internet-Draft directories: http://www.ietf.org/shadow.html
>or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

_______________________________________________
rtgwg mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/rtgwg

---

From: Russ White <[email-masked]>
Date: Sat, 23 Mar 2013 10:37:23 -0400

> need for any protocol work in this area.  So far the answer is a

I actually disagree...

The point of this draft is to guide the research into what would make
sense and what wouldn't, not to make a definitive statement. I
understand the perspective that, "if it doesn't save x amount of power,
it's not worth doing..."

But technology changes, after all, and in just a few short years the
answer to that question might well change. So, if we come to the point
where the answer changes, how have we defined the question?

This draft is about defining the questions that need to be asked, not
about doing work in answering those questions.

Different topic, IMHO.

:-)

Russ

---

From: Curtis Villamizar <[email-masked]>
Date: Sat, 23 Mar 2013 12:23:42 -0400

> about doing work in answering those questions.

Russ,

If our starting point is total cluelessness, then defining a taxonomy
of the solution space would be very counterproductive.

Stretch and microsleeps are solution space and I'd argue that the best
solutions for core may involve neither.  There is no highly credible
evidence that either has an impact on core network energy usage and
IMO stronger arguments that it would have no effect or infeasible
(microsleeping finely tuned temperature compensated colored lasers?)
than the the very weak arguments (mostly speculation) that there is
benefits to these two approaches for core.

So the research happen where it belongs - elsewhere.  Then when there
is credible research presented, it might be time to consider a
taxonomy of the problem and solution space in an SDO.

Curtis

---

From: Russ White <[email-masked]>
Date: Sat, 23 Mar 2013 13:45:02 -0400

> benefits to these two approaches for core.

No, stretch and sleep states are not "in the solution space." If you're
going to save energy by using sleep states, then you need to deal with
state. These are bars any given design must overcome, not suggestions
for a design that would work.

> taxonomy of the problem and solution space in an SDO.

So your basic argument is --"we should only consider this when it's
proven to be economically viable." My counter to that is simple:

There's more to this game than saving more than x$/year. There's also
the impact on the control plane to consider as a "cost."

This draft is an attempt to at least point those costs out in a
reasonable and understandable way. In other words, it's trying to get
all the requirements on the table, rather than focusing myopically on
the economic costs.

Again, it's answering a completely different question than you're asking.

Russ

---

From: Mingui Zhang <[email-masked]>
Date: Mon, 25 Mar 2013 02:58:40 +0000

Hi Curtis,

Please find my comments in-line below.

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On Behalf Of
>Curtis Villamizar
>Sent: Friday, March 22, 2013 9:50 AM
>To: Alvaro Retana (aretana)
>Cc: [email-masked]
>Subject: Re: FW: I-D Action: draft-retana-rtgwg-eacp-01.txt
>
>
>Alvaro,
>

The methodology from my side is to put those optical components in a manageable DOWN state (sleeping). If protocol work creates the opportunity for them to stay in the sleeping state for seconds or minutes, energy saving can be significant.
Fiber forms for Ethernet does not support micro-sleeping mode, but they may support deep-sleeping mode. PHYs can negotiate a longer time to stay in deep-sleeping mode.

>where there are opportunities to save energy.  

I agree that we should let numbers tell the truth. According to our observation, line-cards are the most power hungry part of high-end routers and switches. They can consume nearly 70% of the total power. Meanwhile, idle routers/switches consume more than 80% of their peak power. If we can shut-down components on line-cards or whole line-cards, we can save a considerable amount of energy.

This observation treats a router/switch as a black box. Our next step is to open this black box and see what further meaningful numbers we will get.

>We also should be

Energy Efficient Ethernet (EEE) involves a great deal of protocol work. Network equipments supporting EEE have already been shipped around by vendors. Providers may not resist solutions that can save their OPEX, only if vendors can come up with workable ones.

>systems to cable modem) or DSL efficiencies.

Power/Energy Aware NETwork needs the combination of "hardware work" and "protocol work". Protocol work should make use of the hardware characteristics. IOW, protocol work should create the opportunity for these techniques to save most energy. 

Energy saving for composite links may become another low-hanging fruit. As you pointed out, the traffic aggregation onto component links may impact the cooling system. We can have a test to see what will happen, whether we should have some limit on the aggregation.

Thanks,
Mingui

>
>Curtis
>
>
>In message
><[email-masked]>

---

From: Curtis Villamizar <[email-masked]>
Date: Mon, 25 Mar 2013 21:42:59 -0400

Russ,

This isn't a question of whether routing based techniques are
economically viable.  It is a question of whether there is any
real validity at all in the assertion that routing techniques aimed at
reducing power will in fact reduce power at all.

First research or other work (such as in industry, aka provider and
vendor space) must establish with reasonable credibility that there is
something to be gained by a particular strategy.  Then if there is a
potential for reasonable gains, the advantages and disadvatages can be
weighed.  With nothing on the advantages side, there is no sense in
spending any effort weighing the disadvantages.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Mon, 25 Mar 2013 23:42:46 -0400

> Please find my comments in-line below.

Hi Mingui,

> The methodology from my side is to put those optical components in a
> manageable DOWN state (sleeping). If protocol work creates the
> opportunity for them to stay in the sleeping state for seconds or
> minutes, energy saving can be significant.  Fiber forms for Ethernet
> does not support micro-sleeping mode, but they may support
> deep-sleeping mode. PHYs can negotiate a longer time to stay in
> deep-sleeping mode.

BTW- That sounds to me like solution space.

As stated from Pat (don't have her last name handy) at IETF-86, there
was a reason IEEE wrote the existing standard for copper Ethernet
only.

It is harder to power down a laser and power up.  It is very hard to
power down and power back up a temperature compensated finely tuned
laser used in WDM systems, particularly todays DWDM and ULH circuits.
Those can take over a minute to acheive stability, during which time
they cannot be used.  So if you could power down for an extended
period of time, maybe there would be hope.

> I agree that we should let numbers tell the truth. According to our
> observation, line-cards are the most power hungry part of high-end
> routers and switches. They can consume nearly 70% of the total
> power. Meanwhile, idle routers/switches consume more than 80% of their
> peak power. If we can shut-down components on line-cards or whole
> line-cards, we can save a considerable amount of energy.

Your observation used the topology of a real network but had nothing
to do with the power being consumed by the network who's topology you
used.

> This observation treats a router/switch as a black box. Our next step
> is to open this black box and see what further meaningful numbers we
> will get.

Yes.  And that is still in the research stage.

> Energy Efficient Ethernet (EEE) involves a great deal of protocol
> work. Network equipments supporting EEE have already been shipped
> around by vendors. Providers may not resist solutions that can save
> their OPEX, only if vendors can come up with workable ones.

And supports copper only.  Providers don't use copper Ethernet.

> Power/Energy Aware NETwork needs the combination of "hardware work"
> and "protocol work". Protocol work should make use of the hardware
> characteristics. IOW, protocol work should create the opportunity for
> these techniques to save most energy.

The assertion that protocol work is needed is nothing more than an
empty assertion.  There are currently no facts to back it up.

> Energy saving for composite links may become another low-hanging
> fruit. As you pointed out, the traffic aggregation onto component
> links may impact the cooling system. We can have a test to see what
> will happen, whether we should have some limit on the aggregation.
>  
> Thanks,
> Mingui

I'm not convinced that there is any savings for composite link, or
today's simpler LAG.  The switching uses more power than the laser on
a short haul interface.  I'm not sure to shut down the transport
lasers.  If you move the switching to a smaller subset of cards, you
heat that subset of cards more and the fan has to run hard enough to
cool the hottest card in the chassis.  There may be no net decrease in
switching power, negligible laser power decrease, and an larger
increase in fan power, yielding higher power consumption for the same
amount of traffic.  AFAIK, no one has measured using the types of
equipment actually at a provider.  Using vendor data sheets and
extrapolating doesn't work.

Curtis

---

From: Shane Amante <[email-masked]>
Date: Tue, 26 Mar 2013 10:05:25 -0600

Curtis,

One correction, below.

> In message <[email-masked]>
[--snip--]
> And supports copper only.  Providers don't use copper Ethernet.

Sure, we do ... at the [very] **edges** of our network.  

I wholly agree with you that Providers (absolutely) DO NOT use copper Ethernet in the "core" of their network, which I would define as:
a) Inter-City links, (P <-> P)
b) Peering links, (ASBR <-> ASBR)
c) P <-> PE links
d) Intra-Metro [WDM] device interconnection links, e.g.: interconnection of Ethernet switches via their East-West ring-side interfaces, (N-PE <-> U-PE *and* U-PE <-> U-PE).

For the above, providers only ever use fiber and/or WDM over fiber for such interconnections.  It's interesting to note that most of those links, (a) thru (c), are the only links over which providers run routing and signaling protocols, (e.g.: IS-IS, LDP, RSVP, etc.).  While technically possible to run routing and signaling protocols for (d), due to a variety of factors, that is not the case.  Instead, it is more common to see some form of Layer-2 ring protocol protocol, (e.g.: RSTP, G.8032, etc.), being run on those interfaces, since there is only a very simple East-West forwarding decision that needs to be made.  Regardless, I agree with your overall points and question what, if any, value is there in modifying routing and/or signaling protocols for this "core" part of the network to accommodate idle power states, given the substantial complexity in attempting to quickly *and* accurately transition from a quiescent state back to a 'nominal' state of full capacity.

I would say that we *DO* use copper Ethernet at the [furthest] "edge" of the network, which I would define as a U-PE <-> CE hand-offs, where the distances are short enough (> 100m) that it is technically feasible to do so.  (Refer to RFC4026 for definitions of U-PE to N-PE).  Interestingly enough, a substantial portion of the *N*-PE <-> CE circuits are running BGP, (think: Internet and/or IPVPN services).  But, as I stated above, the N-PE <-> U-PE interconnects use: i) fiber/WDM-over-fiber; and, ii) are statistically multiplexed in that multiple U-PE's share the same physical "uplinks" into the N-PE.  So, even if it were possible to somehow "turn-off" BGP from N-PE to CE, then I don't see that that could practically lead to power-savings on the N-PE since the N-PE's ring-side interfaces are both optical and need to stay operational/running in order to provide service to other U-PE's.

Thus, the only practical application I can see of power savings would be on copper interfaces at the deepest "edge" of the network, (U-PE to CE), but there's no active routing protocols on those interfaces.  And, although there's Layer-2 control protocols, e.g.: LLDP and the MEF's "Ethernet LMI", but I've not seen either of those achieve widespread deployment mostly because the CE devices do not support it, (yet).  But, we're the IETF, not the IEEE nor the MEF ... so, I'm not clear what the IETF would be able to work on here.

-shane

---

From: Tony Tauber <[email-masked]>
Date: Tue, 26 Mar 2013 21:15:28 -0400

On Tue, Mar 26, 2013 at 12:05 PM, Shane Amante <[email-masked]>wrote:

>
> Thus, the only practical application I can see of power savings would be
> on copper interfaces at the deepest "edge" of the network, (U-PE to CE),
> but there's no active routing protocols on those interfaces.  And, although
> there's Layer-2 control protocols, e.g.: LLDP and the MEF's "Ethernet LMI",
> but I've not seen either of those achieve widespread deployment mostly
> because the CE devices do not support it, (yet).  But, we're the IETF, not
> the IEEE nor the MEF ... so, I'm not clear what the IETF would be able to
> work on here.

Now that you mention it, datacenter or campus LANs would appear to be the
biggest wins in terms of power and raw numbers of interfaces.
IEEE seems like a better match in that case.  No idea if there are things
going on in that venue.

Tony

---

From: Mingui Zhang <[email-masked]>
Date: Wed, 27 Mar 2013 02:22:46 +0000

>IEEE seems like a better match in that case.  No idea if there are things going
>on in that venue.

As for "datacenter or campus LANs", I'd mention that ISIS can be used as the control protocol (e.g., TRILL and SPB). This is an area that matches IETF.

Mingui

>Tony Tauber
>Sent: Wednesday, March 27, 2013 9:15 AM
>To: Shane Amante
>Cc: [email-masked]
>Subject: Re: I-D Action: draft-retana-rtgwg-eacp-01.txt
>
>	Thus, the only practical application I can see of power savings would be on
>copper interfaces at the deepest "edge" of the network, (U-PE to CE), but
>there's no active routing protocols on those interfaces.  And, although there's
>Layer-2 control protocols, e.g.: LLDP and the MEF's "Ethernet LMI", but I've not
>seen either of those achieve widespread deployment mostly because the CE
>devices do not support it, (yet).  But, we're the IETF, not the IEEE nor the
>MEF ... so, I'm not clear what the IETF would be able to work on here.
>
>
>IEEE seems like a better match in that case.  No idea if there are things going
>on in that venue.
>
>Tony

---

From: Mick Seaman <[email-masked]>
Date: Tue, 26 Mar 2013 20:30:35 -0700

I'll let the TRILL folks speak for themselves, but I would politely 
suggest that you keep your hands off SPB.

So far as SPB is concerned ISIS is a protocol for ensuring that all 
participants converge on the same data as the basis of performing the 
necessary topology and related calculations. Those calculations are 
specified by the IEEE. Some of the data shared, and the way in which the 
calculations are performed, has a close relationship with 
interoperability with parts of the bridged/switched network where other 
topology protocols operate. The fact that ISIS is employed as part of 
the solution is not an invitation for others to "enhance"/rework/invent 
anew or otherwise muck with the current standards.

If there is any doubt about this I can no doubt arrange for the point to 
be made more forcefully.

Mick Seaman

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 28 Mar 2013 17:01:15 -0400

In message <[email-masked]>
Mingui Zhang writes:
> As for "datacenter or campus LANs", I'd mention that ISIS can be used
> as the control protocol (e.g., TRILL and SPB). This is an area that
> matches IETF.
>  
> Mingui

Mingui,

Are you familiar with the english (maybe just american) expression
"grabbing at straws"?

I think datacenter interest in Trill and SPB is declining,
particularly in very large datacenters where scaling limits are being
felt.  The current chaos in datacenters revolves around SDN, or more
precisely at this point, arguing about what SDN is and isn't.

Curtis

---

From: Mingui Zhang <[email-masked]>
Date: Fri, 29 Mar 2013 08:33:37 +0000

Hi Curtis,

Scalability is not the only issue that DC operators concern. Energy conservation is also a critical issue. I wonder whether you are suggesting another "straw" by mentioning SDN. However, we do not intend to work on greening DCNs right now. In PANET-Problem Statement we scope our work in ISP networks. 

Whatever the specific protocol is, only if it can be used to achieve our method then we can make use of it. The method is to aggregate traffic through protocol work and create opportunity for network devices to enter low power mode (e.g., sleeping) in order to achieve the energy conservation of the network. The apparent waste on idle power by high-end routers and switches makes us believe there is opportunity for this kind of energy saving. GreenTE is one of those solutions that reveal this feasibility. 

I feel that we are here trying to obtain conclusions based on guess rather than fact. Power saving of networks requires the cooperation of hardware & protocol. We need "walk on two legs". I think it's time to get more relevant observations along with hardware experts.

Thanks,
Mingui

>From: Curtis Villamizar [mailto:[email-masked]]
>Sent: Friday, March 29, 2013 5:01 AM
>To: Mingui Zhang
>Cc: Tony Tauber; Shane Amante; [email-masked]
><[email-masked]
>awei.com>
>Mingui Zhang writes:
>>
>going

---

From: "Eric Osborne (eosborne)" <[email-masked]>
Date: Fri, 29 Mar 2013 12:42:06 +0000

...

> I feel that we are here trying to obtain conclusions based on guess rather than
> fact. Power saving of networks requires the cooperation of hardware &
> protocol. We need "walk on two legs". I think it's time to get more relevant
> observations along with hardware experts.

Finally, something we can all agree on. :)

eric