Subject: Power aware networks : Comments requested from routing community

From: Balaji venkat Venkataswami <[email-masked]>
Date: Wed, 6 Feb 2013 11:52:31 +0530

Dear all,

We are a group of research and industry individuals tied together with a
common goal towards reducing the energy consumption in the core and edge
networks.

We present a metric-based hierarchical approach to reduce power consumption
in core and edge networks. The proposal considers both unicast and the
multicast cases. For unicast, the metric considered is *consumed-power to
available-bandwidth* and for multicast the metric is *consumed-power to
available-replication-capacity.*

 With unicast, the metric is used to determine a low-power path between
sources and destinations. With multicast, the metric serves the twin
purpose of finding low-power multicast paths as well as multicast
replication points.  We evolve multiple techniques at various hierarchical
levels. One at the Inter-AS level, Inter-Area level within the AS and
intra-Area within an AS. Additionally, the proposed method can also be used
to determine disjoint or redundant paths for load balancing or fault
tolerance. Additionally since TCAMs are one of the biggest power guzzlers
in all the components on a router/switch, we have presented a solution for
intra-AS purposes to use the TCAM according to the traffic matrix passing
through the system and shut down those TCAM banks that are unused. With
this in mind, we have also advocated taking into account a TCAM-POWER-Ratio
in order to compute the paths from source to destination based on this
metric. Once low-power paths, in either the unicast or the multicast cases,
are identified then currently available traffic engineering techniques
could be used to route the data packets. In the case of inter-AS BGP path
selection is also modified to arrive at paths which are low-power paths.

 Our main objective is as follows...

We now outline four important aspects that any approach for power reduction
should be capable of addressing.

 *Should cater for both unicast and multicast scenarios*

Multicast provides an important scenario for the Internet. Today, most
proposals consider mainly low-power path routing with unicast traffic.
Multicast traffic has received a lot of attention in wireless networks, but
not in the wired domain. Any new proposal should be able to address both
the unicast and the multicast traffic scenarios. Having different methods
for these two scenarios might lead to unnecessary processing burden in the
routers, which might hinder its scalability.

 *Should not rely on just switching off unused links*

Most approaches to optimize energy pursue the following approach: measure,
monitor and respond to the system energy usage by switching off unused or
under-utilized links. Such an approach could be effective for reducing
power locally. The effect on the network is not clearly understood.
Further, the power usage involved in turning on and rebooting/reconfiguring
the device is often not explicitly considered. We note that Service Level
Agreement (SLA) requirements may not even permit the links to be switched
off. Also services provided by ISPs like Virtual Private Networks (VPNs)
can be affected by such re-routing decisions.

 *Should follow an hierarchical and distributed approach*

For scalability, it is important that the algorithms proposed for inter-AS
should also be applicable to intra-AS situations. Networks do not work in
isolation, so any proposal should be both distributed and hierarchical. The
algorithms should be applicable at every level of the hierarchy.

 *Should  provide incentives for ISP for adoption*

The engineering proposals should be aligned with commercial incentives for
rapid and widespread adoption. Today, the device manufacturers and the ISPs
operate independently of each other, and there is no incentive for
manufacturers to work towards low-power and high bandwidth devices. An
ISP’s revenue model is based on the consumed bandwidth, which in turn lead
naturally to more power consumption. If the proposed method chooses routers
that consume low-power and increase the data flow through them, then this
indirectly provides encouragement for ISPs to purchase low-power and high
bandwidth devices.

We now present our metric-based proposals in the below mentioned drafts
which addresses the aforementioned design aspects.

We would like the routing community to provide feedback on these drafts. We
also intend to present this work in an abridged format in the upcoming IETF.

 The drafts are as follows....

   - mjsraman-panet-bgp-power-path<http://tools.ietf.org/html/draft-mjsraman-panet-bgp-power-path>
    (timeline)<http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-bgp-power-path-timing.html>
    Inter-AS-Proposal
   - mjsraman-panet-ecmp-redirect-power-repl-cap<http://tools.ietf.org/html/draft-mjsraman-panet-ecmp-redirect-power-repl-cap>
    (timeline)<http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-ecmp-redirect-power-repl-cap-timing.html>
    Multicast
   - mjsraman-panet-inter-as-power-source<http://tools.ietf.org/html/draft-mjsraman-panet-inter-as-power-source>
    (timeline)<http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-inter-as-power-source-timing.html>
Inter-AS
   Proposal
   - mjsraman-panet-inter-as-psp<http://tools.ietf.org/html/draft-mjsraman-panet-inter-as-psp>
    (timeline)<http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-inter-as-psp-timing.html>
Inter-AS
   Proposal
   - mjsraman-panet-inter-as-psp-protect<http://tools.ietf.org/html/draft-mjsraman-panet-inter-as-psp-protect>
    (timeline)<http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-inter-as-psp-protect-timing.html>
Inter-AS
   Proposal
   - mjsraman-panet-pce-power-mcast-replic<http://tools.ietf.org/html/draft-mjsraman-panet-pce-power-mcast-replic>
    (timeline)<http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-pce-power-mcast-replic-timing.html>
    Multicast
   - mjsraman-panet-pim-power<http://tools.ietf.org/html/draft-mjsraman-panet-pim-power>
    (timeline)<http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-pim-power-timing.html>
    Multicast
   - mjsraman-panet-tcam-power-efficiency<http://tools.ietf.org/html/draft-mjsraman-panet-tcam-power-efficiency>
    (timeline)<http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-tcam-power-efficiency-timing.html>
TCAM
   related
   - mjsraman-panet-tcam-power-ratio<http://tools.ietf.org/html/draft-mjsraman-panet-tcam-power-ratio>
    (timeline<http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-tcam-power-ratio-timing.html>)
   TCAM related
   - mjsraman-pce-power-replic<http://tools.ietf.org/html/draft-mjsraman-pce-power-replic>
    (timeline)<http://www.arkko.com/tools/lifecycle/draft-mjsraman-pce-power-replic-timing.html>
    Multicast
   - mjsraman-panet-intra-as-psp-te-leak<http://tools.ietf.org/html/draft-mjsraman-panet-intra-as-psp-te-leak>
    (timeline)<http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-intra-as-psp-te-leak-timing.html>
Inter-Area
   within an AS
   - mjsraman-panet-ospf-power-topo<http://tools.ietf.org/html/draft-mjsraman-panet-ospf-power-topo>
    (timeline)<http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-ospf-power-topo-timing.html>
Intra-Area
   within an AS

We understand it is a lot of matter to go through. We would much appreciate
if some of you could review the inter-AS proposals while others take up
multicast and Intra-AS unicast and multicast.

Thanks again for your time on this matter.

thanks and regards,
balaji venkat

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 06 Feb 2013 13:41:11 -0500

Balaji,

"We" in the context of your first paragraph seems to be a
misrepresentation.  The authors of all of these drafts seem to be from
the same university in India.  From prior attempts on your part to get
a draft of this sort into IDR and a brief reading of a few of the
drafts that you have just submitted, you don't seem to have a good
understanding of how networks are built and how network equipment is
built from which to begin to attack the problem of reducing the power
consumption of these networks.

If you want to try to advance a research paper with your theories on
power reduction, please choose an appropriate venue such as a refereed
technical journal.

Curtis

In message <[email-masked]>
Balaji venkat Venkataswami writes:
 
> ISP=92s revenue model is based on the consumed bandwidth, which in turn lea=
> d
> also intend to present this work in an abridged format in the upcoming IETF=
> .
>  
>    - mjsraman-panet-bgp-power-path<http://tools.ietf.org/html/draft-mjsrama=
> n-panet-bgp-power-path>
>     (timeline)<http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-bg=
> p-power-path-timing.html>
>     Inter-AS-Proposal
>    - mjsraman-panet-ecmp-redirect-power-repl-cap<http://tools.ietf.org/html=
> /draft-mjsraman-panet-ecmp-redirect-power-repl-cap>
>     (timeline)<http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-ec=
> mp-redirect-power-repl-cap-timing.html>
>     Multicast
>    - mjsraman-panet-inter-as-power-source<http://tools.ietf.org/html/draft-=
> mjsraman-panet-inter-as-power-source>
>     (timeline)<http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-in=
> ter-as-power-source-timing.html>
> Inter-AS
>    Proposal
>    - mjsraman-panet-inter-as-psp<http://tools.ietf.org/html/draft-mjsraman-=
> panet-inter-as-psp>
>     (timeline)<http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-in=
> ter-as-psp-timing.html>
> Inter-AS
>    Proposal
>    - mjsraman-panet-inter-as-psp-protect<http://tools.ietf.org/html/draft-m=
> jsraman-panet-inter-as-psp-protect>
>     (timeline)<http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-in=
> ter-as-psp-protect-timing.html>
> Inter-AS
>    Proposal
>    - mjsraman-panet-pce-power-mcast-replic<http://tools.ietf.org/html/draft=
> -mjsraman-panet-pce-power-mcast-replic>
>     (timeline)<http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-pc=
> e-power-mcast-replic-timing.html>
>     Multicast
>    - mjsraman-panet-pim-power<http://tools.ietf.org/html/draft-mjsraman-pan=
> et-pim-power>
>     (timeline)<http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-pi=
> m-power-timing.html>
>     Multicast
>    - mjsraman-panet-tcam-power-efficiency<http://tools.ietf.org/html/draft-=
> mjsraman-panet-tcam-power-efficiency>
>     (timeline)<http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-tc=
> am-power-efficiency-timing.html>
> TCAM
>    related
>    - mjsraman-panet-tcam-power-ratio<http://tools.ietf.org/html/draft-mjsra=
> man-panet-tcam-power-ratio>
>     (timeline<http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-tca=
> m-power-ratio-timing.html>)
>    TCAM related
>    - mjsraman-pce-power-replic<http://tools.ietf.org/html/draft-mjsraman-pc=
> e-power-replic>
>     (timeline)<http://www.arkko.com/tools/lifecycle/draft-mjsraman-pce-powe=
> r-replic-timing.html>
>     Multicast
>    - mjsraman-panet-intra-as-psp-te-leak<http://tools.ietf.org/html/draft-m=
> jsraman-panet-intra-as-psp-te-leak>
>     (timeline)<http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-in=
> tra-as-psp-te-leak-timing.html>
> Inter-Area
>    within an AS
>    - mjsraman-panet-ospf-power-topo<http://tools.ietf.org/html/draft-mjsram=
> an-panet-ospf-power-topo>
>     (timeline)<http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-os=
> pf-power-topo-timing.html>
> Intra-Area
>    within an AS
>

---

From: Balaji venkat Venkataswami <[email-masked]>
Date: Thu, 7 Feb 2013 00:22:54 +0530

Dear Curtis,

As already stated to you in a private email, "We" are a group of people
with 40 years
of collective industry experience in the networking industry. We dont need
to be
patronized by anybody since we have independent minds that have the
capability
to digest information not based on somebody else's interpretation of how
future
networks need to be built and how power reduction needs to play a part in
it.

As you might recall one such research paper on GreenTE by Beichuan Zhang
rang a lot of bells in the IETF by collecting the ANRP prize. So please
dont misstate
facts as you might know them and shove your ideas of how networks are built
down our throats.

We think the IETF is a free and fair body that accepts opinions and ideas
from
all sides. We want to keep this a free and fair organization. So incumbent
people like you ought to encourage us and have a fair argument when we
present
such work to you. Dont follow the policy of exclusion to the nth degree.

It is but fair to say that you seem to be in a minority on this matter.
Nobody else
responded with unkind misstatements of facts and mis-understanding of our
technical antecedents.

Suffice to say that if we get an opportunity to present this (since IETF is
OUR
organization too) you need not sit through it. If you want to give us a
fair hearing
please follow a policy of kind inclusion and not create a ruckus about us
research
folks trying to make an earnest attempt at some practical research that
could
well save the planet.

Yours sincerely,
Thanks and regards,
balaji venkat and shankar raman

On Thu, Feb 7, 2013 at 12:11 AM, Curtis Villamizar <[email-masked]> wrote:

> In message <
> [email-masked]>
> >    - mjsraman-panet-bgp-power-path<
> http://tools.ietf.org/html/draft-mjsrama=
> > n-panet-bgp-power-path>
> >     (timeline)<
> http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-bg=
> >     (timeline)<
> http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-ec=
> >     (timeline)<
> http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-in=
> >    - mjsraman-panet-inter-as-psp<
> http://tools.ietf.org/html/draft-mjsraman-=
> > panet-inter-as-psp>
> >     (timeline)<
> http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-in=
> > ter-as-psp-timing.html>
> > Inter-AS
> >    Proposal
> >     (timeline)<
> http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-in=
> >     (timeline)<
> http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-pc=
> >    - mjsraman-panet-pim-power<
> http://tools.ietf.org/html/draft-mjsraman-pan=
> > et-pim-power>
> >     (timeline)<
> http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-pi=
> > m-power-timing.html>
> >     Multicast
> >     (timeline)<
> http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-tc=
> >     (timeline<
> http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-tca=
> >    - mjsraman-pce-power-replic<
> http://tools.ietf.org/html/draft-mjsraman-pc=
> > e-power-replic>
> >     (timeline)<
> http://www.arkko.com/tools/lifecycle/draft-mjsraman-pce-powe=
> > r-replic-timing.html>
> >     Multicast
> >     (timeline)<
> http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-in=
> >     (timeline)<
> http://www.arkko.com/tools/lifecycle/draft-mjsraman-panet-os=

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 06 Feb 2013 15:05:12 -0500

In message <CAHF4apPfmB5gveLAULhbGaHQTf=[email-masked]>
Balaji venkat Venkataswami writes:
 
> As you might recall one such research paper on GreenTE by Beichuan
> Zhang rang a lot of bells in the IETF by collecting the ANRP prize. So
> please dont misstate facts as you might know them and shove your ideas
> of how networks are built down our throats.

That was a research paper published in IEEE International Conference
on Network Protocols (ICNP).

Power savings potential (a computation, not a measurement) were as
high as 20-40%.  That is good but ...

Line cards considered were old OC3, OC48, and one port OC192 cards.
The paper ignores the reality that shutting off one port on a
multiport card makes little difference, certainly far less than a
reduction of 1/N.  The power consumption of transport was not
included.  Impact on fault restoration was not considered.  Lots of
other flaws, but still good work and a potential 20-40% is nice, even
if factoring in reality might get only half that or less.

The point though is that a 1 port OC192 at 174 watts (10 Gb/s) is no
match for a 500 Gb/s card at roughly 650 watts.  That is more than
40%.  It is more than a decimal order of magnitude greater.  It is not
comparing apples to oranges, one is an IP/MPLS router with full
routing tables and short haul interfaces, the other does OTN switching
drives either short haul or long haul transport interfaces, and could
do IP/MPLS, but with only IGP routes and MPLS LSPs at that power
figure.  So maybe the way the network is built needs to change.

Anyway, if *I* were to continue this discussion, then *I* should get
off the IETF lists and go write an academic paper.  [Or better yet,
work for a service provider or equipment vendor and work toward
building better equipment and networks.]

The point is that efficiency has gone up by orders of magnitude as we
went from OC3 to OC12 to OC48 to OC192 (circa 1995-2000) and then
multiport OC192 and OC768 and 10GbE (circa 2005-2010) to 100GbE and
ODU4 (somewhat recent) to multiport 100GbE and ODU4 (even more
recent).  Power per bit drops about one to two decimal order of
magnitude per decade and for any given a specific snapshot in the
technology timeline, there are means to reduce power that get far more
than the 20-40% that GreenTE potentially offers.

Of course a new generation of equipment means CapEx and GreenTE is
perceived as just routing and therefore free, so therefore the
interest.

Another point is that no one ever tried this on a real network.  With
RSVP-TE no protocol changes would be necessary, just offline
computation, a configured preferred explicit-path, and some interface
shutdowns.  I don't know that anyone has even measured the reduction
in power of an idled or administratively shut down interface or line
card.  There was nothing in the paper on this.

> > power reduction, please choose an appropriate venue such as a refereed

I repeat my main point.  [And hope to be among the reviewers]

Also IMHO GreenTE as described by Zhang is more likely to be feasible
with MPLS and RSVP-TE than with OSPF or BGP or PIM, etc.  But then,
the feasibility and potential benefits of GreenTE using IGP, BGP, and
PIM is what your academic paper can try to defend.

Thank you,

Curtis

---

From: "Eric Osborne (eosborne)" <[email-masked]>
Date: Wed, 6 Feb 2013 20:20:29 +0000

You aim to optimize core networks through traffic engineering such that some links can be shut down when not in use, or switched to lower power.

Have you done studies in core networks with real equipment that show the actual amount of money you could save with this approach?  Let's walk through a quick hypothetical network, you tell me where I got it wrong.

draft-mjsraman-panet-bgp-power-path says
" Power consumption can be
   reduced by trading off performance related measures like latency. For
   example, power savings while switching from 1 Gbps to 100 Mbps is
   approximately 4 W and from 100 Mbps to 10 Mbps around 0.1 Watts."

but provides no documentation to back these costs up nor itemization of where the power is consumed (amortized across the router?  linecard optics?  DWDM gear?)

Let's assume that this is true and that it is linear, so that you burn 4W per Gb.  (side note: I suspect this is inaccurate and that scaling is sublinear, but let's go with it because if it's sub-linear you have even less of a use case.  I also think the real world is far more complex, as you have all the optical transport gear to worry about.  But let's go with it for now.)

Running a 100Gb link thus draws 400W.

Let's say you have a backbone with (300) 100Gb links.  Total power consumption is thus (300*400) == 120 kW

Running all these links for 24 hours thus draws 120*24 == 2,880 kWh  == 2.8mWh
Power costs are maybe $0.10/kWh in the US.  Double that to cover the cost of cooling, so $0.20/kWh.
Thus, running the entire backbone costs $0.2 * 2,880 = $576/day.  $210,000/year.
Let's say your approach can save one third of the power cost, which means about $70,000.

An operator with 300 100Gb links in a network has hundreds of millions of dollars worth of gear and millions or tens of millions in payroll alone.  If you cut $70k from their opex they probably wouldn't even notice.  That's one or two salaries, or 0.002% of what Time Warner spent on advertising in 2006 (see: http://gaia.adage.com/images/random/FactPack06.pdf).  It is a drop in the bucket, if that.  And your proposal comes with significant work attached to it, and significant risk.  If your 40 years of experience in the network industry don't help you understand the risks you're asking an operator to take then you're missing a crucial part of any potential real world solution.  I do not think your work should be presented at the IETF unless it makes a much stronger argument that its benefits outweigh its costs.

These sorts of power optimizations all seem to be "here's how you reshape the problem so that you can throw a linear problem solver at it".  For example, [ http://www.cs.berkeley.edu/~suchara/publications/GreenNetsBundles.pdf ].  I've never seen anything which shows how much more work it will be for a network operator or which quantifies the actual savings.

If you can demonstrate significant savings in real networks at little or no cost to operators, you have an idea worth pursuing.  If your idea will cause more operational angst (e.g. not knowing whether your unused capacity will be there when you need it because you shut a third of it off all the time, increased risk of equipment failure from constant power-cycling, operational tools and training and expertise required to manage, deploy and troubleshoot variable-power links and the centralized NMS required to run them, etc etc etc) then it will find little traction.  Green-TE and power-aware BGP have been floating around for a while and have seen no real uptake in the WGs as far as I can see.  Is that not to be taken as an indication that there may be no real-world interest in them?  If not, what would it take to convince you?

eric

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Balaji venkat Venkataswami
> Sent: Wednesday, February 06, 2013 1:53 PM
> To: [email-masked]
> Cc: Shankar Raman M J; [email-masked]
> Subject: Re: Power aware networks : Comments requested from routing
> community
> 
> Dear Curtis,
> 
> As already stated to you in a private email, "We" are a group of people with 40
> years of collective industry experience in the networking industry. We dont
> need to be patronized by anybody since we have independent minds that have
> the capability to digest information not based on somebody else's
> interpretation of how future networks need to be built and how power
> reduction needs to play a part in it.
> 
> rang a lot of bells in the IETF by collecting the ANRP prize. So please dont
> misstate facts as you might know them and shove your ideas of how networks
> are built down our throats.
> 
> We think the IETF is a free and fair body that accepts opinions and ideas from
> present such work to you. Dont follow the policy of exclusion to the nth degree.
> 
> It is but fair to say that you seem to be in a minority on this matter. Nobody
> else responded with unkind misstatements of facts and mis-understanding of
> our technical antecedents.
> 
> Suffice to say that if we get an opportunity to present this (since IETF is OUR
> organization too) you need not sit through it. If you want to give us a fair
> hearing please follow a policy of kind inclusion and not create a ruckus about us
> research folks trying to make an earnest attempt at some practical research
> that could well save the planet.
> 
> Yours sincerely,
> Thanks and regards,
> 	In message
> <[email-masked].
> com
> <mailto:CAHF4apO9bEkPk7QwA9fgJq9BNUNHNv%2BOFon_9_4Oij61e11r9w@
> mail.gmail.com> >
> 
> 	>    - mjsraman-panet-inter-as-power-
> source<http://tools.ietf.org/html/draft-=
> 	>    - mjsraman-panet-inter-as-psp-
> protect<http://tools.ietf.org/html/draft-m=
> 	>    - mjsraman-panet-tcam-power-
> efficiency<http://tools.ietf.org/html/draft-=
> 	>    - mjsraman-panet-tcam-power-
> ratio<http://tools.ietf.org/html/draft-mjsra=

---

From: Balaji Venkat <[email-masked]>
Date: Thu, 7 Feb 2013 02:36:11 +0530

Dear Eric,

Comments inline...

Sent from my iPad

> You aim to optimize core networks through traffic engineering such that some links can be shut down when not in use, or switched to lower power.

Actually the answer is a No. We do not advocate this approach. The main idea in bgp power path draft has a survey section on previous literature that we quote for reference and state that such approaches cause expensive equipment to be kept unused. You have quoted the survey section of the paper in the above sentence.

>   approximately 4 W and from 100 Mbps to 10 Mbps around 0.1 Watts."

Again this is a survey section of the paper surveying the existing approaches.

> but provides no documentation to back these costs up nor itemization of where the power is consumed (amortized across the router?  linecard optics?  DWDM gear?)

Our main intent is to allow traffic to follow low power paths through ASs (either through pce like entities or through modifying the bgp path selection algorithm) which consume the least power and possess the required bandwidth. By graphing the topology of a set ASs in the immediate neighbourhood using as-path-info strands typically for a provider manning multiple ASs within the providers own control this mechanism will let large chunks of traffic belonging to a macro FEC traverse such low power ASs through an inter-as te MPLS lsp computed by a pce like entity and setup by RSVP-te.

The use of the metric, the way the pwr metric is arrived at , the computation of the topology of ASs, consequent CSPF and then mapping the traffic to the constructed LSP is the main focus if this paper. For dampening fluctuations in the metric which are frequent heuristics algorithms are suggested.

By no means are we advocating shutting off links. This scheme facilitates even the follow the moon strategy naturally.

> An operator with 300 100Gb links in a network has hundreds of millions of dollars worth of gear and millions or tens of millions in payroll alone.  If you cut $70k from their opex they probably wou…

Again we are not asking the operator to shut off the equipment. He needs to keep it running as long as he has traffic to carry except that the pce makes a computation every time a large FEC of traffic needs to be carried through his ASs through a path that consumes the least power with the links therein having the required bandwidth. The ratio of consumed power to the available bandwidth is the metric that is used. Consumed power relates to a weighted average of the power consumed within the AS including the edge and core devices while the available bw relates to the links ingressing into the AS at the ASBRs that are the inter connectors between the ASes.

So in summary, we do not advocate shutting off links at all. In fact we disagree with that approach. The scheme we advocate tries to optimise the power consumption using the metric based approach with some heuristics.

Hope this helps.

Thanks and regards,
Balaji venkat

---

From: Balaji Venkat <[email-masked]>
Date: Thu, 7 Feb 2013 02:49:57 +0530

Dear Curtis,

Thank you for your detailed reply.
 
Comments inline...

Sent from my iPad

> card.  There was nothing in the paper on this.

We don't advocate shutting the links down. Instead we intend to maximise
Usage of all links with least power utilisation. It's an optimization problem 
With heuristics in graph theory involved.
> PIM is what your academic paper can try to defend.

Yes sir. In our proposals we intend to deploy most of our solutions with MPLS te 
Involved with pce entities and RSVP te as the signalling protocol of choice. The ospf
And Isis protocols need to be used only for carrying the power metric used in arriving at a te
LSP that is a low power path. This metric is carried in a opaque manner and kept in the TED.

Thanks and regards,
Balaji venkat

---

From: "Eric Osborne (eosborne)" <[email-masked]>
Date: Wed, 6 Feb 2013 21:34:10 +0000

...

> Again we are not asking the operator to shut off the equipment. He needs to
> keep it running as long as he has traffic to carry except that the pce makes a
> computation every time a large FEC of traffic needs to be carried through his
> ASs through a path that consumes the least power with the links therein having
> the required bandwidth. The ratio of consumed power to the available
> bandwidth is the metric that is used. Consumed power relates to a weighted
> average of the power consumed within the AS including the edge and core
> devices while the available bw relates to the links ingressing into the AS at the
> ASBRs that are the inter connectors between the ASes.
> 
> 
> So in summary, we do not advocate shutting off links at all. In fact we disagree
> with that approach. The scheme we advocate tries to optimise the power
> consumption using the metric based approach with some heuristics.

You totally skipped over the example I put up.  No matter how you save power, you haven't shown you can save enough to make any efforts worth it.  Do you have any case studies, even hypothetical one with real-world architectures and load, that demonstrate that your approach saves enough money to make it worth adopting?  Can you even get close?

eric

> 
> Hope this helps.
> 
> Thanks and regards,
> Balaji venkat
> >
> > These sorts of power optimizations all seem to be "here's how you reshape
> the problem so that you can throw a linear problem solver at it".  For example,
> [ http://www.cs.berkeley.edu/~suchara/publications/GreenNetsBundles.pdf ].
> I've never seen anything which shows how much more work it will be for a
> network operator or which quantifies the actual savings.
> >
> > If you can demonstrate significant savings in real networks at little or no cost
> to operators, you have an idea worth pursuing.  If your idea will cause more
> operational angst (e.g. not knowing whether your unused capacity will be there
> when you need it because you shut a third of it off all the time, increased risk of
> equipment failure from constant power-cycling, operational tools and training
> and expertise required to manage, deploy and troubleshoot variable-power
> links and the centralized NMS required to run them, etc etc etc) then it will find
> little traction.  Green-TE and power-aware BGP have been floating around for a
> while and have seen no real uptake in the WGs as far as I can see.  Is that not to
> be taken as an indication that there may be no real-world interest in them?  If
> not, what would it take to convince you?
> >
> >
> >
> > eric
> >
> >
> >> From: [email-masked] [mailto:[email-masked]] On
> >> Behalf Of Balaji venkat Venkataswami
> >> As already stated to you in a private email, "We" are a group of
> >> people with 40 years of collective industry experience in the
> >> networking industry. We dont need to be patronized by anybody since
> >> we have independent minds that have the capability to digest
> >> information not based on somebody else's interpretation of how future
> >> Zhang rang a lot of bells in the IETF by collecting the ANRP prize.
> >> So please dont misstate facts as you might know them and shove your
> >> ideas of how networks are built down our throats.
> >>
> >> We think the IETF is a free and fair body that accepts opinions and
> >> ideas from all sides. We want to keep this a free and fair
> >> organization. So incumbent people like you ought to encourage us and
> >> have a fair argument when we present such work to you. Dont follow the
> policy of exclusion to the nth degree.
> >>
> >> It is but fair to say that you seem to be in a minority on this
> >> matter. Nobody else responded with unkind misstatements of facts and
> >> mis-understanding of our technical antecedents.
> >>
> >> Suffice to say that if we get an opportunity to present this (since
> >> IETF is OUR organization too) you need not sit through it. If you
> >> want to give us a fair hearing please follow a policy of kind
> >> inclusion and not create a ruckus about us research folks trying to
> >> make an earnest attempt at some practical research that could well save
> the planet.
> >>
> >> Yours sincerely,
> >> Thanks and regards,
> >>    > multicast cases. For unicast, the metric considered is
> >> *consumed- power to
> >>    > off. Also services provided by ISPs like Virtual Private
> >> Networks
> >> (VPNs)
> >>    > For scalability, it is important that the algorithms proposed
> >> for inter- AS

---

From: Balaji Venkat <[email-masked]>
Date: Thu, 7 Feb 2013 03:08:36 +0530

Dear Eric,

We have some experimental results 

Will share it with the you and the alias by tomorrow.

Thanks for your kind review and your time,

Thanks and regards,

Balaji venkat

Sent from my iPad

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 06 Feb 2013 21:05:06 -0500

In message <[email-masked]>
>    approximately 4 W and from 100 Mbps to 10 Mbps around 0.1 Watts."

Interesting example.  I really don't think service providers have core
links that small.  Even the metro is going to 1GbE and 10GbE.

The issue is really how many parallel 10 Gb/s links (older or edge),
or 100 Gb/s links (the newest core stuff) are lit and operational and
for this topic the issue is whether not using some of them saves
power.  Not using some ports and concentrating all the traffic on a
subset of ports saves very little power.

Most cores don't have any lower power higher delay paths.  Most cores
have some paths with less parallel links and some paths with more
parallel links, but generally don't have one link running a partial
capacity and one slower link available as an alternate.  If all the
links are same capacity and its just a matter of how many are running
in parallel, shortest hop count is the lowest power path.

There has been some discussion of building into high end line cards an
ability to go into a reduced power state, but most don't support this
yet (none that I am aware of).  If so, then the leakage power is
saved.  There is active power and leakage power in any ASIC.  No
traffic means almost no active power.  Leakage power is generally
small compared to active power except in a few type of logic blocks
like SRAM.  So the savings might not be all that great.

Anyone studied this?  It would help to know the max power draw and the
power draw of an idle high end card, because the idle power draw would
be the savings if the card could be put in a standby state.  Usually
just max power figures are given.

Data center is similar, typically built using the fat-tree model.
Lots of parallel links.  Often there is a jump up in link capacity at
levels of aggregation (top of rack, end of rack, building
interconnect), but the lowest speed interface would typically be the
two (for redundancy) GbE from a blade server to the top of rack.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 06 Feb 2013 22:13:11 -0500

In message <[email-masked]>
Balaji Venkat writes:
 
> Yes sir. In our proposals we intend to deploy most of our solutions
> with MPLS te Involved with pce entities and RSVP te as the signalling
> protocol of choice. The ospf And Isis protocols need to be used only
> for carrying the power metric used in arriving at a te LSP that is a
> low power path. This metric is carried in a opaque manner and kept in
> the TED.
>  
> Thanks and regards,
> Balaji venkat
>  
> > 
> > Thank you,
> > 
> > Curtis

This is the IETF, not the IRTF, and not an IEEE academic conference or
journal.

In IETF, standards are supposed to be written for protocols whose
underlying mechanism are known to work.  The "running code" and
"deployment" before standards advancement was largely to prevent
standards for unproven theories.

IRTF can take on standards (usually experimental, not standards track)
for important work that is research in nature but regarded as very
promising.

That is why I made the comment:

   If you want to try to advance a research paper with your theories
   on power reduction, please choose an appropriate venue such as a
   refereed technical journal.

IMO that is where this work belongs, not IETF.

You can do studies, like 1) finding out what kinds of interfaces exist
in core networks (hint: not 100 MbE or 10 MbE), 2) how much active
power each draw per bit (if you want to not put interfaces into
standby), and 3) then estimate the potential gains.  If the estimates
are promising *and* the assumptions are realistic, then it may be
something the IRTF or IETF might consider.

That is along the lines of what the Zhang paper did.  I'm not sure the
Zhang assumptions are realistic (who uses an OC3 today?), but as
academic papers go ... it may have touched on some new ground.

Curtis

---

From: Tony Li <[email-masked]>
Date: Wed, 6 Feb 2013 19:40:06 -0800

> for important work that is research in nature but regarded as very

One correction: The IRTF takes on research.  Never standards.

Tony

---

From: Mingui Zhang <[email-masked]>
Date: Thu, 7 Feb 2013 03:53:19 +0000

Hi Curtis,

I am very happy that you know GreenTE so well. Very good comments.
As the designer of the GreenTE model, I feel obliged to clarify some of the confusions. Please see in-lines below.

Thanks,
Mingui Zhang
Huawei Technologies

>Curtis Villamizar
>Sent: Thursday, February 07, 2013 4:05 AM
>To: Balaji venkat Venkataswami
><CAHF4apPfmB5gveLAULhbGaHQTf=[email-masked]
>mail.com>

If you read the model, you will find a variable B_m. It indicates the base power of the line card. This base power is high and can only be saved when all ports on a multiport card are shutting off. So the model did consider the reality that "shutting one port saves far less than a reduction of 1/N". Having said that, I agree that your reality is based on most current practice. But vendors can purposely provide line cards with single port. Then the card can be shut off when the port is shut off. Also, don't forget about composite links which are powered physically separately. GreenTE is now able to handle composite links. We have a demo website to show this: http://greente.65536.cn/.

Let me add another positive reality: when link cards are shutting off in a chassis, blowers in the same chassis may be shutting off or operate at a low speed. This helps the whole chassis save more power.

>figure.  So maybe the way the network is built needs to change.

What do you mean about the number "40%"? Anyway, line cards consume a large percentage of energy of the whole Chassis and this percentage is increasing fast. This judgment always holds. 

GreenTE only cares about line cards. So 20-40% saving actually means 20-40% of the total power consumed by all line-cards.

>than the 20-40% that GreenTE potentially offers.

Besides the power saved from the efficiency of the hardware, GreenTE can add another 20-40% saving, right? :) 

>PIM is what your academic paper can try to defend.

When a link is shut off, no control messages (e.g., HELLOs) can be delivered on it. Link state routing protocols (e.g., ISIS, OSPF), is required to be aware of this. 

[end of my clarifications]

>
>Thank you,
>
>Curtis
>_______________________________________________
>rtgwg mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/rtgwg

---

From: Balaji venkat Venkataswami <[email-masked]>
Date: Thu, 7 Feb 2013 15:27:49 +0530

---------- Forwarded message ----------
From: Shankar Raman <[email-masked]>
Date: Thu, Feb 7, 2013 at 1:59 PM
Subject: Re: Power aware networks : Comments requested from routing
community
To: "Eric Osborne (eosborne)" <[email-masked]>, [email-masked]
Cc: Gaurav Raina <[email-masked]>, Balaji Venkat Venkataswami <
[email-masked]>, [email-masked]

Hi Eric/Curtis,

Since I am not a part of RTGWG, I have been following only half of these
emails. Therefore as an observer of few of these emails, I believe there
are some critical point that have been missed, which needs clarification.

1. We are not looking at shutting down links. This is left up to the
service provider. Probably some ISP may not be able to shut off links, too.

2. The overall idea we are trying to convey is :  the data packets are not
routed today based on power information but  on other criteria (hops, QoS
metric etc.). We simply say that we must consider power as another criteria
while taking a routing decision. In this way, overall our traffic pattern
will incrementally start using low-power consuming devices and finally will
force more installation of low-power devices in the Internet.

3. The approach looks at collaborative reduction in power consumption of
entities in the Internet, not just at a device level or at the technology
point of view (optical or copper etc.). We agree that the idea looks
theoretical at present.

At least, from the simulations which we tried on various network
topologies, we find that there is huge opportunity in power savings using
the protocol based approach, depending on the power distribution of the
entities.

The questions we answer through the methods  are:
1. Even when the devices consume low-power, do we have strategies to route
data through those devices?
2. Don't we need a overall coordination between these devices to achieve
power reduction?

Since you have more experience than us in handling the devices and working
with the ISPs, we were working R&D for device manufacturers,  you have an
idea about the ISP device level issues related to cost and power. Please
note that our work does not DENY any of these facts.

If needed, we are open to collaboration and see how best we can integrate
the device power reduction strategies, along with those facilities offered
by the routing protocols today to get the best power reduction possible,
which is the goal of our research and which is why we are approaching IETF.

If you still believe that protocol level methods have no practical
significance in power reduction, then we will have to keep this idea to a
journal - (as Curtis has mentioned and has agreed to review). We are
positive that his advice will add value to our paper  and will definitely
seek it.

Hope this clarifies any confusion.

Thanks
Shankar

-----Original Message----- From: Balaji Venkat
Sent: Thursday, February 07, 2013 2:36 AM
To: Eric Osborne (eosborne)
Cc: [email-masked] ; Shankar Raman M J ; [email-masked]

Dear Eric,

Comments inline...

Sent from my iPad

On 07-Feb-2013, at 1:50 AM, "Eric Osborne (eosborne)" <[email-masked]>
wrote:

 You aim to optimize core networks through traffic engineering such that
> some links can be shut down when not in use, or switched to lower power.
>

Actually the answer is a No. We do not advocate this approach. The main
idea in bgp power path draft has a survey section on previous literature
that we quote for reference and state that such approaches cause expensive
equipment to be kept unused. You have quoted the survey section of the
paper in the above sentence.

> Have you done studies in core networks with real equipment that show the
> actual amount of money you could save with this approach?  Let's walk
> through a quick hypothetical network, you tell me where I got it wrong.
>
> draft-mjsraman-panet-bgp-**power-path says

Again this is a survey section of the paper surveying the existing
approaches.

> but provides no documentation to back these costs up nor itemization of
> where the power is consumed (amortized across the router?  linecard optics?
>  DWDM gear?)
>

Our main intent is to allow traffic to follow low power paths through ASs
(either through pce like entities or through modifying the bgp path
selection algorithm) which consume the least power and possess the required
bandwidth. By graphing the topology of a set ASs in the immediate
neighbourhood using as-path-info strands typically for a provider manning
multiple ASs within the providers own control this mechanism will let large
chunks of traffic belonging to a macro FEC traverse such low power ASs
through an inter-as te MPLS lsp computed by a pce like entity and setup by
RSVP-te.

The use of the metric, the way the pwr metric is arrived at , the
computation of the topology of ASs, consequent CSPF and then mapping the
traffic to the constructed LSP is the main focus if this paper. For
dampening fluctuations in the metric which are frequent heuristics
algorithms are suggested.

By no means are we advocating shutting off links. This scheme facilitates
even the follow the moon strategy naturally.

> Let's assume that this is true and that it is linear, so that you burn 4W
> per Gb.  (side note: I suspect this is inaccurate and that scaling is
> sublinear, but let's go with it because if it's sub-linear you have even
> less of a use case.  I also think the real world is far more complex, as
> you have all the optical transport gear to worry about.  But let's go with
> it for now.)
>
> An operator with 300 100Gb links in a network has hundreds of millions of
> dollars worth of gear and millions or tens of millions in payroll alone. If
> you cut $70k from their opex they probably wouldn't even notice. That's one
> or two salaries, or 0.002% of what Time Warner spent on advertising in 2006
> (see: http://gaia.adage.com/images/**random/FactPack06.pdf<http://gaia.adage.com/images/random/FactPack06.pdf>).
>  It is a drop in the bucket, if that.  And your proposal comes with
> significant work attached to it, and significant risk.  If your 40 years of
> experience in the network industry don't help you understand the risks
> you're asking an operator to take then you're missing a crucial part of any
> potential real world solution.  I do not think your work should be
> presented at the IETF unless it makes a much stronger argument that its
> benefits outweigh its costs.
>

Again we are not asking the operator to shut off the equipment. He needs to
keep it running as long as he has traffic to carry except that the pce
makes a computation every time a large FEC of traffic needs to be carried
through his ASs through a path that consumes the least power with the links
therein having the required bandwidth. The ratio of consumed power to the
available bandwidth is the metric that is used. Consumed power relates to a
weighted average of the power consumed within the AS including the edge and
core devices while the available bw relates to the links ingressing into
the AS at the ASBRs that are the inter connectors between the ASes.

So in summary, we do not advocate shutting off links at all. In fact we
disagree with that approach. The scheme we advocate tries to optimise the
power consumption using the metric based approach with some heuristics.

Hope this helps.

Thanks and regards,
Balaji venkat

> the problem so that you can throw a linear problem solver at it".  For
> example, [ http://www.cs.berkeley.edu/~**suchara/publications/**
> GreenNetsBundles.pdf<http://www.cs.berkeley.edu/~suchara/publications/GreenNetsBundles.pdf>]. I've never seen anything which shows how much more work it will be for a
> If you can demonstrate significant savings in real networks at little or
> no cost to operators, you have an idea worth pursuing.  If your idea will
> cause more operational angst (e.g. not knowing whether your unused capacity
> will be there when you need it because you shut a third of it off all the
> time, increased risk of equipment failure from constant power-cycling,
> operational tools and training and expertise required to manage, deploy and
> troubleshoot variable-power links and the centralized NMS required to run
> them, etc etc etc) then it will find little traction. Green-TE and
> power-aware BGP have been floating around for a while and have seen no real
> uptake in the WGs as far as I can see.  Is that not to be taken as an
> indication that there may be no real-world interest in them?  If not, what
> would it take to convince you?
>
>
>
> eric
>
>

---

From: Balaji venkat Venkataswami <[email-masked]>
Date: Thu, 7 Feb 2013 18:21:05 +0530

Dear all,

Adding the rtgwg alias again since message size was exceeded.

thanks and regards,
balaji venkat
---------- Forwarded message ----------
From: Shankar Raman <[email-masked]>
Date: Thu, Feb 7, 2013 at 1:59 PM
Subject: Re: Power aware networks : Comments requested from routing
community

Hi Eric/Curtis,

Thanks
Shankar

Dear Eric,

Comments inline...

Sent from my iPad

Hope this helps.

Thanks and regards,
Balaji venkat

---

From: Balaji venkat Venkataswami <[email-masked]>
Date: Thu, 7 Feb 2013 18:22:30 +0530

Dear all,

Adding the rtgwg alias again since the message size was exceeded.

thanks and regards,
balaji venkat

---------- Forwarded message ----------
From: Balaji venkat Venkataswami <[email-masked]>
Date: Thu, Feb 7, 2013 at 3:24 PM
Subject: Fwd: Power aware networks : Comments requested from routing
community
To: [email-masked], "Eric Osborne (eosborne)" <[email-masked]>, Curtis
Villamizar <[email-masked]>

Adding the rtgwg alias since some of us dont have access to sending mails
on it.

thanks and regards,
balaji venkat

---------- Forwarded message ----------
From: Shankar Raman <[email-masked]>
Date: Thu, Feb 7, 2013 at 3:17 PM
Subject: Re: Power aware networks : Comments requested from routing
community
To: Balaji Venkat <[email-masked]>, "Eric Osborne (eosborne)" <
[email-masked]>
Cc: [email-masked], [email-masked]

  Dear Eric,
Looks like we have not been clear in conveying our idea.

We abstract the power consumed by every router/switch, use it as the power
rating. We then assign it to the link as a metric characteristic to get
algorithms like SPF
and CSPF to choose the lowest power paths or any metric based routing
algorithm.

 So the link metric is a combination of power consumed including the
physical link and router components. In effect the link metric is not just
the physical link’s metric alone (but the cost of the power consumed by the
router/switch on which it connects to the neighbor).

Just like routing protocols consider delay, hops as metric we consider this
power consumption as yet another metric.

 Thanks
Shankar

-----Original Message-----
From: Balaji Venkat
Sent: Thursday, February 07, 2013 2:36 AM
To: Eric Osborne (eosborne)
Cc: [email-masked] ; Shankar Raman M J ; [email-masked]
Subject: Re: Power aware networks : Comments requested from routing
community

Dear Eric,

Comments inline...

Sent from my iPad

> but provides no documentation to back these costs up nor itemization of
where the power is consumed (amortized across the router?  linecard
optics?  DWDM gear?)

> An operator with 300 100Gb links in a network has hundreds of millions of
dollars worth of gear and millions or tens of millions in payroll alone.
If you cut $70k from their opex they probably wouldn't even notice.  That's
one or two salaries, or 0.002% of what Time Warner spent on advertising in
2006 (see: http://gaia.adage.com/images/random/FactPack06.pdf).  It is a
drop in the bucket, if that.  And your proposal comes with significant work
attached to it, and significant risk.  If your 40 years of experience in
the network industry don't help you understand the risks you're asking an
operator to take then you're missing a crucial part of any potential real
world solution.  I do not think your work should be presented at the IETF
unless it makes a much stronger argument that its benefits outweigh its
costs.

Hope this helps.

Thanks and regards,
Balaji venkat
the problem so that you can throw a linear problem solver at it".  For
example, [
http://www.cs.berkeley.edu/~suchara/publications/GreenNetsBundles.pdf ].
I've never seen anything which shows how much more work it will be for a
network operator or which quantifies the actual savings.

---

From: Hannes Gredler <[email-masked]>
Date: Thu, 7 Feb 2013 14:07:38 +0100

balaji,

may i put your attention to a talk that  francois lemarchand
gave on future net 2010.
https://www.dropbox.com/s/3jb9au2zca4nl3x/10_fn_S29.pdf
contains a copy.

slide 2 initially shows that the core and edge layer might be appealing
due to their per-box power print. however slide 15 discusses the
big picture where 99.99% of the power is burned by the home
appliances and subscriber line kit.

Do you think that optimizing a part of the network which gives only limited
overall savings is a worthwhile goal ? Note that SPs already ask their vendors
about reducing the power footprint of those core devices by improved, (sometimes
less functional) silicon forwarding engines.

/hannes

---

From: Balaji venkat Venkataswami <[email-masked]>
Date: Thu, 7 Feb 2013 19:22:46 +0530

Dear Hannes,

We are familiar with the presentation that you have referred to.

I have 3 questions for you.

a) While we agree with the presentation that you have mentioned, can we in
the IETF have control over the home appliances and subscriber line kit ? Is
there a methodology by which we can optimize the power consumption on these
devices ?

b) The problem space that the IETF can take up can be only in the edge and
core devices in the ISP networks and the Campus/DC/Metro etc...Assume that
I have n devices today with the same bandwidth offering but differing power
consumption
footprint, is there a way today to automatically send traffic through low
power paths in the topology of these devices.

c) So we can optimize power only on what we can control. What are your
views on this ?

thanks and regards,
balaji venkat

---

From: Thomas Nadeau <[email-masked]>
Date: Thu, 7 Feb 2013 07:06:07 -0800

The other problem with this (frankly naive approach which has been investigated by many) is that yes, while adding in another constraint to the metric computation, it may result in unexpected consequences.    This is why it is likely to be more suitable as an IRTF topic of research until those "details" are sorted out. 

	--Tom

On Feb 7, 2013:5:52 AM, at 5:52 AM, Balaji venkat Venkataswami <[email-masked]> wrote:

> a) While we agree with the presentation that you have mentioned, can we in the IETF have control over the home appliances and subscriber line kit ? Is there a methodology by which we can optimize the power consumption on these devices ?
> 
> b) The problem space that the IETF can take up can be only in the edge and core devices in the ISP networks and the Campus/DC/Metro etc...Assume that I have n devices today with the same bandwidth offering but differing power consumption
> footprint, is there a way today to automatically send traffic through low power paths in the topology of these devices.
> 
> c) So we can optimize power only on what we can control. What are your views on this ?
> 
> thanks and regards,
> balaji venkat 
>

---

From: Thomas Nadeau <[email-masked]>
Date: Thu, 7 Feb 2013 10:14:04 -0500

> Let me add another positive reality: when link cards are shutting off in a chassis, blowers in the same chassis may be shutting off or operate at a low speed. This helps the whole chassis save more…

	The problem with the model is that it makes assumptions that just do not reflect the reality of much of the equipment that is deployed today. For instance, while there is a lot of power used in the line card optics, the idle power supply capacity is another giant waste of power. This cannot be mitigated without shutting down the entire device.  Further considerations need to be made around the back-up facility power as well, which while not directly influenced by network traffic, does influence overall power utilization.  

> GreenTE only cares about line cards. So 20-40% saving actually means 20-40% of the total power consumed by all line-cards.

	The point that is being made I believe, is that while overall power consumption of devices is something we can strive to optimize, the means of using "green TE" might not be the best one given the reductions actually possible.

> Besides the power saved from the efficiency of the hardware, GreenTE can add another 20-40% saving, right? :) 

	Yes, but if the equipment is already very power efficient, the law of diminishing returns (or unexpected consequences) may dominate the equation.

	--Tom

---

From: Acee Lindem <[email-masked]>
Date: Thu, 7 Feb 2013 15:19:21 +0000

+1
Acee
On Feb 7, 2013, at 10:06 AM, Thomas Nadeau wrote:

--Tom

On Feb 7, 2013:5:52 AM, at 5:52 AM, Balaji venkat Venkataswami <[email-masked]<mailto:[email-masked]>> wrote:

Dear Hannes,

thanks and regards,
balaji venkat

On Thu, Feb 7, 2013 at 6:37 PM, Hannes Gredler <[email-masked]<mailto:[email-masked]>> wrote:
balaji,

/hannes

On Feb 6, 2013, at 7:41 PM, Curtis Villamizar wrote:

> In message <[email-masked]<mailto:[email-masked]>>
> [email-masked]<mailto:[email-masked]>

_______________________________________________
rtgwg mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/rtgwg

_______________________________________________
rtgwg mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/rtgwg

---

From: Balaji venkat Venkataswami <[email-masked]>
Date: Thu, 7 Feb 2013 21:01:39 +0530

Dear Tom, Acee,

Thank you for your comments.

We are currently approaching IRTF for this very purpose.

Additionally we would like to state that there are existing mechanisms for
aspects like delay (QoS) being additional metrics for the computation of
paths inter-AS and intra-AS. We feel that this additional power metric can
be handled in a way similar to how say aspects like delay are handled today.

Since you have said that this approach has already been tried out, we would
request you to give us some of the citations which will be of help to us.
We have already searched literature available in the web today. In case we
have missed out anything your citations would help us.

Thanks and regards,
balaji venkat

On Thu, Feb 7, 2013 at 8:49 PM, Acee Lindem <[email-masked]>wrote:

>  The other problem with this (frankly naive approach which has been
> investigated by many) is that yes, while adding in another constraint to
> the metric computation, it may result in unexpected consequences.    This
> is why it is likely to be more suitable as an IRTF topic of research until
> those "details" are sorted out.
>
>  --Tom
>
>
>  a) While we agree with the presentation that you have mentioned, can we
> in the IETF have control over the home appliances and subscriber line kit ?
> Is there a methodology by which we can optimize the power consumption on
> these devices ?
>
>  b) The problem space that the IETF can take up can be only in the edge
> and core devices in the ISP networks and the Campus/DC/Metro etc...Assume
> that I have n devices today with the same bandwidth offering but differing
> power consumption

---

From: "Eric Osborne (eosborne)" <[email-masked]>
Date: Thu, 7 Feb 2013 15:48:58 +0000

The root of your position seems to be that if you don't use a link, you consume less power, and thus avoiding certain links means consuming less power means saving money.

I see two problems with this:

1) I'm not sure that's actually true.  SONET, for one, sends empty frames when there's no data, so the laser is always firing.

2) even if it was true, or if you had some analogue to 802.3az that you could use in backbones, I don't believe you'd save anywhere near enough money to make it worth it.  

For #1, please explain the circumstances under which leaving a link idle saves power, and explain how much power and what drives that power draw.

For #2, please see my example in my original email.

eric

> From: Shankar Raman [mailto:[email-masked]]
> Sent: Thursday, February 07, 2013 4:48 AM
> To: Balaji Venkat; Eric Osborne (eosborne)
> Cc: [email-masked]; [email-masked]
> rating. We then assign it to the link as a metric characteristic to get algorithms
> like SPF and CSPF to choose the lowest power paths or any metric based routing
> algorithm.
> 
> So the link metric is a combination of power consumed including the physical
> link and router components. In effect the link metric is not just the physical
> link’s metric alone (but the cost of the power consumed by the router/switch on
> which it connects to the neighbor).
> 
> Actually the answer is a No. We do not advocate this approach. The main idea
> in bgp power path draft has a survey section on previous literature that we
> quote for reference and state that such approaches cause expensive equipment
> to be kept unused. You have quoted the survey section of the paper in the
> above sentence.
> 
> >
> actual amount of money you could save with this approach?  Let's walk through
> a quick hypothetical network, you tell me where I got it wrong.
> >
> > draft-mjsraman-panet-bgp-power-path says " Power consumption can be
> > but provides no documentation to back these costs up nor itemization
> > of where the power is consumed (amortized across the router?  linecard
> > optics?  DWDM gear?)
> 
> Our main intent is to allow traffic to follow low power paths through ASs (either
> through pce like entities or through modifying the bgp path selection
> algorithm) which consume the least power and possess the required bandwidth.
> By graphing the topology of a set ASs in the immediate neighbourhood using
> as-path-info strands typically for a provider manning multiple ASs within the
> providers own control this mechanism will let large chunks of traffic belonging
> to a macro FEC traverse such low power ASs through an inter-as te MPLS lsp
> computed by a pce like entity and setup by RSVP-te.
> 
> The use of the metric, the way the pwr metric is arrived at , the computation of
> the topology of ASs, consequent CSPF and then mapping the traffic to the
> constructed LSP is the main focus if this paper. For dampening fluctuations in
> the metric which are frequent heuristics algorithms are suggested.
> 
> > Let's assume that this is true and that it is linear, so that you burn
> > 4W per Gb.  (side note: I suspect this is inaccurate and that scaling
> > is sublinear, but let's go with it because if it's sub-linear you have
> > even less of a use case.  I also think the real world is far more
> > complex, as you have all the optical transport gear to worry about.
> > But let's go with it for now.)
> >
> > Running all these links for 24 hours thus draws 120*24 == 2,880 kWh
> > == 2.8mWh Power costs are maybe $0.10/kWh in the US.  Double that to
> cover the cost of cooling, so $0.20/kWh.
> dollars worth of gear and millions or tens of millions in payroll alone.  If you cut
> $70k from their opex they probably wouldn't even notice.  That's one or two
> salaries, or 0.002% of what Time Warner spent on advertising in 2006 (see:
> http://gaia.adage.com/images/random/FactPack06.pdf).  It is a drop in the
> bucket, if that.  And your proposal comes with significant work attached to it,
> and significant risk.  If your 40 years of experience in the network industry don't
> help you understand the risks you're asking an operator to take then you're
> missing a crucial part of any potential real world solution.  I do not think your
> work should be presented at the IETF unless it makes a much stronger
> argument that its benefits outweigh its costs.
>

---

From: "Shankar Raman" <[email-masked]>
Date: Thu, 7 Feb 2013 15:17:36 +0530

Dear Eric,

We abstract the power consumed by every router/switch, use it as the power rating. We then assign it to the link as a metric characteristic to get algorithms like SPF
and CSPF to choose the lowest power paths or any metric based routing algorithm. 

So the link metric is a combination of power consumed including the physical link and router components. In effect the link metric is not just the physical link’s metric alone (but the cost of the power consumed by the router/switch on which it connects to the neighbor). 

Just like routing protocols consider delay, hops as metric we consider this power consumption as yet another metric. 

Thanks
Shankar

-----Original Message----- 
From: Balaji Venkat 
Sent: Thursday, February 07, 2013 2:36 AM 
To: Eric Osborne (eosborne) 
Cc: [email-masked] ; Shankar Raman M J ; [email-masked] 
Subject: Re: Power aware networks : Comments requested from routing community 

Dear Eric,

Comments inline...

Sent from my iPad

Hope this helps.

Thanks and regards,
Balaji venkat

---

From: Balaji venkat Venkataswami <[email-masked]>
Date: Thu, 7 Feb 2013 21:30:57 +0530

CCing the alias as message went out of bounds in size.

On Thu, Feb 7, 2013 at 9:28 PM, Balaji venkat Venkataswami <
[email-masked]> wrote:

> Dear Eric,
>
> We seem to be discussing at cross purposes.
>
> The root of our position is not what you have stated. We do not say if you
> dont use a link you consume less power. We are not for avoiding links but
> we are for avoiding nodes/routers/switches which have a large power
> footprint. The essence of using link metrics is to avoid nodes with larger
> power footprints. That is how SPF or CSPF works. When it computes a low
> power path it avoids nodes and also links that lead to that node which has
> a larger power footprint. So the premise that we are just avoiding links is
> WRONG.
>
> For #1 question, we dont even talk about the power the link draws. We talk
> about the power footprint of the node that it is connected to.
>
> For #2 question, again we state that your premise of avoiding links in
> your example is a non-starter for us.
>
> thanks and regards,
> balaji venkat
>
>
> On Thu, Feb 7, 2013 at 9:18 PM, Eric Osborne (eosborne) <
> [email-masked]> wrote:
>
>> The root of your position seems to be that if you don't use a link, you
>> consume less power, and thus avoiding certain links means consuming less
>> power means saving money.
>>
>> 1) I'm not sure that's actually true.  SONET, for one, sends empty frames
>> when there's no data, so the laser is always firing.
>>
>> 2) even if it was true, or if you had some analogue to 802.3az that you
>> could use in backbones, I don't believe you'd save anywhere near enough
>> money to make it worth it.
>>
>> For #1, please explain the circumstances under which leaving a link idle
>> saves power, and explain how much power and what drives that power draw.
>>
>> algorithms
>> physical
>> idea
>> through
>> computation of
>> > By no means are we advocating shutting off links. This scheme
>> facilitates even
>> > the follow the moon strategy naturally.
>> >
>> >
>> > >
>> > > Let's say your approach can save one third of the power cost, which
>> means
>> > about $70,000.
>> > >
>> >
>> >
>> >
>> makes a
>> disagree
>> example,
>> > [ http://www.cs.berkeley.edu/~suchara/publications/GreenNetsBundles.pdf].

---

From: "Eric Osborne (eosborne)" <[email-masked]>
Date: Thu, 7 Feb 2013 16:08:44 +0000

[size trimmed]

OK, so you want to avoid nodes which draw more power.
Are you assuming that minimizing the traffic across a node also minimizes the power that node draws?

If yes:
 - please provide concrete examples of how this actually works.  In my experience the power draw by a router or switch is pretty constant, more driven by the *existence* of links (because links -> linecards -> power draw) than by the *traffic* on those links.

If not, what do you hope to accomplish by avoiding higher-power nodes?

eric

> From: Balaji venkat Venkataswami [mailto:[email-masked]]
> Sent: Thursday, February 07, 2013 10:59 AM
> Cc: Shankar Raman; [email-masked]; [email-masked]
> The root of our position is not what you have stated. We do not say if you dont
> use a link you consume less power. We are not for avoiding links but we are for
> avoiding nodes/routers/switches which have a large power footprint. The
> essence of using link metrics is to avoid nodes with larger power footprints.
> That is how SPF or CSPF works. When it computes a low power path it avoids
> nodes and also links that lead to that node which has a larger power footprint.
> So the premise that we are just avoiding links is WRONG.
> 
> For #2 question, again we state that your premise of avoiding links in your
> example is a non-starter for us.
> 
> thanks and regards,
> balaji venkat
>

---

From: Tony Li <[email-masked]>
Date: Thu, 7 Feb 2013 08:09:06 -0800

On Feb 7, 2013, at 5:07 AM, Hannes Gredler <[email-masked]> wrote:

> overall savings is a worthwhile goal ? 

Hannes, 

I'll just point out that this argument that you and Eric are espousing is skirting dangerously close to the quagmire of business.  And we know from long experience that the IETF does not do business models.

I'd like to strongly suggest that we simply restrict ourselves to the goal of saving power.  I think that we can agree, in general, that saving power is a worthwhile goal.  As to whether or not it is significant or makes economic sense is very much an issue that should be left to the operator community to decide.  Limited overall savings may be worthwhile in one context and pointless in another.

I know of one country where they are purportedly mandating power reductions.  In such situations, saving that last watt is the difference between a fine and not.  On the other hand, in a situation where power is very cheap, it's obviously silly.

Let's not argue about the marginal value of energy.  That's a business model issue.  Let's talk about how technology can actually save power.

Regards,
Tony

---

From: Balaji venkat Venkataswami <[email-masked]>
Date: Thu, 7 Feb 2013 21:52:01 +0530

Dear Eric,

Please take a look at the tcam-efficiency draft that we have published.

The URL to the draft is
http://tools.ietf.org/html/draft-mjsraman-panet-tcam-power-efficiency-00

A follow up draft that defines TCAM efficiency in terms of a metric is also
outlined in...

http://tools.ietf.org/html/draft-mjsraman-panet-tcam-power-ratio-02

With the help of the above schemes and additional work that is in progress
we intend to reduce the
power footprint of a device/router/switch/chassis such that important power
guzzling components of
the router are optimized.

In summary, the above 2 drafts coupled with the protocol schemes will we
feel result in better power
saving than what we have today. So essentially optimizing the power
consumed within a device
coupled with using protocol schemes between devices will result in saving
power.

Hope this helps.

thanks and regards,
balaji venkat

On Thu, Feb 7, 2013 at 9:38 PM, Eric Osborne (eosborne)
<[email-masked]>wrote:

>  - please provide concrete examples of how this actually works.  In my
> experience the power draw by a router or switch is pretty constant, more
> driven by the *existence* of links (because links -> linecards -> power
> draw) than by the *traffic* on those links.
>

---

From: Alia Atlas <[email-masked]>
Date: Thu, 7 Feb 2013 11:22:15 -0500

There are two different aspects to power-aware routing that I think are of
interest.
The first is developing a framework and method for evaluating different
possible solutions.
We had a discussion of draft-retana-rtgwg-eacp-00 in RTGWG in July 2012.

The second is examining and considering control plane and routing solutions
in that context.
That seems to me to be more likely something for the IRTF, rather than the
IETF.

The former, however, is where I think that the IETF's operational knowledge
and experience
could be useful in framing what would be a useful solution.

Would it be possible to discuss this work that Balaji is presenting in that
context?
In other words, if the goal of the solution is X,Y, and Z, then this
mechanism seems useful/
not useful, has stability concerns, etc.

And then there are the questions about assumptions and goals...  All of
this is great to
discuss in RTGWG - there's been significant interest, but what is mature
enough for the
IETF vs. the IRTF and whether there are sufficient people to work in the
area is not clear.

Alia

On Thu, Feb 7, 2013 at 11:09 AM, Tony Li <[email-masked]> wrote:

> I'll just point out that this argument that you and Eric are espousing is
> skirting dangerously close to the quagmire of business.  And we know from
> long experience that the IETF does not do business models.
>
> I'd like to strongly suggest that we simply restrict ourselves to the goal
> of saving power.  I think that we can agree, in general, that saving power
> is a worthwhile goal.  As to whether or not it is significant or makes
> economic sense is very much an issue that should be left to the operator
> community to decide.  Limited overall savings may be worthwhile in one
> context and pointless in another.
>
> I know of one country where they are purportedly mandating power
> reductions.  In such situations, saving that last watt is the difference
> between a fine and not.  On the other hand, in a situation where power is
> very cheap, it's obviously silly.
>
> Let's not argue about the marginal value of energy.  That's a business
> model issue.  Let's talk about how technology can actually save power.
>
> Regards,
> Tony
>

---

From: Balaji venkat Venkataswami <[email-masked]>
Date: Thu, 7 Feb 2013 22:04:07 +0530

Dear Eric,

You had written...

If yes...

<balajivenkat> Our answer to the above question is this. We are rephrasing
the question as "Can you send more traffic through low power consuming
nodes ?". Therefore If yes does not come into the picture.

The end objective slowly the traffic will migrate to the low power nodes
and the high power consuming nodes can be dismantled.

The other mail for TCAM that I had sent will deal with answer to your
problem.

thanks and regards,
balaji venkat

On Thu, Feb 7, 2013 at 9:52 PM, Balaji venkat Venkataswami <
[email-masked]> wrote:

> A follow up draft that defines TCAM efficiency in terms of a metric is
> also outlined in...
>
> power footprint of a device/router/switch/chassis such that important
> power guzzling components of
> On Thu, Feb 7, 2013 at 9:38 PM, Eric Osborne (eosborne) <

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 07 Feb 2013 15:18:25 -0500

In message <[email-masked]>
Mingui Zhang writes:
 
> Hi Curtis,
>  
> I am very happy that you know GreenTE so well. Very good comments.  As
> the designer of the GreenTE model, I feel obliged to clarify some of
> the confusions. Please see in-lines below.
>  
> Thanks,
> Mingui Zhang
> Huawei Technologies

Mingui,

Thank you for the clarifications.

A few comments are inline.

Curtis

> If you read the model, you will find a variable B_m. It indicates the
> base power of the line card. This base power is high and can only be
> saved when all ports on a multiport card are shutting off. So the
> model did consider the reality that "shutting one port saves far less
> than a reduction of 1/N". Having said that, I agree that your reality
> is based on most current practice. But vendors can purposely provide
> line cards with single port. Then the card can be shut off when the
> port is shut off. Also, don't forget about composite links which are
> powered physically separately. GreenTE is now able to handle composite
> links. We have a demo website to show this: http://greente.65536.cn/.

You are correct.  B_m does consider baseline power.

Line cards with single ports are impractical.  The fastest interfaces
are 100 Gb/s.  Packet processors ranging up to 500 Gb/s (line side
plus fabric side, so max 2x 100G) exist today (though they are few and
very recent) and 100G chips exist today (100G line side, plus > 100G
fabric side).  Some chassis today make use of a 500+ Gb/s card slot.

Composite Link or any other means of shutting down specific ports may
help.  Shutting down ULH transport capacity on a DWDM is more
problematic than shutting down a VSR Ethernet.  Even so, when the
forwarding chip supports more than one port, as is common, only the
MSA can be shut off.  In such a case, shutting down the receive side
saves nothing, so either side could initiate a bring up.

For lower speed interfaces, today's forwarding chips support on the
order of 12-24 x 10 GbE and much larger numbers of GbE.

> Let me add another positive reality: when link cards are shutting off
> in a chassis, blowers in the same chassis may be shutting off or
> operate at a low speed. This helps the whole chassis save more power.

Yes.  Even running an average packet size of larger than minimum
packet size should reduce power draw (less PPS) unless the vendor is
cheating a bit in the max power draw figure.  The fan speed will not
be at max except with fully utilized cards in all slots and AC fail.

> What do you mean about the number "40%"? Anyway, line cards consume a
> large percentage of energy of the whole Chassis and this percentage is
> increasing fast. This judgment always holds.
>  
> GreenTE only cares about line cards. So 20-40% saving actually means
> 20-40% of the total power consumed by all line-cards.

I think the range of potential savings in the figures indicates a
maximum of 20-40% saving network wide.

> can add another 20-40% saving, right? :)

Yes.  I am not arguing against continued work on GreenTE.  I am saying
that it is still in the research stage and there are bigger gains
elsewhere.

> When a link is shut off, no control messages (e.g., HELLOs) can be
> delivered on it. Link state routing protocols (e.g., ISIS, OSPF), is
> required to be aware of this.

The network should never be partitioned.  If partitioned due to
network fault, leaving the recieve side up with minimal logic enabled
allows either side to enable reactivation.  Detection of a partition
could also trigger bringing up idled capacity.

> [end of my clarifications]

Thanks again for the clarifications.  GreenTE is a worthwhile research
topic to pursue.  My primary point is that it still remain a topic of
research and attempts at IETF standardization are premature.

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 07 Feb 2013 16:23:39 -0500

Balaji,

This response was directed to Hannes, but I'll point out that we do
have a homenet WG within IETF.  See inline.

Curtis

In message <[email-masked]>
Balaji venkat Venkataswami writes:
> a) While we agree with the presentation that you have mentioned, can
>    we in the IETF have control over the home appliances and subscriber
>    line kit ? Is there a methodology by which we can optimize the
>    power consumption on these devices ?

Perhaps homenet WG could take on work in this area.

>    and core devices in the ISP networks and the Campus/DC/Metro
>    etc...Assume that I have n devices today with the same bandwidth
>    offering but differing power consumption footprint, is there a way
>    today to automatically send traffic through low power paths in the
>    topology of these devices.

There is no routing based solution but there are ways to put home
devices into standby or otherwise save significant amounts of power.

> c) So we can optimize power only on what we can control. What are your

If "we" is IETF, IETF can do work in home networking and historically
has done lots of work in the applications area and now has a homenet
WG which could recharter but may feel that power expertise in home
networking devices lies elsewhere.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 07 Feb 2013 16:57:59 -0500

In message <[email-masked]>
Balaji venkat Venkataswami writes:
 
> With the help of the above schemes and additional work that is in
> progress we intend to reduce the power footprint of a
> device/router/switch/chassis such that important power guzzling
> components of the router are optimized.
>  
> In summary, the above 2 drafts coupled with the protocol schemes will
> we feel result in better power saving than what we have today. So
> essentially optimizing the power consumed within a device coupled with
> using protocol schemes between devices will result in saving power.
>  
> Hope this helps.
>  
> thanks and regards,
> balaji venkat

balaji,

This is in the realm of router design.  

You are suggesting a TCAM in which banks can be shut off by reducing
the number of destination addresses in the FIB.

A better solution to reduce the power consumed by TCAM is don't use a
TCAM.  It is fast, but a radix trie based approach or other tree
approach consumes less power and pipelining makes such an approach
feasible if the lookup memory can be kept on-chip.  MPLS lookup can be
easily done in on chip SRAM rather than TCAM consuming mush less power
than an on-chip TCAM and a fraction of the power of off chip TCAM due
to the SERDES power needed to go off chip.

This pair of drafts in particular should be dropped.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 07 Feb 2013 17:00:10 -0500

Balaji,

If TCAM is your best example, you are not making a convincing
argument.  See the prior response.

Curtis

In message <CAHF4apPa++YH-B4f-cJdY62Vt-qUF=[email-masked]>

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 07 Feb 2013 17:44:03 -0500

In message <[email-masked]>
Alia Atlas writes:
> 
> There are two different aspects to power-aware routing that I think
> are of interest.  The first is developing a framework and method for
> evaluating different possible solutions.  We had a discussion of
> draft-retana-rtgwg-eacp-00 in RTGWG in July 2012.

Can you point to any message on the RTGWG mailing list?  I searched
locally for draft-retana-rtgwg-eacp and found none except a few
related to meeting agenda scheduling.

draft-retana-rtgwg-eacp is a 00 individual draft that has expired.

I just looked at this for the first time and I'm not at all convinced
that it is useful.

IMO this is not a useful framework and also proposes techniques that
are unlikely to have much if any impact (such as Section 5.4.
Introducing Jitter Through Microsleeps and Section 5.4.1.  An Example
of Microsleeps to Reduce Energy Usage.

> The second is examining and considering control plane and routing
> solutions in that context.  That seems to me to be more likely
> something for the IRTF, rather than the IETF.
>  
> The former, however, is where I think that the IETF's operational
> knowledge and experience could be useful in framing what would be a
> useful solution.
>  
> Would it be possible to discuss this work that Balaji is presenting in
> that context?  In other words, if the goal of the solution is X,Y, and
> Z, then this mechanism seems useful/ not useful, has stability
> concerns, etc.
>  
> And then there are the questions about assumptions and goals...  All
> of this is great to discuss in RTGWG - there's been significant
> interest, but what is mature enough for the IETF vs. the IRTF and
> whether there are sufficient people to work in the area is not clear.
>  
> Alia

If we did have a useful requirements or framework, then it might be
useful to evaluate some new approach against it, but IMO we don't.

I don't think this is even material ready for IRTF consideration, but
that is up to Tony and IRTF to decide.

Curtis

---

From: Tony Li <[email-masked]>
Date: Thu, 7 Feb 2013 15:02:08 -0800

> that is up to Tony and IRTF to decide.

As always, the IRTF does not require well-formed ideas as a pre-requisite for starting work.  Instead, there needs to be a clear problem statement (i.e., a charter) and a sufficient group of serious researchers who are committed to following through in a committed manner.

Note that it is, by definition, research.  It is expected to have a non-trivial likelihood of failure.

Regards,
Tony

---

From: Mingui Zhang <[email-masked]>
Date: Fri, 8 Feb 2013 07:09:38 +0000

Hi Tony and Curtis,

Let me retrospect some history to clear up the chaos floating around. 

On Feb 4th, I organized a conference call among a dozen of guys (from operators, vendors and Universities) who are interested in PANET, in order to prepare for a side meeting in IETF. There was a rough consensus that a side-meeting is a good way to call for interest, coordination and contribution to PANET. However, requests was _prematurely_ sent out afterwards, without including all the effort from the conference attendees. I hope this email can get us back on track. So I change its title.

I believe the most efficient way for us is to play the game according to the rule of IETF/IRTF. Therefore, we are trying to find a place to accommodate the side-meeting. We should not delve into discussion of specific solutions before we make clear the problem scope. A charter should be published, which will help us figure out what is the problem we are trying to solve. Before that, let me add some pointers of drafts from my side. I believe these drafts can also help people to figure out the scope of PANET. 
Power-Aware Networks (PANET): Problem Statement, https://datatracker.ietf.org/doc/draft-zhang-panet-problem-statement/
Use Cases for Power-Aware Networks, https://datatracker.ietf.org/doc/draft-zhang-panet-use-cases/
Requirements for Power Aware Network, https://datatracker.ietf.org/doc/draft-dong-panet-requirement/

On the 84th IETF rtgwg f2f meeting, there were two presentations relevant to greening. I also list them as follows FYI. 
A Framework and Requirements for Energy Aware Control Planes, http://tools.ietf.org/html/draft-retana-rtgwg-eacp-00, [slides] http://www.ietf.org/proceedings/84/slides/slides-84-rtgwg-3.pptx
Power-aware Routing and Traffic Engineering: Requirements, Approaches, and Issues, https://datatracker.ietf.org/doc/draft-zhang-greennet/, [slides] http://www.ietf.org/proceedings/84/slides/slides-84-rtgwg-4.pptx

Thanks,
Mingui Zhang
Huawei Technologies

>Tony Li
>Sent: Friday, February 08, 2013 7:02 AM
>To: [email-masked]
>As always, the IRTF does not require well-formed ideas as a pre-requisite for
>starting work.  Instead, there needs to be a clear problem statement (i.e., a
>charter) and a sufficient group of serious researchers who are committed to
>following through in a committed manner.
>

---

From: Mingui Zhang <[email-masked]>
Date: Fri, 8 Feb 2013 08:17:00 +0000

Hi Thomas,

>	The problem with the model is that it makes assumptions that just do not
>reflect the reality of much of the equipment that is deployed today. For instance,
>while there is a lot of power used in the line card optics, the idle power supply
>capacity is another giant waste of power. This cannot be mitigated without
>shutting down the entire device.  Further considerations need to be made
>around the back-up facility power as well, which while not directly influenced by
>network traffic, does influence overall power utilization.

Yes, network traffic does not directly influence the power consumption of some facilities such as air conditioners. Then energy conservation for these facilities is clearly out the scope of GreenTE (or PANET). It may as well be out of the scope of IETF.

We focus on how to coordinate network devices through the control/management plane _protocols_ to achieve energy conservation. That's the problem scope.

>	Yes, but if the equipment is already very power efficient, the law of
>diminishing returns (or unexpected consequences) may dominate the equation.

I know, they are most power efficient when packets get through at their peak rate. But, when they are idle (i.e., no packets are being forwarded), the power efficient is awkwardly very low. Line-cards can consume 80% of the MAX power even when they do not forward any packet.

So, power efficient of a single device does not lead to power efficient of whole network, right?

Thanks,
Mingui Zhang
Huawei Technologies

---

From: Hannes Gredler <[email-masked]>
Date: Fri, 8 Feb 2013 09:25:14 +0100

tony,

agree that saving power is a worthwhile goal;

in fact existing hardware technology is making that happen today by
e.g. automatically shutting down unused lookup engines, CPU cores, memory banks etc.
when there is low processing demand.

the part where i am not yet convinced is that additional off-peak "optimization" of
infrastructure links by e.g. computing a routing mesh which only uses
70% of the nominal links does actually give much power savings.

note that line cards which are running at 70% have already throttled down their
power consumption - so what is the point emptying the link and loading another ?
appears to me a zero sum game.

my concern about the core (and SP edge) is not about business or technology -
it is more about if we try to optimize an already optimized (and solved) problem.

/hannes

---

From: Tony Li <[email-masked]>
Date: Fri, 8 Feb 2013 00:48:02 -0800

> it is more about if we try to optimize an already optimized (and solved) problem.

If there are no more optimizations to be had, then talking about it isn't going to hurt anything.  

The point of emptying links is the ability to turn off NPUs (50-100W ea.), turn off edge optics (2-20W ea.) and most importantly turn off long-haul optics (100-400W ea.).  Yes, I realize that the latter part is really problematic because DWDM systems are balanced with all lambdas active, and that disabling any one of them can cause issues with other lambdas.  

Still, it would be nice to understand what CAN be done, if only to feel like we've truly explored every corner in the solution space and exhausted every option.

Tony

---

From: Mingui Zhang <[email-masked]>
Date: Fri, 8 Feb 2013 08:53:32 +0000

Hi Hannes,

"Optimization" tries to empty all links on one line-card with priority, then this line card can be shut off to save the high baseline power.

Thanks,
Mingui Zhang
Huawei Technologies

>Hannes Gredler
>Sent: Friday, February 08, 2013 4:25 PM
>To: Tony Li
>> I'll just point out that this argument that you and Eric are espousing is skirting
>dangerously close to the quagmire of business.  And we know from long
>experience that the IETF does not do business models.
>>
>> I'd like to strongly suggest that we simply restrict ourselves to the goal of
>saving power.  I think that we can agree, in general, that saving power is a
>worthwhile goal.  As to whether or not it is significant or makes economic
>sense is very much an issue that should be left to the operator community to
>decide.  Limited overall savings may be worthwhile in one context and pointless
>in another.
>>
>> I know of one country where they are purportedly mandating power reductions.
>In such situations, saving that last watt is the difference between a fine and not.
>On the other hand, in a situation where power is very cheap, it's obviously silly.
>>
>> Let's not argue about the marginal value of energy.  That's a business model
>issue.  Let's talk about how technology can actually save power.
>>
>> Regards,
>> Tony
>>
>>
>
>

---

From: Hannes Gredler <[email-masked]>
Date: Fri, 8 Feb 2013 10:17:16 +0100

mingui,

wouldn't designing the hardware architecture such that there is
no high baseline rather than a more demand based
power draw curve be the better alternative then ?

only box local changes needed, immediate returns,
no need to bother the network with the fact that the local
box cannot do the power housekeeping well ?

/hannes

---

From: Mingui Zhang <[email-masked]>
Date: Fri, 8 Feb 2013 09:33:13 +0000

Hannes,

That's the dream of line-card hardware manufactures. If the hardware is 100% power proportional, then PANET will become useless.

However, the practice is that only 10%-15% power proportional is possible, which means a high 85%-90% of baseline power is being wasted when there is few traffic.

Thanks,
Mingui

>From: Hannes Gredler [mailto:[email-masked]]
>Sent: Friday, February 08, 2013 5:17 PM
>To: Mingui Zhang
>Cc: Tony Li; Shankar Raman M J; [email-masked]
>> "Optimization" tries to empty all links on one line-card with priority, then this
>line card can be shut off to save the high baseline power.
>>
>> Thanks,
>> Mingui Zhang
>> Huawei Technologies
>>
>>> the part where i am not yet convinced is that additional off-peak
>"optimization"
>>> of
>skirting
>reductions.
>model

---

From: Balaji venkat Venkataswami <[email-masked]>
Date: Fri, 8 Feb 2013 17:26:10 +0530

Dear Curtis,

Comments inline.... <balajivenkat>

On Fri, Feb 8, 2013 at 3:27 AM, Curtis Villamizar <[email-masked]> wrote:

>
> In message <
> [email-masked]>

<balajivenkat> Most modern commercial chipsets provide the capability of
switching off TCAM banks.
TCAMs nowadays come in the form factor of multiple banks that can be
switched off and switched on
with register operations. I wouldnt say that this is uncommon. This
algorithm we suggest falls within
the realm of software design where the software is responsible for managing
multiple TCAM banks of
suitable granularity. Its because such chips are available that these
algorithms are feasible today.

> TCAM.  It is fast, but a radix trie based approach or other tree

<balajivenkat>
Most vendors and commercial chipsets manufactured use TCAMs for the Longest
prefix match lookup.
Fast is good so TCAMs are in widespread use.

Radix trees too could be used. But then again we could optimize the SRAM
they use by applying
similar approaches to reducing the radix tree to hold only routes that are
in use. So it is the
general philosophy that is of importance and does not hinge on usage of
TCAMs in routers.

> This pair of drafts in particular should be dropped.

Hence we think this work is of importance in the current context.
This is currently work in progress.

thanks and regards,
balaji venkat

> > > Are you assuming that minimizing the traffic across a node also
> minimizes
> > > the power that node draws?
> > >
> > > If yes:
> > > > The root of our position is not what you have stated. We do not say
> if
> > > you dont
> > > > use a link you consume less power. We are not for avoiding links but
> we
> > > are for

---

From: John E Drake <[email-masked]>
Date: Fri, 8 Feb 2013 13:26:42 +0000

http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PALL&p=1&u=%2Fnetahtml%2FPTO%2Fsrchnum.htm&r=1&f=G&l=50&s1=7162572.PN.&OS=PN/7162572&RS=PN/7162572

Irrespectively Yours,

John

From: [email-masked] [mailto:[email-masked]] On Behalf Of Balaji venkat Venkataswami
Sent: Thursday, February 07, 2013 8:22 AM
To: Eric Osborne (eosborne)
Cc: Shankar Raman; [email-masked]
Subject: Re: Power aware networks : Comments requested from routing community

Dear Eric,

The URL to the draft is http://tools.ietf.org/html/draft-mjsraman-panet-tcam-power-efficiency-00

A follow up draft that defines TCAM efficiency in terms of a metric is also outlined in...

With the help of the above schemes and additional work that is in progress we intend to reduce the
power footprint of a device/router/switch/chassis such that important power guzzling components of
the router are optimized.

In summary, the above 2 drafts coupled with the protocol schemes will we feel result in better power
saving than what we have today. So essentially optimizing the power consumed within a device
coupled with using protocol schemes between devices will result in saving power.

Hope this helps.

thanks and regards,
balaji venkat

On Thu, Feb 7, 2013 at 9:38 PM, Eric Osborne (eosborne) <[email-masked]<mailto:[email-masked]>> wrote:
[size trimmed]

If yes:

eric

> From: Balaji venkat Venkataswami [mailto:[email-masked]<mailto:[email-masked]>]
> Cc: Shankar Raman; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>

---

From: "Eric Osborne (eosborne)" <[email-masked]>
Date: Fri, 8 Feb 2013 14:53:24 +0000

Will this side meeting be in Orlando?  I'm sceptical of some of the current approaches but am very interested in where they might go.

eric

> Sent: Friday, February 08, 2013 2:10 AM
> To: Tony Li; [email-masked]
> Cc: Susan Hares; [email-masked]
> Subject: PANET side-meeting
> 
> Hi Tony and Curtis,
> 
> On Feb 4th, I organized a conference call among a dozen of guys (from
> operators, vendors and Universities) who are interested in PANET, in order to
> prepare for a side meeting in IETF. There was a rough consensus that a side-
> meeting is a good way to call for interest, coordination and contribution to
> PANET. However, requests was _prematurely_ sent out afterwards, without
> including all the effort from the conference attendees. I hope this email can get
> us back on track. So I change its title.
> 
> I believe the most efficient way for us is to play the game according to the rule
> of IETF/IRTF. Therefore, we are trying to find a place to accommodate the side-
> meeting. We should not delve into discussion of specific solutions before we
> make clear the problem scope. A charter should be published, which will help
> us figure out what is the problem we are trying to solve. Before that, let me
> add some pointers of drafts from my side. I believe these drafts can also help
> people to figure out the scope of PANET.
> Power-Aware Networks (PANET): Problem Statement,
> https://datatracker.ietf.org/doc/draft-zhang-panet-problem-statement/
> Requirements for Power Aware Network,
> https://datatracker.ietf.org/doc/draft-dong-panet-requirement/
> 
> On the 84th IETF rtgwg f2f meeting, there were two presentations relevant to
> greening. I also list them as follows FYI.
> A Framework and Requirements for Energy Aware Control Planes,
> http://tools.ietf.org/html/draft-retana-rtgwg-eacp-00, [slides]
> http://www.ietf.org/proceedings/84/slides/slides-84-rtgwg-3.pptx
> Power-aware Routing and Traffic Engineering: Requirements, Approaches, and
> Issues, https://datatracker.ietf.org/doc/draft-zhang-greennet/, [slides]
> http://www.ietf.org/proceedings/84/slides/slides-84-rtgwg-4.pptx
> 
> Thanks,
> Mingui Zhang
> Huawei Technologies
> 
> 
> >From: [email-masked] [mailto:[email-masked]] On Behalf
> >Of Tony Li
> >As always, the IRTF does not require well-formed ideas as a
> >pre-requisite for starting work.  Instead, there needs to be a clear
> >problem statement (i.e., a
> >charter) and a sufficient group of serious researchers who are
> >committed to following through in a committed manner.
> >

---

From: "Alvaro Retana (aretana)" <[email-masked]>
Date: Fri, 8 Feb 2013 15:41:38 +0000

No, there won't be a side meeting in Orlando.

However, we secured more time (than the regular 90 min) for rtgwg so that we can have a discussion there.

Alia and I will send out more details (objectives, ground rules, etc.) soon.

Thanks!

Alvaro.

Sent from my iPhone

---

From: "Fedyk, Donald (Don)" <[email-masked]>
Date: Fri, 8 Feb 2013 16:22:16 +0000

I too think this is a research area.  

I will comment on the metric though. Path computation on multiple metrics is hard so we usually minimize one metric subject to a set of constraints.  Is a power metric a primary minimization metric? I would bet if you asked providers the answer is no. So it would be compute a minimum path based on Bandwidth or delay subject to a set of equipment that qualifies as "green".  Or prefer green links if possible. 

How do you determine Green links?
The PWR metric proposed has two items: Power per XGIG and Available Utilization (draft-mjsraman-panet-ospf-power-topo-00)

I'd argue that Power per XGIG has to be measured and verified on an equal scale across vendors (so it is unlikely to be populated by vendors in any easily agreed manner ;-) ) and it is related to utilization (Higher utilization = more power/bit).
So if you divide by available utilization, the goal of the metric seems to be to produce an average power per XGIG. 
So the metric is roughly a constant per port.

Comparing all the link metrics in the network some set would qualify as green,  others perhaps yellow or red. 

So an easy way to get 90% of the value of this is Measure your equipment. Color the best ports Green (using some threshold), and have a path computation that prefers green links all other things being equal. 

This puts the onus on Operators to measure power and color links (administrative group (color) exists) and requires a path computation that can prefer green. A requirement that is softer than must be green.

Just saying..
Don 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Mingui Zhang
Sent: Friday, February 08, 2013 4:33 AM
To: Hannes Gredler
Cc: Shankar Raman M J; [email-masked]
Subject: RE: Power aware networks : Comments requested from routing community

Hannes,

Thanks,
Mingui

>wouldn't designing the hardware architecture such that there is no high 
>baseline rather than a more demand based power draw curve be the better 
>alternative then ?
>
>only box local changes needed, immediate returns, no need to bother the 
>network with the fact that the local box cannot do the power 
>housekeeping well ?
>
>/hannes
>
>>> Behalf
>Of
>>> infrastructure links by e.g. computing a routing mesh which only 
>>> uses 70% of the nominal links does actually give much power savings.
>>>
>>> note that line cards which are running at 70% have already throttled 
>>> down
>their
>>> my concern about the core (and SP edge) is not about business or 
>>> technology
>-
>>> it is more about if we try to optimize an already optimized (and 
>>> solved)
>problem.
>>>
>>> /hannes
>>>
>>>>> Do you think that optimizing a part of the network which gives 
>>>>> only limited overall savings is a worthwhile goal ?
>>>>
>>>>
>>>> Hannes,
>>>>
>>> dangerously close to the quagmire of business.  And we know from 
>>> saving power.  I think that we can agree, in general, that saving 
>>> power is a worthwhile goal.  As to whether or not it is significant 
>>> or makes economic sense is very much an issue that should be left to 
>>> the operator community to decide.  Limited overall savings may be 
>>> worthwhile in one context and
>pointless
>>> in another.
>>>>
>>> In such situations, saving that last watt is the difference between 
>>> a fine and
>not.
>>> On the other hand, in a situation where power is very cheap, it's 
>>> obviously
>silly.
>>>>

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 08 Feb 2013 12:25:51 -0500

In message <[email-masked]>
Mingui Zhang writes:
 
> operators, vendors and Universities) who are interested in PANET, in
> order to prepare for a side meeting in IETF. There was a rough
> consensus that a side-meeting is a good way to call for interest,
> coordination and contribution to PANET. However, requests was
> _prematurely_ sent out afterwards, without including all the effort
> from the conference attendees. I hope this email can get us back on
> track. So I change its title.

I could find no record of this in a google search.  Nothing that
seemed relevant for "PANET meeting", "PANET conference call", or
"Power Aware Networking" other than IETF chatter and the already
discussed academic papers.  Is there any reference to this meeting?

An closed ad-hoc conference call does not justify a new WG, even an
IRTF WG.

Would you care to say who attended the conference call and whether
anyone took notes that could be shared?

Was this just among the authors of the docs sent to RTGWG?  [If so, it
would seem more like a Huawei internal meeting with a few guests.]

> I believe the most efficient way for us is to play the game according
> to the rule of IETF/IRTF. Therefore, we are trying to find a place to
> accommodate the side-meeting. We should not delve into discussion of
> specific solutions before we make clear the problem scope. A charter
> should be published, which will help us figure out what is the problem
> we are trying to solve. Before that, let me add some pointers of
> drafts from my side. I believe these drafts can also help people to
> figure out the scope of PANET.  Power-Aware Networks (PANET): Problem
> Statement,
> Use Cases for Power-Aware Networks,
> https://datatracker.ietf.org/doc/draft-zhang-panet-use-cases/

The usual way to do this formally in IETF is to request a BOF.
Informally you can ask for a room or just get together at the bar (the
venue of choice for an informal ad hoc BOF a decade or more ago).

> On the 84th IETF rtgwg f2f meeting, there were two presentations
> relevant to greening. I also list them as follows FYI.  A Framework
> and Requirements for Energy Aware Control Planes,
> Power-aware Routing and Traffic Engineering: Requirements, Approaches,
> and Issues, https://datatracker.ietf.org/doc/draft-zhang-greennet/,
> [slides]

At IETF, no activity at all on the WG mailing list usually means not
enough interest and therefore no advancement.

I mentioned earlier on the prior thread that I had just quickly read
draft-retana-rtgwg-eacp-00 and did not think this work was worth the
IETF (or IRTF, if I can speak up in advance) in its current form.  The
topic might be worth considering, but the work seems to take a naive
approach.  Whether treating interfaces and links as a black box and
using routing provides any appreciable gain in real networks remains
to be proven.

These documents are trying to write requirements, build a framework,
and specify protocols, all based on unproven assertions.  

If the work goes to IRTF, the first order of business must be to
provide compelling proof or at least compelling argument that a
specific approach will have benefit in *real* networks, not hypothetic
networks, and not real topologies with a hypothetic network
architecture (ie: choice of type of equipment and build out).

Curtis

> > problem statement (i.e., a charter) and a sufficient group of serious
> > researchers who are committed to following through in a committed
> > manner.
> >

---

From: Tony Li <[email-masked]>
Date: Fri, 8 Feb 2013 09:41:29 -0800

> architecture (ie: choice of type of equipment and build out).

I think that this sets the bar just a bit too high.

Again, the first thing to tackle is a clear problem statement and demonstrating that there is a community willing to take this on and see it through.

As Curtis is suggesting, showing that a solution is possible would be a very good thing.  Spending time arguing and finding that we are inherently fighting the laws of physics is simply not productive.

However, in showing that a solution exists, we not be constrained to existing, deployed and off-the-shelf constraints.  Again, the IRTF does research and must be open-minded enough to consider alternate possibilities, above and beyond what already exists today.  If it implies new router architectures, new network architectures, or new optimization techniques, then that should be on the table.  To be sure, they must be feasible and practical, but they need not be instantiated today.  

Yours in research,
Tony

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 08 Feb 2013 12:44:13 -0500

In message <CAHF4apMcWCY0oKo9PYWwP4iAp=[email-masked]>
Balaji venkat Venkataswami writes:
 
> <balajivenkat> Most modern commercial chipsets provide the capability
> of switching off TCAM banks.  TCAMs nowadays come in the form factor
> of multiple banks that can be switched off and switched on with
> register operations. I wouldnt say that this is uncommon. This
> algorithm we suggest falls within the realm of software design where
> the software is responsible for managing multiple TCAM banks of

You seemed to have missed the point.  TCAMs are power hungry.
External TCAMs use power in the logic that runs in parallel, and in
the SERDES to go off-chip on both sides.  Getting rid of the external
TCAM reduces SERDES power to zero.  Using an alternate to TCAM
eliminates that power hungry peice of logic.

> <balajivenkat> Most vendors and commercial chipsets manufactured use
> TCAMs for the Longest prefix match lookup.  Fast is good so TCAMs are
> in widespread use.

Quick fix, but power hungry.  That is why some designs are now
revisiting the idea of external TCAM.

> Radix trees too could be used. But then again we could optimize the
> SRAM they use by applying similar approaches to reducing the radix
> tree to hold only routes that are in use. So it is the general
> philosophy that is of importance and does not hinge on usage of TCAMs
> in routers.

The active power of a large SRAM is a small fraction of the active
power of a large TCAM.  For a large SRAM, the active power is so low
that the leakage power becomes more significant.  That leakage power
is constant whether the SRAM is active or not.  If the SRAM is being
used for route lookup.  the small active power comes into play.

External TCAM was an easier solution back in the 130nm days and
probably up to about the 65nm or 40nm days because, at least back in
the 130 nm days, putting the forwarding lookup on chip was
unthinkable.

There has been a lot of work in higher radix lookups to reduce the
number of pipeline stages, or the amount of parallelism needed, and
these simply use a small number of low active power SRAM lookups
rather than one power hungry TCAM lookup.

If only IGP routes and MPLS LSP are present, then the small IGP lookup
on a small subset of traffic (however it is done), and the MPLS ILM
lookup (a single SRAM lookup) would yield lower power.  Even keeping
the power hungry external TCAM but providing a separate SRAM for ILM
would be a means to lower the power needed for route lookup.

> This is currently work in progress.

I disagree for reasons cited above.  You are free to continue to
design your routers with TCAM if you prefer.

Even if this work had merit, which I do not think it does, IETF has
never written standards covering ASIC design techniques and forwarding
hardware design and there is currently no plans to do so.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 08 Feb 2013 15:13:20 -0500

In message <[email-masked]>
Tony Li writes:
 
> The point of emptying links is the ability to turn off NPUs (50-100W
> ea.), turn off edge optics (2-20W ea.) and most importantly turn off
> long-haul optics (100-400W ea.).  Yes, I realize that the latter part
> is really problematic because DWDM systems are balanced with all
> lambdas active, and that disabling any one of them can cause issues
> with other lambdas.
>  
> Still, it would be nice to understand what CAN be done, if only to
> feel like we've truly explored every corner in the solution space and
> exhausted every option.
>  
> Tony

Tony,

It may be that the industry is already doing what they can but for a
completely different reason that CO2 emissions.

The reason is getting more capacity in not much more space and power.
Network capacity grows exponentially, about a decimal order of
magnitude in 5-8 years is a reasonable estimate (circa 2000 up to 10G
IP core links, circa 2012 approaching 1T IP core links, transport
capacities of 3.2T to 8T per fiber).

Each generation of equipment replaces a row of racks with a rack or
less of the same capacity at far less power per bit.  As each
generation ages, the row of racks fills again and the cycle repeats.
The older stuff used to be redeployed further out from the core in
edge, but I'm not sure if that occurs as much anymore (the older stuff
may simply be retired now).  This is true of IP/MPLS and transport.

Given the goal of replacing existing space and power footprint with
5-10 times the capacity, it is space density and peak power density
that has been the focus.  The key question after "do we need to look
at leasing more space or relocating the facility" has been "are the
power feeds and the AC going to need to be upgraded and is that
feasible".

Operating power has been a secondary concern.  Operating power is a
secondary concern, but it dictates the power bill, and therefore is a
big part of the OpEx equation.  Secondary doesn't mean ignored, and
given both big power bills and potential political presure due to
having multiple facilities requiring 100s of KW the problem is not
being ignored.  OTOH - the datacenters of content providers and the
Internet exchanges, requiring many MW of power have even more
incentive to think about operating power.

A useful question is, if equipment vendors and providers placed a
greater focus on operating power, how much could be gained and what
are the best means of doing so.

The question is really not whether in theory a routing or TE based
approach can make gains in a hypothetic network, it is whether it is
worthwhile approach given how networks are actually built and where
the power actually goes.  This is assuming a solution before
understanding the problem, and IMHO it is starting off with a poor
solution.

Anyone who thinks the industry does not look at this problem, can look
to past NANOG presentations.  Most have focused on data centers and
IXP.  At least one specifically on core routers, but appealing to
relax NEBS temperatures and sustained PPS rates to reduce power
http://www.nanog.org/meetings/nanog54/presentations/Wednesday/Wobker.pdf
This has good info on slides 11 and 12.  An interesting point is on
slide 10 "cooling 1 slot ~= cooling ALL slots" (reason is airflow past
cooling fins has to be maintained at a given rate per ambient temp).

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 08 Feb 2013 16:10:36 -0500

In message <[email-masked]>
Mingui Zhang writes:
 
> Hannes,
>  
> That's the dream of line-card hardware manufactures. If the hardware
> is 100% power proportional, then PANET will become useless.
>  
> However, the practice is that only 10%-15% power proportional is
> possible, which means a high 85%-90% of baseline power is being wasted
> when there is few traffic.
>  
> Thanks,
> Mingui

Is this a measured value, and if so, measured on what type of
equipment?  That seems to me to indicate very high leakage power in an
NPU or ASIC if the NPU and ASIC is 50% or more of board power.

Did you mean to say that an idle card runs at 85-90% of the power that
it would run at with a full packet load?

AFAIK only certain types of logic blocks like SRAM have high ratio of
leakage to active power.  Plain old logic gates have a high active to
leakage ratio and for a given logic block you need to know what
percentage of the gates tend to be in an active state at any given
time.  TCAM, when in use, has a very high active power because almost
all the gates, except those in disabled banks, go active when doing a
TCAM lookup.  I'm not an ASIC designer, far from it, but I've reviewed
the results of ASIC designers going through this power estimation
exercise.  If I remember correctly, active power was higher than
leakage overall for NPU and other ASIC used in routers and transport.

Active power is important enough that chips draw more power and run
hotter when testing with small packet loads that with a larger average
packet size (the so called Intenet mix or IMIX testing).  I remember
having a conversation with a chip vendor about the likelyhood of
anyone doing a continuous 40 byte packet test at max NEBS temps and
whether the intersection of those two requirements was needed.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 08 Feb 2013 16:42:00 -0500

In message <[email-masked]>
John E Drake writes:
> 
> http://patft.uspto.gov/netacgi/nph-Parser?Sect1=3DPTO1&Sect2=3DHITOFF&d=3DP=
> ALL&p=3D1&u=3D%2Fnetahtml%2FPTO%2Fsrchnum.htm&r=3D1&f=3DG&l=3D50&s1=3D71625=
> 72.PN.&OS=3DPN/7162572&RS=3DPN/7162572
>  
> Irrespectively Yours,
>  
> John

John,

Thanks.  A better TCAM.  If anything the bashing of conventional TCAM
in the background reinforces what I said about TCAM being power pigs.

  There are many disadvantages with the conventional ternary CAM
  structure, however. Since each cell contains two memory cells, and a
  mask-and-compare circuit, implementation of a table of size
  w.times.2.sup.n requires w.times.2.sup.n+1 memory elements, and
  w.times.2.sup.n mask-and-compare circuits. Since every lookup in the
  table requires the activation of all the cells, power consumption is
  proportional to w.times.2.sup.n. For large values of n, the cost is
  considerable, and the power consumption is prohibitive.

I'm quite sure radix trie is still more power efficient.

A 4 bit radix trie (16 entry tables doing four binary radix trie
operations at once) requires a maximum of 8 SRAM lookups, seldom more
than 6, and often 4 or less.  This can be pipelined using separate
SRAM blocks per radix trie stage.  Other highly creative lookup
techniques exist.

Curtis

---

From: John E Drake <[email-masked]>
Date: Fri, 8 Feb 2013 21:50:51 +0000

Curtis,

What I heard before joining Juniper was that they were quite happy that Cisco made this acquisition because they don't use TCAM and they thought Cisco wasted a lot of money on a suboptimal technology. 

Irrespectively Yours,

John

> From: Curtis Villamizar [mailto:[email-masked]]
> Sent: Friday, February 08, 2013 1:42 PM
> To: John E Drake
> Cc: Balaji venkat Venkataswami; Eric Osborne (eosborne); Shankar Raman;
> [email-masked]
> <[email-masked]
> .outlook.com>
> John E Drake writes:
> >
> > http://patft.uspto.gov/netacgi/nph-
> Parser?Sect1=3DPTO1&Sect2=3DHITOFF&
> > d=3DP=
> >
> than 6, and often 4 or less.  This can be pipelined using separate SRAM
> blocks per radix trie stage.  Other highly creative lookup techniques
> exist.
> 
> Curtis

---

From: John E Drake <[email-masked]>
Date: Fri, 8 Feb 2013 21:54:07 +0000

Hi,

My apologies.  Please excuse what I meant to be a private comment.

Irrespectively Yours,

John

> Of John E Drake
> Sent: Friday, February 08, 2013 1:51 PM
> To: [email-masked]
> What I heard before joining Juniper was that they were quite happy that
> Cisco made this acquisition because they don't use TCAM and they
> thought Cisco wasted a lot of money on a suboptimal technology.
> 
> Irrespectively Yours,
> 
> John
> 
> 
> > Cc: Balaji venkat Venkataswami; Eric Osborne (eosborne); Shankar
> > Raman; [email-masked]

---

From: Tony Li <[email-masked]>
Date: Fri, 8 Feb 2013 14:42:58 -0800

Curtis,

As you and I both know, there is a huge diurnal cycle to traffic, plus macro cycles at the weekly period.  Even with local power optimization in place, our experience with traffic engineering alone suggests that there are further, global optimizations that are possible.

Again, I think that constraining our thinking to what is 'worthwhile' today, at current power rates is shortsighted.  We need to understand what can be done, what the returns are, and only then, let various operators judge as to whether it is of commercial interest to them or not.

Tony

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 08 Feb 2013 20:23:32 -0500

In message <[email-masked]>
Tony Li writes:
 
> As you and I both know, there is a huge diurnal cycle to traffic, plus
> macro cycles at the weekly period.  Even with local power optimization
> in place, our experience with traffic engineering alone suggests that
> there are further, global optimizations that are possible.
>  
> Again, I think that constraining our thinking to what is 'worthwhile'
> today, at current power rates is shortsighted.  We need to understand
> what can be done, what the returns are, and only then, let various
> operators judge as to whether it is of commercial interest to them or
> not.
>  
> Tony

Tony,

I agree with that you said.

What I was trying to get at was:

  1.  The discussion should be based on realistic assumptions.
      Discussions not based on realistic assumptions often turn out to
      be a waste of time.

  2.  The solution space should be within the range of deployable
      options.

Discussions on many topics in the research world too often fails to
adhere to these.  The IRTF Routing Requirements WG is a good example,
with the occasional proposals to completely replace routing with pet
ideas and inability to stick to the deployable solution space.

Curtis

---

From: Tony Li <[email-masked]>
Date: Fri, 8 Feb 2013 17:31:36 -0800

Curtis,

You're correct, we're generally in agreement except that we have completely different expectations of what the words 'realistic' and 'deployable' mean in a research context vs. an engineering context.

I propose that we agree to disagree.

Regards,
Tony

---

From: Mingui Zhang <[email-masked]>
Date: Sun, 10 Feb 2013 13:28:14 +0000

Hi Curtis,

Comments in-line below.

Thanks,

Mingui

________________________________________
From: Curtis Villamizar [[email-masked]]
Sent: Saturday, February 09, 2013 1:25
To: Mingui Zhang
Cc: Tony Li; [email-masked]; [email-masked]; Susan Hares
Subject: Re: PANET side-meeting

[zmg] We never argue this is a conference call justifies anything. No commitment is promised either.

[zmg] I did record the conference (web and audio). However, the recording is currently shared only among guys who are interested in pushing PANET _forward_. As I said, they are from more than one communities. I will talk with other attendees whether they feel it is proper to announce their names. Anyway, you can come to attend the side-meeting to grasp what was going on.

[zmg] The situation was totally different as what you speculated.

[zmg] Yes, you are right. Our target was a side-meeting (bar-BoF) and we are trying to request a room.

[zmg] I'd like to hear your _specific_ feedbacks on other drafts.

[end of comments]

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Mon, 11 Feb 2013 11:18:32 -0500

> [zmg] I'd like to hear your _specific_ feedbacks on other drafts.

I have better things to do with my time than provide detailed review
of drafts that are ill conceived and so do most of the people on the
mailing list.

Below is the list of drafts with comments on whether they are relevant.

    draft-mjsraman-panet-bgp-power-path
    draft-mjsraman-panet-ecmp-redirect-power-repl-cap
    draft-mjsraman-panet-inter-as-power-source
    draft-mjsraman-panet-inter-as-psp
    draft-mjsraman-panet-inter-as-psp-protect
    draft-mjsraman-panet-pce-power-mcast-replic
    draft-mjsraman-panet-pim-power
    draft-mjsraman-pce-power-replic
    draft-mjsraman-panet-intra-as-psp-te-leak
    draft-mjsraman-panet-ospf-power-topo

      These are all founded on the premise that there are significant
      gains that can result from routing that is aware of power
      disipation differences in hardware related to the traffic load
      placed on the hardware.  The topic of the relevance of these
      drafts to RTGWG has just been discussed and the concensus seems
      to me to be that these are not relevant to RTGWG.  The
      discussion so far has been to move all of this work to IRTF if
      IRTF wants to pursue this topic.

    draft-mjsraman-panet-tcam-power-efficiency
    draft-mjsraman-panet-tcam-power-ratio

      Alreadty discussed on list.  Theee TCAM drafts are ill conceived
      and technically flawed.  They should be dropped completely.

There may be merit to doing work in this area but you are going about
it entirely the wrong way.  IMO you need to start with a BOF proposal
with a brief problem statement and a charter to address the problem.
No solution should be mentioned in the initial charter, with a
detailed problem statement draft, requirements draft and framework
draft.  The outcome of this initial exercise would dictate what sort
of solutions, if any, to pursue.  For example, if any work is done, it
may be best to start with IGP and inter-domain MPLS since they can be
deployed in a single AS.

Multicast right now contributes a much smaller percentage of load than
unicast, so it makes sense to do IGP first.  For BGP you can always
use DFA (just declared historic, my point being that overloading
global BGP with new inter-AS metrics is going to be a "hard sell").

> architecture (ie: choice of type of equipment and build out).

My main point stands.  IMO You are out of line to submit nearly a
dozen drafts to RTGWG on topics for which research is needed to
determine if there is any merit to the approach.

Curtis

---

From: "Fedyk, Donald (Don)" <[email-masked]>
Date: Mon, 11 Feb 2013 16:51:15 +0000

I agree with Curtis on this.  As I said in another thread. It is not clear to me even if operators wanted to do Power aware routing why we would need the extensions proposed in some of these documents. As a vendor I'm not likely to keep dynamic power levels per port. 

Don      

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Curtis Villamizar
Sent: Monday, February 11, 2013 11:19 AM
To: Mingui Zhang
Cc: [email-masked]; Susan Hares
Subject: Re: PANET side-meeting

> [zmg] I'd like to hear your _specific_ feedbacks on other drafts.

I have better things to do with my time than provide detailed review of drafts that are ill conceived and so do most of the people on the mailing list.

There may be merit to doing work in this area but you are going about it entirely the wrong way.  IMO you need to start with a BOF proposal with a brief problem statement and a charter to address the problem.
No solution should be mentioned in the initial charter, with a detailed problem statement draft, requirements draft and framework draft.  The outcome of this initial exercise would dictate what sort of solutions, if any, to pursue.  For example, if any work is done, it may be best to start with IGP and inter-domain MPLS since they can be deployed in a single AS.

Multicast right now contributes a much smaller percentage of load than unicast, so it makes sense to do IGP first.  For BGP you can always use DFA (just declared historic, my point being that overloading global BGP with new inter-AS metrics is going to be a "hard sell").

> architecture (ie: choice of type of equipment and build out).

My main point stands.  IMO You are out of line to submit nearly a dozen drafts to RTGWG on topics for which research is needed to determine if there is any merit to the approach.

Curtis

---

From: Thomas Nadeau <[email-masked]>
Date: Mon, 11 Feb 2013 19:29:49 +0000

From: Balaji venkat Venkataswami <[email-masked]<mailto:[email-masked]>>
Date: Thursday, February 7, 2013 8:52 AM
To: Hannes Gredler <[email-masked]<mailto:[email-masked]>>
Cc: Shankar Raman M J <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: Power aware networks : Comments requested from routing community

Dear Hannes,

The last thing that many people want is for service providers to be able to control the equipment that is inside their homes, which includes the CPE device which they may in fact own.    However, if you want to go down that route, you might want to check out the HomeNet WG.

b) The problem space that the IETF can take up can be only in the edge and core devices in the ISP networks and the Campus/DC/Metro etc...Assume that I have n devices today with the same bandwidth offering but differing power consumption footprint, is there a way today to automatically send traffic through low power paths in the topology of these devices.

Again, checkout HomeNet.

--Tom

thanks and regards,
balaji venkat

/hannes

---

From: Mingui Zhang <[email-masked]>
Date: Tue, 12 Feb 2013 06:06:36 +0000

Hi Curtis,

One clarification: when I said the "other drafts", I actually meant the following drafts. 

Requirements for Power Aware Network, https://datatracker.ietf.org/doc/draft-dong-panet-requirement

Thanks,
Mingui

________________________________________
From: Curtis Villamizar [[email-masked]]
Sent: Tuesday, February 12, 2013 0:18
To: Mingui Zhang
Cc: [email-masked]; Tony Li; [email-masked]; Susan Hares
Subject: Re: PANET side-meeting

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 12 Feb 2013 11:49:55 -0500

In message <[email-masked]>
Mingui Zhang writes:
 
> https://datatracker.ietf.org/doc/draft-dong-panet-requirement
>  
> Thanks,
> Mingui

Mingui,

OK.  Comments below.  Do to other obligations, I have to go through
these quickly but they are so far off the mark that it is OK to go
through them quickly.

The underlying flaw in all of this is that you have not done the
fundamental research into where power goes and where gains can be
made.  That exercise needs to be done for each of the type of domain
within scope: core, edge, aggregation, access, enterprise, data
center.  The problem statement should have a clear statement of which
of these are within scope and an analysis of where power goes for each
within scope, and where gains can be made for each within scope.

You can't have a good requirements document and framework until you
have a good problem statement, so don't bother to write them until you
have done the underlying research and have a good problem statement.

Curtis

Power-Aware Networks (PANET): Problem Statement
draft-zhang-panet-problem-statement

  Intro could be shorter.  We all know that power is an issue for
  numerous resasons.  Money is an issue and that CO2 is an issue.  In
  the past fitting higher capacity into the same space and power has
  been the dominant issue.

  A problem statement should not have solution approaches.  This
  narrows the solution space unnecessarily too early in the process.
  The choice of solution approaches is poor and ignores the many types
  of equipment drawing power.

  There is no mention of the scope of the work.  Is it core networsk,
  edge, aggregation (metro), access, enterprise, data center, or all
  of the above.  Different solutions will apply to difference
  situations.

  For example, does putting a router link into sleep mode affect the
  400W laser in the transport equipment?  If transport lasers are
  turned off, how long do they need to restabilize (hint: minimum for
  ULH is usually minutes to get power levels right, stabilize
  temperatures within the actively cooled lasers, etc.  Can take tens
  of minutes before fully stabilized during which time periods of
  errors may occur).

  A good engineering approach is to start by figuring out where most
  of the power goes, and focus on where large gains can be made.  That
  is what *should* go into a problem statement.

  Problem Areas for IETF outlines what should be in this document.
  You could take the bullet list and try to make that into a WG BOF
  rather than offer this as a problems statement.

  If you are going to write a problem statement draft, address the
  questions under Motivation and Problem Scope.

  The bullets in the Technical Development and Operation Practice
  belong in requirements and framework documents, filled in, not
  unanswered.

  A proper "problem statement" in IETF is a clear and concise
  statement of a well defined problem to be solved.  You don't have
  that.

Use Cases for Power-Aware Networks
https://datatracker.ietf.org/doc/draft-zhang-panet-use-cases/

  This document is useless as a use cases document.  It mostly
  reiterates the solution set proposed in the problem statement.

  Instead of anecdotal examples a good analysis of where power goes in
  core, edge, aggregation (metro), access, enterprise, data center and
  where power could be potentially reduced would make for a better use
  case document if that wasn't in the problem statement.

Requirements for Power Aware Network
https://datatracker.ietf.org/doc/draft-dong-panet-requirement

  The requirements makes the assumption that a solution involving
  changes in the control plane will be found necessary.  Since we
  don't have the research to support that, this document is premature
  and should be ignored for now.

  It is entirely possible that some 90% of the gains could come from
  local power optimizations alone and the remaining 10% requires
  enough complexity that it is not worth it.  We don't know that until
  the underlying research into where the power goes and where we can
  make gains is done.

---

From: Susan Hares <[email-masked]>
Date: Wed, 13 Feb 2013 16:36:24 +0000

Eric, Minqui and Curtis:

+1 on interest.  Whether we get a IETF room or not - I'd like to really chat about this topic.

Sue 

-----Original Message-----
From: Eric Osborne (eosborne) [mailto:[email-masked]] 
Sent: Friday, February 08, 2013 6:53 AM
To: Mingui Zhang; Tony Li; [email-masked]
Cc: Susan Hares; [email-masked]
Subject: RE: PANET side-meeting

eric