Subject: RE: [Ipoverib] Please read - proposed WG termination

From: "H.K. Jerry Chu" <[email-masked]>
Date: Thu, 1 Sep 2005 11:19:23 -0700 (PDT)

[co-chair hat off]

...
<snip>

>These performance problems are primarily implementation-specific and have 
>little to do with IB technology itself.  In addition, nearly all IB 
>solutions use a 2KB not the smallest MTU to transfer data - no different 
>than Ethernet.

Ethernet is adopting jumboframe to get more firing power. Where is IB's
equivalent of jumboframe?

>As I and others have raised over the years, the enablement 
>of IP over IB to perform well is a local HCA issue not a standards 
>issue.  Addition of checksum off-load support to the HCA is rather trivial 
>and does not require standardization (this is what is done for Ethernet 
>today and is non-standard).  Addition of large send off-load support is a 
>local HCA issue not a standards issue and effectively provides the same 
>benefit as connected mode.

Yes LSO (or TSO as some call it) is relatively easy. But LRO (large receive
offload) is a heck more difficult. IB connected transports already have all
silicons to do it. Why not just use it?

>The use of multiple QP to spread work across 
>CPU for both send / receive ala the multi-queue support I've worked with 
>various Ethernet IHV to get in place is again a local HCA issue (does not 
>have to be visible as part of the layer 2 address resolution).  One can 
>construct a very nice performing IP over IB solution but there hasn't been 
>much public progress to implement these de facto capabilities found in 
>Ethernet solutions on IB.  Getting these into a HCA implementation is a 
>heck of a lot easier and faster to do than to develop a standard and 
>getting all of the OS changes made (the HCA implementation issues can all 
>be done underneath the IP stack just like with Ethernet so no real OS impacts).

I don't understand the large MTU issue to the OS (requiring continguous physical
addresses). Aren't all decent hardware capable of scatter/gather these days?

What's more hairy to the OS stack is the per-destination MTU and different
MTU for multicast than for unicast inherited in IPoIB CM.

Jerry

>
>
>>For commercial clusters, if IB is used for storage, then you save a network
>>by having fast IP performance and can use the IB network for both. Why use
>>IB and another network for the commercial cluster, when the other network
>>supports similar bandwidth for storage and IP.
>
>There will always be Ethernet in any cluster so the fabric is there.  The 
>question is whether it is just for low-bandwidth / management services or 
>for applications.  For storage, need to separate the discussion into 
>whether it is block or file.  For block, IB gateways to Fibre Channel, etc. 
>can and are being used today quite nicely.  Performance is reasonable and 
>the ecosystem costs, target availability, customer "pain", etc. are much 
>lower than attempting to move to native IB storage.  The same applies to 
>file based where IB gateways to Ethernet which then attaches to file 
>servers works quite nicely.  In fact, the original vision of IB was that of 
>an I/O fabric to create modular server solutions.  The addition of IPC came 
>later in the process when it was found to be relatively low cost to 
>define.  So, IB is successful in the HPC world and slowly entering some 
>commercial solutions.  To state that its future relies on getting an IP 
>over IB RC solution is perhaps blowing it a bit out of proportion.   The 
>easier path for all is to simply use the techniques I and others have 
>advocated for years now and solve the problems within the HCA 
>implementation.  Much lower costs and will result in delivering a good 
>performance solution.
>
>BTW, RNIC / Ethernet solutions implement these techniques today.  With the 
>arrival of 10 GbE and the lower prices of RNIC and 10 GbE switch ports, 
>lower latency switches (competitive enough with IB for commercial and many 
>HPC clusters), etc. the success of IB must lie elsewhere and not on an IETF 
>spec.   This was noted at the recent IEEE Hot Interconnects conference as 
>well so isn't just my opinion.
>
>Mike
>
>>Implementing IPoIB-CM makes IB viable in the HPC cluster and some
>>commercial clusters. Otherwise I don't think it competes economically with
>>other network technologies.
>>
>>Regards.
>>
>>Bernie King-Smith
>>IBM Corporation
>>Server Group
>>Cluster System Performance
>>[email-masked]    (845)433-8483
>>Tie. 293-8483 or wombat2 on NOTES
>>
>>"We are not responsible for the world we are born into, only for the world
>>we leave when we die.
>>So we have to accept what has gone before us and work to change the only
>>thing we can,
>>-- The Future." William Shatner
>>
>>
>>
>>              Dror Goldenberg
>>              <[email-masked]
>>              o.il>                                                      To
>>              Sent by:                  [email-masked],
>>              ipoverib-bounces@         "H.K. Jerry Chu"
>>              ietf.org                  <[email-masked]>
>>                                                                         cc
>>                                        [email-masked],
>>              08/30/2005 09:32          [email-masked],
>>              AM                        [email-masked]
>>                                                                    Subject
>>                                        RE: [Ipoverib] Please read -
>>                                        proposed WG termination
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> > From: Vivek Kashyap [mailto:[email-masked]]
>> > Sent: Tuesday, August 30, 2005 8:39 AM
>> >
>> > On Mon, 29 Aug 2005, H.K. Jerry Chu wrote:
>> >
>>
>>
>><snip>
>>
>>
>> > > 1. IPoIB connected mode draft-ietf-ipoib-connected-mode-00.txt
>> > > updated recently
>> >
>> > Well, in recent days there has been a discussion going on
>> > based on Dror's input. I also made some updates after some
>> > discussion on OpenIB (not on
>> > IETF though).  This draft itself became a working group draft
>> > this february
>> > after some lively discussion just before that.  It appears to
>> > me that we
>> > should be possible to finalise this draft soon enough.
>> >
>> > 20th sept. might be long enough to know one way or the other...
>> >
>> > vivek
>> >
>>
>>
>>We would like to see IPoIB-CM being finalized in IETF. We see
>>great value in having a standard for connected mode which effectively
>>increases the MTU. We are willing to contribute to the standardization
>>effort. We're also looking at the implementation of IPoIB-CM in Linux.
>>
>>
>>-Dror _______________________________________________
>>IPoverIB mailing list
>>[email-masked]
>>https://www1.ietf.org/mailman/listinfo/ipoverib
>>
>>
>>
>>
>>
>>_______________________________________________
>>IPoverIB mailing list
>>[email-masked]
>>https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Michael Krause <[email-masked]>
Date: Thu, 01 Sep 2005 12:04:30 -0700

>equivalent of jumboframe?

Jumboframe causes QoS issues for multiple flows.  The whole reason for the 
variable MTU was to allow people to configure the different virtual lanes / 
paths to meet QoS concerns.  Examination of the hardware-based SAR provided 
by IB illustrates that there is no definitive performance difference 
between using a 2KB per and say a larger 9KB MTU.  BTW, examination of many 
IP workloads shows that majority of the units of work exchange fit quite 
nicely in a 2KB payload.  Those that are larger, are easily SAR without 
impacting either side of the communication, i.e. requiring a connection to 
be established.

Keep in mind that Ethernet is also doing other activities to solve various 
issues - multi-path support to compensate for the spanning tree 
limitations, conceptually similar virtual lanes are being investigated, 
improved security, link-level congestion management, etc.  All of these 
changes combined with de facto use of large send / receive-side scaling, 
etc. will likely make the issue of 9KB rather moot except for bulk data 
transfers in the WAN (even then the QoS issues of multiple flows across a 
common link will likely limit the use of 9KB).

In the end, IB HCA implementations can adopt the same techniques as 
Ethernet and deliver the same performance as a connected model without the 
overhead / complexity and ecosystem cost of a new specification.  IB 
already has enough problems getting the basics into the world 
today.  Adding in yet another spec isn't going to solve these problems or 
make life easier - too many things within the ecosystem are impacted.

>silicons to do it. Why not just use it?

Large receive off-load isn't that hard - really an implementation problem 
from most of the designs I've examined.  I helped implement something 
equivalent to TSO and LRO over 10 years ago and it wasn't that bad or 
expensive.  As for why not just use it?  Simple, cost to specify, 
implement, get adopted within the industry is already high for IB.  Adding 
more to the stack of things when there is questionable benefit compared to 
simply re-using the existing Ethernet infrastructure seems like a poor 
choice of resource expenditure.  Of course, most people only care about 
making Linux happen these days so I guess feel free to knock yourselves out 
if that is all that is required.

>addresses). Aren't all decent hardware capable of scatter/gather these days?

Nothing above indicates any issue regarding physical addresses.  IB 
hardware as does Ethernet comprehends V-to-P mapping and makes no 
requirements on it being physically contiguous beyond the base physical 
page size.

>MTU for multicast than for unicast inherited in IPoIB CM.

Agreed.  This is one aspect of the additional complexity that is simply 
easier to avoid.   The cost / benefit of connected mode is quite 
questionable compared to leveraging the existing Ethernet implementation 
concepts and associated infrastructure.

Mike

---

From: Vivek Kashyap <[email-masked]>
Date: Thu, 1 Sep 2005 17:58:30 -0700 (PDT)

On Thu, 1 Sep 2005, Michael Krause wrote:

> by IB illustrates that there is no definitive performance difference between 
> using a 2KB per and say a larger 9KB MTU.  BTW, examination of many IP 
> workloads shows that majority of the units of work exchange fit quite nicely 
> in a 2KB payload.  Those that are larger, are easily SAR without impacting 
> either side of the communication, i.e. requiring a connection to be 
> established.

NFS will happily use 32K block size and it is the default on some NFS configs.
There will be other examples though I agree that a large workload set is 
probably satisfied with the 2KB MTU. That is why IPoIB-CM portion is proposed 
as an optional, interoperable solution for networks that need it.

> issues - multi-path support to compensate for the spanning tree limitations, 
> conceptually similar virtual lanes are being investigated, improved security, 
> link-level congestion management, etc.  All of these changes combined with de 
> facto use of large send / receive-side scaling, etc. will likely make the 
> issue of 9KB rather moot except for bulk data transfers in the WAN (even then 
> the QoS issues of multiple flows across a common link will likely limit the 
> use of 9KB).
>
> In the end, IB HCA implementations can adopt the same techniques as Ethernet 
> and deliver the same performance as a connected model without the overhead /

however connected mode is already available to us in all HCAs. Why not use it
as it is ?

> complexity and ecosystem cost of a new specification.  IB already has enough 
> problems getting the basics into the world today.  Adding in yet another spec 
> isn't going to solve these problems or make life easier - too many things 
> within the ecosystem are impacted.
>
>
> expensive.  As for why not just use it?  Simple, cost to specify, implement, 
> get adopted within the industry is already high for IB.  Adding more to the 
> stack of things when there is questionable benefit compared to simply 
> re-using the existing Ethernet infrastructure seems like a poor choice of 
> resource expenditure.  Of course, most people only care about making Linux 
> happen these days so I guess feel free to knock yourselves out if that is all 
> that is required.
>
>
> Nothing above indicates any issue regarding physical addresses.  IB hardware 
> as does Ethernet comprehends V-to-P mapping and makes no requirements on it 
> being physically contiguous beyond the base physical page size.
>
>

Even as it is now a subnet can be set up with a common MTU and leave the
'per-connection mtu' to implementations that want to use it. Such 
setups have existed before in many stacks to support different MTU 
token-rings for example.

The WG discussed making the IB  mode a link characteristic i.e. 
IPoIB-UD and IPoIB-CM connectivity would be through a  router.
The WG, at the time chose (see late 2004/early 2005 threads) a common subnet for
both. However, if we resurrect the separate subnet proposal a common MTU
becomes easier across any given subnet.

Multicast will stay limited to the physical MTU however. It is a tossup 
between using a different mechanism for multicasting or using what the medium 
provides. Also it is easier to make the unicast to multicast distinction in 
a stack. e.g. BSD stacks by default used to limit multicast to physical MTU
whereas allowed IP packets to be larger than it (and those were fragmented
in the case of ethernet).

Vivek

> easier to avoid.   The cost / benefit of connected mode is quite questionable 
> compared to leveraging the existing Ethernet implementation concepts and 
> associated infrastructure.
>
> Mike
>
>
>> Jerry
>> 
>> >
>> >
>> >>              o.il> 
>> To

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Fri, 02 Sep 2005 09:48:26 +0200

> MTU for multicast than for unicast inherited in IPoIB CM.

I don't quite understand why this should be an issue.
At the moment, TCP/IPv4 stacks have 3 options:

- Set MTU to 576 bytes (not b* likely)
- Set MTU to physical interface MTU and depend on fragmentation in the 
network to save their bacon in the case of smaller MTUs (*curse* PPPoE)
- Set initial MTU to physical interface MTU, set DF bit, and support Path 
MTU discovery

I *think* all "modern" IP stacks take the last choice. Which means that 
they have built-in support for variable MTU already.
Setting the initial MTU for the connection depending on info from the 
underlying infrastructure retrieved before connection startup may be a bit 
hairy. But there SHOULD be quite a bit of "variable MTU" support there 
already.

                    Harald

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Fri, 02 Sep 2005 09:42:07 +0200

It seems that the question of closure of the WG has elicited far more 
active discussion on the CM spec than the revised CM spec actually did :-)

But since this is far afield from the original subject, I've changed the 
subject line.

--On 1. september 2005 12:04 -0700 Michael Krause <[email-masked]> wrote:

> Ethernet and deliver the same performance as a connected model without
> the overhead / complexity and ecosystem cost of a new specification.  IB
> already has enough problems getting the basics into the world today.
> Adding in yet another spec isn't going to solve these problems or make
> life easier - too many things within the ecosystem are impacted.

Actually the IETF has a fairly long and robust tradition of standardizing 
stuff that MAY be useful in a very limited context, and later turn out not 
to be terribly useful at all - for a list of some candidates for that 
honor, see draft-lear-newtrk-decruft-experiment-01.

The important questions to ask are, I think:

- What will be the cost of implementing be to willing participants who want 
to implement the extension?
- What will the cost of implementing be to participants who do not wish to 
implement the extension?

If the first one is "high", that may not be a show-stopper; people who want 
to try are welcome to, and they hurt nobody but themselves.

If the second one is "high", or there's a risk that it's high, we shold be 
VERY careful, and perhaps run official experiments before releasing it on 
the standards track - this is traditionally done, for instance, on 
extensions to TCP like ECN, or on extensions to routing protocols.

I can see the argument that for a particular implementation, the cost of 
implementing IPoIB-CM with a large MTU may be high - but since it seems a 
facility that's relatively easy to negotiate for, I don't quite see that it 
has a large ecosystem impact on IPoIB-UD implementations.
If it's a Bad Idea, it may get published, and wither away like so many 
other "proposed standards" that nobody cares about any more.

OTOH, if it's really a Good Idea, the choice is not really between a 
standard and no IPoIB-CM; it's between a standard and multiple, possibly 
incompatible IPoIB-CM variants. Nobody can prevent people from implementing 
from the current draft, assigning their own service-ID(?) and just shipping.

(Of course, there's the "markitecture" problem that some vendors will want 
to claim to have implemented "all relevant standards" - the problem that 
reportedly led some vendors to advertise support for "IP over avian 
carriers" at one point.... but that's a problem that the IETF can't solve, 
and probably shouldn't solve, I think.)

Thoughts?

                     Harald

---

From: Mark Townsley <[email-masked]>
Date: Fri, 02 Sep 2005 15:21:01 +0200

Harald Tveit Alvestrand wrote:

> --On 1. september 2005 11:19 -0700 "H.K. Jerry Chu" 
> <[email-masked]> wrote:
>
> - Set initial MTU to physical interface MTU, set DF bit, and support 
> Path MTU discovery
>
> I *think* all "modern" IP stacks take the last choice. Which means 
> that they have built-in support for variable MTU already.

Many do, but then ICMP is filtered with reckless abandon on the Internet 
(big I), breaking the feedback loop for a perfectly good PMTU enabled 
stack. MTU must still hand-managed *way* more than it should be.

- Mark

> underlying infrastructure retrieved before connection startup may be a 
> bit hairy. But there SHOULD be quite a bit of "variable MTU" support 
> there already.

- Mark

>
>                    Harald
>
>
>
>------------------------------------------------------------------------
>