Subject: RE: [Ipoverib] Please read - proposed WG termination

From: Dror Goldenberg <[email-masked]>
Date: Fri, 2 Sep 2005 23:02:46 +0300

-----Original Message-----
From: Michael Krause [mailto:[email-masked]]
Sent: Thursday, September 01, 2005 10:09 PM 

At 10:51 AM 9/1/2005, Vivek Kashyap wrote:

On Thu, 1 Sep 2005, Roland Dreier wrote:

   Bernard> In the HPC world most clusters use the cluster fabric (
   Bernard> and IB is the future direction ) for both MPI and IP
   Bernard> traffic. The IP traffic is usually for parallel file
   Bernard> systems and system management and control. This high
   Bernard> bandwidth IP network is required in most production HPC
   Bernard> clusters.  With the current IPoIB only using UD, the
   Bernard> performance is dismal. Our simulations using the small
   Bernard> packet MTU of IB says that the parallel file systems (
   Bernard> GPFS, PVFS, Lustre etc ) can only get 25% of a 4X IB link
   Bernard> today and at 12X it will be about 10%.

It's not clear to me that IPoIB-CM is really the answer here.  First
of all, for Linux to take advantage of a 64K MTU, major surgery to the
network stack will be required.  Production systems have trouble even
enabling 8K jumbo frames, because the kernel can't allocate two
physically contiguous buffers to use for receive.  Given that no

That is an implementation issue in Linux and shouldn't effect the protocol.

Linux does support large send off-load today so it is not clear what the
issue is you are speaking of above. 
 
[dg]  Large send offload solves the send side, but doesn't help on the
receive side. ipoib-cm solves both ends. 
Roland has a point that in Linux, it might be more complicated to support
>4KB frames. But I agree that this is an OS specific issue. In this specific
case, solving it may also be beneficial for other applications such as
Ethernet jumbo frames (if it makes any difference). 

current IB hardware can offload TCP/IP checksums for multi-packet RC
messages, it's not clear how feasible this network stack surgery is.

Checksum offload is not part of the specification but an HCA feature.

It is also not that hard to implement.
 
[dg] Sounds right for ipoib-ud. For ipoib-cm csum offload implementation is
complicated. By the time you already transmitted 64KB, you need to go and
find the first packet (which is probably already in your peer' memory) and
place the csum there...  
 Anyway , I still see great value in ipoib-cm even without the csum offload.
The main reason is that to my knowledge, in Linux, large send and zero copy
are only supported for sendfile(). The rest of the operations are on the
copy path anyway. Calculating csum while copying, is not a big deal.

 -Dror 

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Michael Krause <[email-masked]>
Date: Fri, 02 Sep 2005 15:41:55 -0700

At 01:02 PM 9/2/2005, Dror Goldenberg wrote:
>Roland has a point that in Linux, it might be more complicated to 
>support >4KB frames. But I agree that this is an OS specific issue. In 
>this specific case, solving it may also be beneficial for other 
>applications such as Ethernet jumbo frames (if it makes any difference).

I think there is confusion perhaps here.  Most OS can map arbitrarily large 
buffers and provide V-to-P mappings that enable an I/O device to DMA large 
and variable amounts of data.  The 4KB limit is generally associated with 
the physical page size of many OS (though that too varies with some OS 
supporting up 1GB) as well as the physical maximum unit of DMA which is 4KB 
(PCI / PCI-X / PCIe).   Most device drivers / devices can also handle a SGL 
for a given target DMA address and sans separating on 4K address 
boundaries, can handle large DMA quite nicely.

The more interesting issues are:
- Must a large receive be handled as a single unit of work and thus must 
progress up the network stack as a single "message".  This has QoS and 
performance implications.  Obviously, iWARP and IB can perform such 
operations since that is the basis for RDMA as well as basic SEND 
operations so the basic hardware infrastructure is present.  The same can 
be said for various Ethernet implementations which support jumbo frames.

- Is the network stack limited in anyway in terms of message buffer size 
and therefore requires the implementation to post a set of message buffers 
which may be also SGL in order to receive the larger network sends?  This 
is where some of the complexity lies.

- Is there any impact to the recovery paths when a checksum, etc. requires 
only a portion of the large message to be retransmitted?  SACK, etc. are 
generally implemented but some implementations may make assumptions as the 
granularity of how the buffers are handled.

- etc.

>[dg] Sounds right for ipoib-ud. For ipoib-cm csum offload implementation 
>is complicated. By the time you already transmitted 64KB, you need to go 
>and find the first packet (which is probably already in your peer' memory) 
>and place the csum there...
>  Anyway , I still see great value in ipoib-cm even without the csum 
> offload. The main reason is that to my knowledge, in Linux, large send 
> and zero copy are only supported for sendfile(). The rest of the 
> operations are on the copy path anyway. Calculating csum while copying, 
> is not a big deal.

It is true that copy-n-checksum is a de facto method to mitigate the 
performance impacts of not having checksum off-load support. Yet, there are 
many workloads where copy operations do not occur thus the performance hit 
is a big deal.  I don't think one can simply say IB connected provides 
superior performance compared to implementing something like a large send 
off-load, etc.

BTW, the new Sockets API extensions (async execution, event management, 
explicit memory management, etc.) enable more zero-copy opportunities as 
well as OS bypass.  It is important to keep in mind that an evaluation of 
whether something is of value needs a bit more context and future usages in 
mind.  In general, it is not clear why we are debating this so 
much.  People can define a spec and as we have seen  on multiple occasions 
either all, only a subset or none of the industry may implement even if the 
technology is superior in some facet.  If people really want to do this, 
then feel free to define but I suspect it might only appear on one OS at 
the end of the day and while that OS is relatively high volume for some 
usage models, that does not mean the spec will be deployed in practice or 
that it will be viewed as successful.  I personally do not believe the 
industry needs this problem solved as there are sufficiently good methods 
already available to solve the performance issues including using native 
RDMA ULP where performance is critical.

Mike