Subject: RE: [Ipoverib] Comments on IPoIB Connected Mode Connection Establ ishment

From: Dror Goldenberg <[email-masked]>
Date: Sun, 11 Sep 2005 08:23:41 +0300

Sounds like we agree that active/active is very difficult and will be really
hard
to define given how IB service ID and communication manager work. Therefore,

I think that we should go for the active/passive with the ability to reject
at the 
application (ipoib) level.

For the case of more than one connection between two peers, we first need to
decide whether we want to allow it (now or in the future). If we disallow
it, then
it's very simple, for each crossing request we need to decide which side
should
accept and which side should reject it (e.g. GID compare). 

If we want to allow  more than one connection in the future, then maybe we 
should think of embedding a "channel index" into the private data. This way,

the two peers can add the channel index on which they are transmitting.
BTW, it'll take more than that to enable more than one channel, for example,
we should try to avoid reordering of IP datagrams that can happen when more 
than one channel is being used to send them.

What about connection termination ? This should probably be added to the RFC
draft. I can think of two cases where one would want to terminate a
connection.
1) When the ipoib driver goes down; 2) Inactive connections can be taken
down
to reduce resource consumption.

-Dror

> -----Original Message-----
> From: Vivek Kashyap [mailto:[email-masked]] 
> Sent: Tuesday, September 06, 2005 10:40 AM
> To: Dror Goldenberg
> Cc: IPoverIB
> Subject: RE: [Ipoverib] Comments on IPoIB Connected Mode 
> Connection Establ ishment
> 
> 
> On Sun, 21 Aug 2005, Dror Goldenberg wrote:
> 
> >
> >
> >> From: Vivek Kashyap [mailto:[email-masked]]
> >> Sent: Thursday, August 18, 2005 9:36 AM
> >>
> >> On Wed, 17 Aug 2005, Dror Goldenberg wrote:
> >>
> >>>
> >>>
> >>>> From: Vivek Kashyap [mailto:[email-masked]]
> >>>> Sent: Monday, August 15, 2005 9:41 AM
> >>>>
> >>>> On Wed, 10 Aug 2005, Dror Goldenberg wrote:
> >
> > <snip>
> >
> >>
> >>>>> (2) Active/Active Model Support
> >>>>>
> >
> >
> > <snip>
> >
> >>>> 3. In case of the active-active connection the request from the 
> >>>> numerically smaller MAC Address (QPN+GID or probably just
> >> the GID) is
> >>>> terminated. The error code 'state connection' is a
> >> possibility here.
> >>>>
> >>>> The active-active situation is known when it is determined
> >> that there
> >>>> is an outstanding request to the remote peer (as determined by
> >>>> GID+QPN) while a request from the same is received on local 
> >>>> GID+ServiceID.
> >>>
> >>> In IB, active-active is only determined by equal Service 
> IDs. See IB 
> >>> spec 1.2 vol1 12.9.7.1, the Active CM state machine goes to 
> >>> Peer-Compare state based solely on the Service ID. 
> Therefore, if you 
> >>> want to have active-active, then you must
> >> ensure that
> >>> both service IDs are identical. In the current draft, the
> >> Service IDs are
> >>> different, as they include a remote QP number.
> >>>
> >>
> >> Yes, if we are determining it based on the CM. However, I was 
> >> suggesting the above as what the ipoib component will do 
> when it is 
> >> informed by CM. It can return a suitable error to CM when it 
> >> determines that a particular
> >> connection shouldn't be allowed. Is that not possible?
> >>
> >
> > That wouldn't be "active/active" per se. But maybe it's 
> just as good, 
> > because it just pushes this model to the ipoib layer. In a case of 
> > crossing connection
> > requests one side is supposed to accept, while the other is 
> supposed to
> > reject. The rules for rejection have to be described in the 
> RFC. Maybe it
> > worth describing the state machine that now needs to be implemented
> > in ipoib.
> 
> yes, need to look into it in detail. My initial thought is that the 
> method of determining if there are connections in flight is 
> based on receiving a request source GID+QPN  while attempting 
> a connection to the same.
> 
> >
> > BTW, what will be the rejection reason ? Seems like the most 
> > appropriate is 28 - consumer reject.
> 
> seems ok to me.
> 
> >
> > Note that it becomes tricky if you want to allow more than one 
> > connection on the same service ID. So, is there a way you can tell 
> > whether your peer is trying to establish the 1st or 2nd connection 
> > while you are trying to set up the 1st one ?
> 
> hmm...good point, unless we take it that any request that 
> crosses is deemed active-active. Once a connection is 
> established further requests can be made on the same 
> Service-ID (or denied depending on the implementation).
> >
> > I think it'll be best if we can use the standard active-active 
> > mechanism.
> 
> We need a common Service-ID for it that must be 
> known/derivable by both the peers. Since masks are not 
> supported by CMs using both QPNs in the 
> service-ID is difficult.
> 
> 
> > However, if we don't manage to find how, then this approach 
> (rejection 
> > at the app level) will do the work. However, it will complicate the 
> > ipoib implementation and will duplicate the active-active 
> work that is 
> > already implemented in the cm.
> >
> > -Dror
> >
> > <snip>
> >
> >
> 
> _______________________________________________
> IPoverIB mailing list
> [email-masked] https://www1.ietf.org/mailman/listinfo/ipoverib
> 
_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Michael Krause <[email-masked]>
Date: Mon, 12 Sep 2005 08:34:54 -0700

At 10:23 PM 9/10/2005, Dror Goldenberg wrote:

>Sounds like we agree that active/active is very difficult and will be 
>really hard
>I think that we should go for the active/passive with the ability to 
>reject at the

The IBTA can develop what is required to fix the SID issues.

>accept and which side should reject it (e.g. GID compare).

A multi-connection model just like Ethernet aggregation solutions deployed 
today enables aggregate performance to be achieved without burdening the 
upper layers.  Flows are predictable, i.e. a given heuristic generally 
insures that IP datagrams flow over consistent paths thus eliminating the 
re-ordering problem.  These heuristics are locally controlled with Ethernet 
today and do not need to be specified.  As a general comment, people should 
review what is already defined and deployed today with aggregation 
technology and see how this might be leveraged for IB.  While Ethernet 
currently limits aggregation to a single switch attachment, that 
restriction will be removed in the future thus matching what IB can do with 
its multi-path support.  Given the desire to minimize how much a customer 
really sees of IB from a stack integration perspective, it would be good to 
see both technologies sharing as much of the infrastructure as possible.

>1) When the ipoib driver goes down; 2) Inactive connections can be taken down

Termination should only occur when an unplumb operation occurs.  This is 
when the network stack is effectively taken down.  The cause of the unplumb 
can be any number of reasons thus it does not matter what the specific 
reason is only that an admin has decided to unplumb the stack.  Note: An 
error, loss of cable, etc. are not unplumb operations so should not impact 
the connection until the IB fabric says communication is no longer possible 
and errors it out.

Mike

> > From: Vivek Kashyap 
> [<mailto:[email-masked]>mailto:[email-masked]]
> > >> From: Vivek Kashyap 
> [<mailto:[email-masked]>mailto:[email-masked]]
> > >>>> From: Vivek Kashyap 
> [<mailto:[email-masked]>mailto:[email-masked]]
> <https://www1.ietf.org/mailman/listinfo/ipoverib>https://www1.ietf.org/mailman/listinfo/ipoverib 
>
> >

---

From: Vivek Kashyap <[email-masked]>
Date: Mon, 12 Sep 2005 12:22:57 -0700 (PDT)

> The IBTA can develop what is required to fix the SID issues.

Mike, probably you can answer this the best. Doesn't the CM consult the 
relevant service (service ID) during the connection process? That is when
REQ is receive it probably checks with the service to see if it is willing to
accept it. And so on through the handshake. If so, active/active can be done.

thoughts?

>> accept and which side should reject it (e.g. GID compare).

I beleive we need to allow more than one connection between two peers. It 
seems possible (assuming I'm not missing something obvious :)) to do it if 
the request is verified against the service (in this case the IPoIB driver 
for the relevant serviceID). We need to determine whether QPN+GID compare 
can be used to fail simultaneous or only temporary fail when the 
implementation allows multiple connections.

> review what is already defined and deployed today with aggregation technology 
> and see how this might be leveraged for IB.  While Ethernet currently limits 
> aggregation to a single switch attachment, that restriction will be removed 
> in the future thus matching what IB can do with its multi-path support. 
> Given the desire to minimize how much a customer really sees of IB from a 
> stack integration perspective, it would be good to see both technologies 
> sharing as much of the infrastructure as possible.
>
>
> Termination should only occur when an unplumb operation occurs.  This is when 
> the network stack is effectively taken down.  The cause of the unplumb can be 
> any number of reasons thus it does not matter what the specific reason is 
> only that an admin has decided to unplumb the stack.  Note: An error, loss of 
> cable, etc. are not unplumb operations so should not impact the connection 
> until the IB fabric says communication is no longer possible and errors it 
> out.

Yes, and also the IB connection is created when there is a need to talk to the
remote node. This will reflect in the IP level address resolution structures
which will be controlled by IP requirements (ARP/ND). The implementation can
choose to terminate the connection when the ND/ARP cache is deemded invalid or 
deleted.