Subject: Re: [Tsvwg] SCTP Socket API

From: Kacheong Poon <[email-masked]>
Date: Tue, 03 Jan 2006 19:25:52 +0800

Randall Stewart wrote:

> We have had a LONG thread discussion of the socket API changes
> that have been suggested. Here is a summary of the changes
> (At least I think) that have been proposed:

Thank you Randy for providing a summary of the long thread.
I didn't follow the thread and I just took a very quick
glance at what have been discussed.  Please let me know if
I've missed some important points in the following reply.

> A) SOCK_DGRAM as an option for the socket creation besides
>    SOCK_STREAM and SOCK_SEQ_PACKET. A SOCK_DGRAM descriptor would
>    be one that:
>    1) Is fully reliable.. aka its NOT UDP, msg's are delivered
>    2) Is atomic, i.e. if I do a read() of 100 bytes and a
>       200 byte message is in queue to me, the data would
>       be truncated and I would loose the other 100 bytes of
>       the message aka no MSG_EOR flag.

Let's not discuss the new socket type for SCTP.  I think we
should first decide if the above feature is actually a
good thing to have in SCTP.  I believe the only valid point
raised in the long thread in support of this feature is
preventing denial of service attack.  The kind of attack is
to send a huge message which is larger than the receive
buffer of the victim SCTP socket so that partial delivery
is triggered and thus blocking the delivery of other messages.
I think some people have already pointed out that it is trivial
to guard against this attack as an app should know how big
the app protocol message is.  So it can just abort the
offending association when it detects such a huge message.
I don't see why this defense is not effective.

The other point supporting this feature is ease of app
programming.  But I actually find this feature confusing
for app programmers.  First of all, I think a well defined
message oriented app protocol should have a limit of how
big a message is.  IMHO, unbounded message does not make
much sense in real world app.  SCTP does not impose a limit
on message size does not necessarily mean that a well defined
app protocol should use unbounded message size.  And if an
app knows how big a valid message is, I believe the current
API draft already provides the so called atomic message delivery
mechanism.  Having the suggested feature is confusing for app
writers.  What is the point of using this feature to only allow
100 bytes message delivery if a correctly written app will
never write more than 100 bytes in a message?  This feature
seems redundant.

The example given for the use of this feature, running SMTP
over SCTP, does not seem right.  If we are going to run
SMTP over SCTP, I guess it is not correct to send the whole
message as one SCTP message.  Is there a technical advantage
for doing it this way?

As some people have pointed out, the semantics also seems
counter intuitive to app writer requiring a reliable message
delivery transport protocol.  To put it this way, this feature
provides a "reliable message delivery service, but sometimes
the message received is not the whole message sent."  It does
not sound right, does it?

Moreover, using this feature does not mean that it is easier
for an app writer to program against the aforementioned DoS
attack.  So instead of checking for partial message using the
current API draft to abort an offending association, it now
needs to check for truncated message.  What is the difference?

> B) A new set/get option, SCTP_MAXMSGSIZE. This option
>    would set a point in a message size where the stack
>    would discard the rest of a fragmented message when
>    the size is exceeded... gee I did not say that very
>    well ... basically the idea is:
> 
>    senderA                                receiverB (
>                                           maxmsg set at 4000)
> 
>    --------TSN=1 B bit set, 1000 bytes----->
>    --------TSN=2,no bits, 1000 bytes------->
>    --------TSN=3,no bits, 1000 bytes------->
>    --------TSN=4,no bits, 1000 bytes------->
>    --------TSN=5,no bits, 1000 bytes------->
>    --------TSN=6,E bit set, 1000 bytes----->
> 
>    Now when receiverB gets TSN 1, 2 and 3, it
>    does its normal thing, but when number 4
>    arrives without the E bit set, it delivers
>    the 4000 bytes to the upper layer, sets some
>    sort of MSG_TRUNC flag, maybe we need a
>    SCTP_MSG_TRUN_TOBIG.. or something?? and then
>    leaves a dead marker in place so that when 5
>    and 6 arrive they are immediately freed and
>    discarded as if they had been read.
> 
>    For the new SOCK_DGRAM the default would be that
>    the SCTP_MAXMSGSIZE is 1/2 the initial
>    socket receive window. For SOCK_STREAM and
>    SOCK_SEQPACKET the default would be 0, which
>    indicates there is NO LIMIT to how big the
>    message would be.

This seems to me to be a change in the protocol, rather
than a suggestion on changing the API draft.  And I don't
really understand the rationale behind this protocol change.
Does it help the stack reduce memory usage?  Probably not, as
it is obvious from the semantics above that the stack can
have the chance the discard packets only if they are arrived
in order, with no drop.  Otherwise, the stack probably does
not have enough info to decide if out of order TSNs can be
dropped or not.  So any attack using large message should
still have the same effect.

Does it help app writer?  Probably not.  As discussed above,
a well defined app protocol should never need such option.
Why does an app protocol want to use SCTP if it does not
want to receive its message intact?  If I have missed some
important points in the long thread, please raise them again.

> C) new socket option SCTP_FRAGMENT_INTERLEAVE. This
>    would be at a minimum a gettable option that would
>    tell the user if the stack interleaves messages
>    from different streams and/or associations. So
>    in other words, if a partial delivery is started
>    on stream 1, and a message arrives on stream 2 that
>    can be delivered. Will the stack send up the message
>    on stream 2 while its waiting for more data (a retransmit
>    say) on stream 1. The user, if the value returned from
>    this is 1, must make sure and read the ancillary data
>    on every message... or else it might have an issue
>    with mixing up a message from stream one vs stream two :-d
>    If the stack allows this option to be set, then it enables/disables
>    this behavior.

SCTP has partial delivery mechanism to avoid a deadlock in the
protocol layer because SCTP supports message delivery.  When a
message is as big as the receive buffer, SCTP has to partially
deliver the message as it is not allowed to consume more than the
receive buffer.  So it only happens if an app sends a message
as big as the buffer size.  How can this happen?  The app
protocol should have control on what to send.  So the partial
delivery mechanism should never be triggered.  In a sense, the
mechanism is there to safe guard against poorly written app.  I
don't see the need to really make it very efficient, hence no
need to have this option.

> D) new socket option SCTP_PARTIAL_DELIVERY_POINT. Note that
>    this was suggested off-list... This value would be a
>    message size that the application would like partial
>    delivery to begin at. Normally the stack decides this
>    based on resources and other hueristics. If this value is
>    set then the application requests that as soon as that
>    much information is received (in order) start delivering the message.
>    Setting this value to 0, tells t the stack that it should use
>    its judgment as to when to start the pd-api.

I don't understand the rational of this option.  It seems to me
that this option is for those app protocols which choose the wrong
message size.  If it wants the receive call to return for
every X bytes, it should only send message of at most X bytes
big.  Using the current API draft, I believe an app protocol
writer already has control on whether partial delivery will be
triggered or not.  So it seems to me that this option is not
really about partial delivery.  Again, if I have missed an
important point why this option is really needed, please raise
it again.

> Now, I think all of the above are reasonable requests... IMO.. and I
> don't have a problem with them being added to the socket api.. however
> I would like to hear your opinion.. especially from those who
> have not joined the discussion and are responsible for the
> stacks out there (Michael, Kacheong, Shridhar .. come to
> mind and are in my address book.. others as well that
> implement SCTP please respond to this :-D)
> 
> If I hear no strong objections... I will see if I can
> wordsmith up something for the next socket-api iteration ;-D

I don't really see the reason why any app writer needs the above
suggested features.  So I object to all of them.

-- 

						K. Poon.
						[email-masked]

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Randall Stewart <[email-masked]>
Date: Tue, 03 Jan 2006 08:15:58 -0500

Kacheong:

> I've missed some important points in the following reply.

Your welcome, Kacheong.. I am glad you have joined the
discussion :-D

I think you have hit all the major points of the discussion .. at
least as I have understood them... and in fact I like the
way you have raised it to a "meta-point" ... I often
get too bound up in trying to make sure we have
the right feature set for the app writer.. without
looking at the higher level :-D

I think I am in agreement with your assessment with
respect to all but <C>... see below..

> need to have this option.

I understand your point here.. but I think there is one
case in which having the FRAGEMENT_INTERLEAVE option might
be a good thing.. for the 1-2-many socket. In this case
you could have two different associations actively
sending data.. and one sends a large message blocking
the other....

Now its true a app could peel off the one giving the
pd-api.. so that might be one solution.. but
one thing that makes me wonder is the MPI folks.

A group up at the Univ of BC in Vancouver have been
working on making MPI run over the SCTP stack on FreeBSD
to do various testing... one of the first things that they
assumed was that the partial delivery worked with inter-leave.
Basically they assumed that they could get other messages
in-between a large pd-api.... which they were using (found
a lot of interesting bugs too I might add :-D)

The fact that they assumed interleave was the way things
work tell me either:

a) It might be a good thing to have..
and
b) At a minimum we need more details on this in the socket
    api document .
and even
c) There may be some SCTP stacks that do this.. I believe
    Brian said his does..

So one would at LEAST want a way to see if the stack
was doing this... aka a read only option that
told you if it did interleave or not.. and for the 1-2-many
model I am not to sure it would not be a good idea to have
this capability..

Of course it would make an application harder to write
since every time it read... if the MSG_EOF is not set
the next read it would have to validate that its part
of the same assoc-id... where-as if its blocked (aka no
interleave) then just reading the rest of the message
until a MSG_EOR would work...

R

-- 
Randall Stewart
803-345-0369 <or> 815-342-5222(cell)

---

From: Mark Butler <[email-masked]>
Date: Tue, 03 Jan 2006 14:16:49 -0700

Kacheong Poon wrote:

>>   SOCK_STREAM and SOCK_SEQ_PACKET. 
>>
>

There is no way for an application to have any idea how an incoming 
message is until the first delivery (partial or otherwise).  By that 
time, the message may have consumed a large portion of a shared buffer.  
That is not the worst problem, however - locked partial delivery is 
fundamentally unsafe on a one-to-many socket.  There is no way to know 
when the rest of the message will arrive, if ever.  Meanwhile nothing 
else gets delivered, likely for several seconds.  If you have a several 
peers over slow connections, a partial delivery on a one-to-many socket 
will block everything else for at least hundreds of milliseconds, 
possibly much longer (depending on the size of the message), even though 
packets are arriving from each as fast as possible.  It is not an 
accident POSIX doesn't even define a non-atomic one-to-many socket 
type.  It is SOCK_DGRAM or nothing, as far as they are concerned. 

>for app programmers.  
SOCK_DGRAM is confusing?  Compared to what we have now?

>First of all, I think a well defined
>mechanism.  
Not in the slightest. 

>Having the suggested feature is confusing for app
Because you cannot control who your peers are, generally speaking.

>message as one SCTP message.  Is there a technical advantage
That is a hypothetical example, given to demonstrate why locked partial 
delivery is a bad idea on a one-to-many socket.  However, there are 
large numbers of applications that have relatively large messages (at 
least 64K) on one to many sockets.  NFS and MPI are good examples.  It 
is not wise to lock up a file server for several seconds at a time.

>not sound right, does it?
Sure it does, if messages larger than some tolerance is an indication of 
an attack or a sender defect.  Is there any real application that is 
going to accept a 3 TB message on a one-to-many socket? Or a 3 GB one? 
How many want to accept 3 MB ones?

SCTP does not provide (nor can it provide) guaranteed reliability in the 
first place.  Only applications can do that, by writing messages to 
redundant durable media and exchanging application level confirmations. 
If not, messages or parts of messages will be lost when the receiver 
crashes.   Any system providing "whole message" reliability has implicit 
or explicit message size limits, because no system has infinite ram nor 
disk space.

Messages *will* be truncated.  That is why there are partial delivery 
abort events.  They are just an unusually obtruse means of signalling 
that fact.  The simple POSIX standard for message truncation signalling 
is MSG_TRUNC.

>needs to check for truncated message.  What is the difference?
Partial deliveries may occur on a one-to-many socket for nearly any 
message size, there is no guarantee - strictly best effort, if at all.  
So you have to receive in a loop like TCP, and keep a running total of 
the message size, and adjust the passed read buffer size accordingly.   
Checking the message size is just the beginning - every one to many 
SOCK_SEQPACKET application must subscribe to partial delivery abort 
events, or it will run messages together.  The only way to fix that is 
for the implemenation to return a zero size read with MSG_EOR and 
MSG_TRUNC set after an aborted partial delivery.  Checking for MSG_TRUNC 
is ridiculously simpler than mandatory event processing.

>>B) A new set/get option, SCTP_MAXMSGSIZE. 
>>    
>>
><snip>
>
If you are going to be strict about it, the protocol has no support for 
one-to-many relationships, period. All this one-to-many stuff is another 
_extensive_ layer added on top of the protocol, which is strictly 
one-to-one. It is basically a ULP implemented at the socket layer.  And 
like any ULP it comes with its own requirements.  And one of those is 
limited message sizes on one-to-many sockets, at least if latency and 
throughput are any consideration whatsoever.

>>C) new socket option SCTP_FRAGMENT_INTERLEAVE. 
>>    
>>
>
>
>as big as the buffer size.  
>
Solving the deadlock problem on a one-to-many socket that way just 
replaces it with a different problem, a very serious one, namely 
indefinite stalling.

>How can this happen?  The app
Again, there is no general way to control who sends messages to you.  
Even if you have perfect control, partial delivery is triggered on a 
regular basis under heavy load, even with very small (<4K) messages,  as 
the Redhat cluster DLM people found out rather surprisingly. (They were 
expecting standard atomic SOCK_SEQPACKET semantics, and ended up having 
to re-insert the same layer they were using previously with TCP).

>>D) new socket option SCTP_PARTIAL_DELIVERY_POINT. 
>>    
>>
>
>
Again, not in the slightest.  The draft makes no guarantees about when 
partial delivery will be started.  Specifying early delivery on a 
one-to-one socket is extremely useful for some applications.  Message 
boundaries are there to convey useful information - if you have to 
insert them willy nilly, you defeat their primary purpose.  Suppose 
someone calls sendfile() on an SCTP socket - should the sendfile() 
handler insert message boundaries every 4K or whatever?  Or what if you 
have plenty of buffer space on the receiving side - should you 
necessarily have to be woken up every 4K just because someone has 
downgraded all the message sizes to support least common denominator 
clients?

What about implementations that can copy directly to user memory? - then 
there is no corresponding socket buffer requirement at all.  Downsizing 
messages is a bad idea on ULPs designed to work over one-to-one 
sockets.  DECnet, XNS, SPX, and Unix SOCK_SEQPACKET sockets all support 
arbitrary record lengths and partial reads.  Certainly if you ever want 
to properly support TCP semantics you need both partial reads and 
partial writes, with early delivery in both directions.

There is a large body of software that could benefit from SCTP if 
standard SOCK_STREAM, SOCK_DGRAM, or SOCK_SEQPACKET semantics were 
supported.  Right now none of them are.  Not only that, they can't be 
emulated effectively at the application level, because deliveries are 
uncontrollable in one or both directions, or require either application 
or socket buffers as large as the message size.  If SCTP can't be used 
with POSIX standard socket semantics, it is likely to remain a specialty 
protocol for obscure telephony applications.  I think it is capable of 
much better than that.

Finally, if complexity is the issue - these proposed options are in the 
direction of the common, the simple, and the expected - *especially* 
with standard socket types.  We have an enormous number of socket 
options providing fine granualarity control over and statistics for 
things that the majority of applications will never need.  Some TCP 
implementations have similar options, and they are rarely necessary.  
Very helpful for some, but neglectable by most.  These extremely simple 
and basic options have much greater general benefit than virtually every 
option implemented so far.  The only reason for their existence is a 
shortage of socket types, and the fact that existing applications depend 
on non-standard semantics for the existing ones.   Ideally, POSIX socket 
semantics should have been the default in the first place, such that one 
would have to use options to get unusual behavior.  Assuming they ever 
get around to it, are they likely to bless default semantics that are 
such an unusual, unexpected, and unnecessary departure from what they 
have already standardized? Especially, if there isn't even an option to 
get normal behavior?

   - Mark

---

From: Randall Stewart <[email-masked]>
Date: Tue, 03 Jan 2006 18:40:44 -0500

A couple of comments.. I will let Kacheong
answer most of these (trying to stay out of
the debate.. :-D)

> is not wise to lock up a file server for several seconds at a time.

I agree.. MPI usually sends Megabyte messages.. to me the
fragment inter-leave helps in this (if using the 1-2-many model)..

> that fact.  The simple POSIX standard for message truncation signalling 

This is arguable.. the pdapi event is a convience only.. one could
easily look at the sndrcvinfo structure that comes up.. assuming
you are doing a sctp_recvmsg OR a recvmsg() with ancillary
data... of course you have to turn on the sndrcvinfo stuff
so in that sense you do need to do something.. but the pdapi
truncation is NOT the only way...

Usually a pdapi trunction is closely followed by an ABORT or
COMM_LOST more correctly... one could come up with the
same thing without it.. its just you would need to subscribe
to other things too... of course if you are doing 1-2-many you
really should be doing association events anyway...

> is ridiculously simpler than mandatory event processing.

No .. see above.. you can use other than the pd-api event to figure
out its been aborted..

> throughput are any consideration whatsoever.

I am not so sure about the above statements.. I think just having
fragment interleave really is a big help..

As to trunctation.. I am ambivilent.. it may be a useful
feature.. I can go either way on it :-)

I will see how your discussion with Kacheong goes ;-)

> to re-insert the same layer they were using previously with TCP).

Hmm.. interesting... I am surprised that at less than 4k you would
get pd-api... I doubt you would see this on FreeBSD.. maybe.. but
I would be surprised if you did....

R
> ------------------------------------------------------------------------
>

---

From: Mark Butler <[email-masked]>
Date: Tue, 03 Jan 2006 19:45:36 -0700

> really should be doing association events anyway...

There is another case - if PR-SCTP is enabled, an implementation can 
receive a pertinent FORWARD TSN after a partial delivery has already 
occurred.  Then the delivery needs to be aborted without aborting the 
association, or PR-SCTP is hazardous to use in real life.  In short 
PR-SCTP implies truncated messages, whenever and wherever partial 
deliveries are allowed.

- Mark

---

From: Randall Stewart <[email-masked]>
Date: Wed, 04 Jan 2006 06:22:34 -0500

> PR-SCTP implies truncated messages, whenever and wherever partial 
True, but all things can be hazardous to use, at least if
they have any power of flexibility.

This is another area where the app designer should think
about what they are doing. If I design a application that
is sending 1Meg PR-SCTP messages then I had best be
prepared for having pd-api's be aborted... and thus
turn on notifications...

The reality of it, however IMO, are that a pd-api
writer is NOT going to send a 1Meg message.. more
like 160 bytes messages ... and should not ever
see any message fragmentation...

R

---

From: Mark Butler <[email-masked]>
Date: Wed, 04 Jan 2006 21:56:25 -0700

> they have any power of flexibility.

I agree.  Rhetorical question: So if we allow senders to abandon 
messages mid transfer for PR-SCTP, resulting in arbitrary truncation at 
any threshold, why is predictable truncation at a specified threshold 
supposed to be a problem?

> see any message fragmentation...

Most of the time yes.  An implemenation may split even relatively small 
messages (ones smaller than the fragmentation point) across two 
different packets for efficiency, especially if it was pre-fragmented at 
too large a threshold for the necessary control chunks.   In general I 
think that a PR-SCTP sender should never unilaterally abandon 
multi-fragment messages mid or post transmission.  If the receiver wants 
to abandon it, that is another story...

- Mark

---

From: Randall Stewart <[email-masked]>
Date: Thu, 05 Jan 2006 05:39:57 -0500

> supposed to be a problem?

Good point... Kacheong... I am still waiting to see
your reply to Mark's original response :-D

> to abandon it, that is another story...

Hmm.. that makes sense... but it really is an implementation choice.
I can't even remember if we (in the bsd world) make that decision.. I
know once one fragment is considered old they all are marked for
skipping.. but I can't remember if this is only pre-ack of a piece...

R

---

From: Kacheong Poon <[email-masked]>
Date: Fri, 06 Jan 2006 00:20:42 +0800

Instead of replying point by point (which will likely
lead to a long thread), let me try to summarize the
reasons posted so far to support what I call "truncated
message" delivery.

1. Partial delivery can happen any time and the app cannot
control it.
2. Partial delivery will block 1-N style socket.
3. Because of 2, partial delivery can lead to DoS attack.
4. "Truncated message" delivery can already happen with
PR-SCTP.

I think the major reason why this thread was brought up
is because of 1.  But in reality, I think this is an
implementation issue rather than an API issue.  To make
it clear, we can easily solve this by saying that if an
app specifies the receive buffer to be X bytes, no partial
delivery will be performed for message smaller than X
(or X - a fixed number of bytes).  This avoids the protocol
deadlock issue.  And an app can control when partial
delivery can happen.  As Randy has pointed out, this
is what implementations should probably do.  Note that
this is different from the SCTP_PARTIAL_DELIVERY_POINT
option.

So if 1 is not an issue, 2 should not be an issue.  The
app can avoid partial delivery altogether if it desires.
As I have mentioned, 3 is not an issue because an app
can just abort the offending association if it receives
an invalid message (in this case, a message too big).
And because an app can control when partial delivery
can happen, it means that even with PR-SCTP, no truncated
message will ever be received if an app desires.

There are other minor points in saying that the current
API is not usable by many apps.  For example, sendfile()
cannot be used efficiently.  But it seems to me that it is
because of app architecture design, not because of the
API itself.  Otherwise, I can't see how there can be many
SCTP apps running in production environment using the draft
API at this very moment.

-- 

						K. Poon.
						[email-masked]

---

From: Mark Butler <[email-masked]>
Date: Thu, 05 Jan 2006 15:15:52 -0700

Kacheong Poon wrote:

>deadlock issue.  

Having a tolerance below which atomic deliveries would always be 
performed is a good thing.  However, basing the tolerance on the receive 
buffer size is problematic.  In many environments setsockopt(SO_RCVBUF) 
silently clips your request to system limits and succeeds.  This is done 
at the socket level, so the protocol driver doesn't know what was 
originally requested.  

With shared buffer implementations, if the tolerance is near the receive 
buffer size, then any comparably sized message still serializes the 
whole socket, stalling as if a partial delivery had been performed.   
Basically, the stack would have to decide which message it is going to 
deliver, renege everything else, advertise a zero window for every other 
association, and so on. A user specified threshold a fraction of the 
shared buffer space would be much more appropriate.  If messages greater 
than or equal to that size are ULP violations, you want to know about it 
as soon as possible.  So we need something like 
SCTP_PARTIAL_DELIVERY_POINT (with slightly altered semantics) regardless.

However, there are also people who want to reliably receive large 
messages on one-to-many sockets.  With locked partial delivery, it can't 
be done.  Locked partial delivery is fundamentally flawed on one-to-many 
sockets - so you either force  the ULP to use small messages, or allow 
fragment interleaving.  Artificially limiting the message size is a bad 
idea for many reasons.  Fragment interleaving is a much better 
alternative for applications like MPI, hence the proposed 
SCTP_FRAGMENT_INTERLEAVE option.

>And an app can control when partial
If we change the specification accordingly...

>message will ever be received if an app desires.
It can only control partial delivery effectively on a shared buffer if 
there is an option to specify what the threshold is.

>SCTP apps running in production environment using the draft
Are there any non-telephony SCTP applications in wide distribution? I 
don't think so.  SCTP is a natural replacement for UDP, but there is no 
SOCK_DGRAM.   It is a natural replacement for TCP, but there are no 
partial writes or early delivery.  It can't be used to multi-stream 
large messages because there is no interleaving.  These are primarily 
API, not protocol limitations.

Even where traditional semantics are not strictly necessary, most 
applications have to work in environments where SCTP support is not 
available or where SCTP is blocked or hindered by firewalls.  An 
application that supports SCTP must generally also support either TCP or 
UDP.  If it is difficult to use the same code base for both protocols, 
an application developer is likely to dispense with SCTP support completely.

The main issue is should SCTP be a general purpose transport protocol or 
not?  I can understand the sentiment to stay focused on supporting small 
messages well, and letting adaptation layers handle larger ones.  I 
would like to see those adaptation layers in-kernel, but that is beside 
the point.

Comments anyone?

 - Mark

---

From: Kacheong Poon <[email-masked]>
Date: Sat, 07 Jan 2006 11:40:17 +0800

> at the socket level, so the protocol driver doesn't know what was

This is an implementation issue, not a specification issue.  So
if an implementation cannot honor a requested buffer size, it
should report an error.  If it silently rejects the request yet
returns no error, I consider this a bug.  And to make it worse,
the above seems to suggest that there is no communication between
the socket layer and the protocol stack.  This seems to be a
serious bug.  And for those apps which need to run on this buggy
stack, I guess a getsockopt() afterwards can be a workaround.
Anyway, I don't think the above implementation issue should
interfere with the specification.

> whole socket, stalling as if a partial delivery had been performed.  

I think the socket draft actually recommends stack not to use a
shared buffer for 1-N style socket because of many difficulties.
It is difficult, but not doable.  The implementation needs to
be very careful.

> SCTP_PARTIAL_DELIVERY_POINT (with slightly altered semantics) regardless.

If we think about it, the above implementation already breaks
an important protocol recommendation, not to shrink the receive
window.  So what the above says is that a receiver A tells a
sender B that its receive window is X.  Then after B sends A 10
bytes, A tells B that the receive window, without ack'ing that 10
bytes, is now much less than X - 10 because a sender of another
association controlled under the same 1-N style socket has sent
64K bytes.  I consider this a bug in the implementation.

So again, this is an implementation issue.

> SCTP_FRAGMENT_INTERLEAVE option.

I don't see why there is a lock in partial delivery for normal
app.  Using the proposed wording change (no code change for most
implementations out there) I made in last email, an app specifies
a receive buffer size of 128KB as it is the app protocol message
size.  Will the app then send a 200KB message just to trigger
partial delivery?

It seems that the above point is that the draft cannot specify
certain things if an implementation chooses not to follow what
the draft specifies nor what the protocol specifies.  The draft
does not artificially limit the message size.  The app chooses
what the message size should be.  And by doing that, the app expects
certain API behavior.  The app has all the controls.

Having said that, if people really want this option, I will not
object to it.  I guess this is what Randy thinks too.

> there is an option to specify what the threshold is.

I guess the above point is that if an implementation chooses to
use a shared buffer for all associations in a 1-N style socket,
it should be very careful in how to manipulate that buffer.
Otherwise, it will run into all sorts of problems and may break
the protocol as mentioned above.

> partial writes or early delivery.  

I will let Randy and other comment on the above.  But I don't
think I have ever heard people who were involved in the SCTP
protocol specification effort make the above claim regarding
TCP/UDP and SCTP.

> It can't be used to multi-stream

As I have mentioned a couple times, this is an implementation
issue.  I don't think we should mix implementation and
specification together.

> an application developer is likely to dispense with SCTP support

I still do not see a strong example about the difficulty in using
SCTP.  It all seems to be an app architecture design issue.  For
example, I read a claim that to use 1-1 style socket, accept() and
then fork() should be done.  This is simply a design choice and is
not true in general.  It seems that there are many "myths" in using
1-1 style socket...  And FWIW, I believe most, if not all, current
TCP apps can be changed to use SCTP with very minor changes or
even just one line change, the socket() call.  I remember Randy
and/or Peter has changed the Apache web server to run on top of
SCTP by making some very minor changes.

And it is an "interesting" claim that most UDP apps can be changed
to use SCTP.  People writing VoIP software using UDP may
experiment with DCCP.  But I don't think they will ever consider
using SCTP for that.  I believe this is applicable to most UDP apps.

> would like to see those adaptation layers in-kernel, but that is beside

Why can't an app utilizing large messages be run on top of SCTP?
Can one write a user land NFS server using 64K message size running
on top of the current draft API?  Sure.  I don't see a problem in
message delivery locking.  Is the above claim made because of certain
experience with a specific implementation?  And how can a "truncated
message" delivery mechanism help at all?  Does a NFS server want
to receive a truncated message?

-- 

						K. Poon.
						[email-masked]

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 6 Jan 2006 22:47:21 -0700

Mark,

Mark Butler wrote:                               (Thu, 05 Jan 2006 15:15:52)
> don't think so.

Yes there are non-telephony SCTP applications in wide distribution.

A couple of examples: SCTP has been used by the SCIOP for RT-CORBA.  TAO
has supported it for some time.  It is interesting that CORBA was first
specified by the OMG using UDP, then TCP, and finally SCTP.  NFS also
went from UDP to TCP and will likely land on SCTP.

Another is iSCSI and, of course, DDP.

SCTP is also a natural replacement for Ethernet bonding on cPCI
backplanes.  That is, instead of Ethernet bonding two rails together,
SCTP is used to achieve redundancy across both rails while permitting
twice the failure scenario throughput.  CGL specified this and I think
the SAF too.

Of course BICC, SIP, TI-SCCP, MEGACO/H.248, H.225/H.323, and SIGTRAN UAs
are existing implementations of SCTP, however, any application that does
not wish to experience degraded performance in a noisy environment and
is currently using TCP (such as the RT CORBA was and NFS is) will
benefit and eventually move to SCTP.

Then there is Randall's SCTP web server...

> SCTP is a natural replacement for UDP, but there is no SOCK_DGRAM.

Well, no, it is not a natural replacement for UDP.  UDP is a
connectionless datagram protocol.  SCTP is a connection oriented
protocol.

> It is a natural replacement for TCP, but there are no partial writes
> or early delivery.

Well, no, not a natural replacement.  SCTP preserves (well, is capable
of preserving) record boundaries and TCP cannot.  Partial writes to SCTP
are simple when emulating TCP: the implementation can simply close the
SSN at the end of the partial write (because record boundaries are to be
ignored).  Multiple stream interleaving is not an issue, because when
emulating TCP, the implementation uses only one one stream (that is all
that TCP has).  That is, under a SOCK_STREAM type of socket, it is
possible for an SCTP implementation to exactly mimic the TCP Sockets API
(right down to TCP_MAXSEG, TCP_NODELAY, TCP_KEEPALIVE, being, of course,
the only required standard TCP options).

> It can't be used to multi-stream large messages because there is no
> interleaving.

Well, neither can TCP.

> These are primarily API, not protocol limitations.

SCTP's inability to multi-stream large messages is not a limitation of
the API, it is a limitation of the protocol.

I think you are finally back to where I was with SCCP and the SIGTRAN
SUA.  We had the choice of either letting SCTP do all segmentation and
reassembly of messages for SUA, or putting SCCP's segmentation and
reassembly into SUA on top of SCTP.  That latter was necessary because
of SCTP's inability to interleave large messages.  Therefore, the
application (SUA) fragments and reassembles messages across the SCTP
association.

It is something that SCTP forces on the API and application, but let's
be clear: it is a protocol limitation.

If SCTP had included a segment number in the DATA chunk header, instead
of just a beginning and ending flag, there would be not problem
interleaving any messages.

Strangely, it looks like we are doomed to this limitation because the
editors of the protocol will not change it.

> an application developer is likely to dispense with SCTP support completely.

Well, no, that is really not the case either.  When some application or
protocol is adapted to SCTP, is it because SCTP provides some capability
that is lacking in TCP (such as multi-homing, multi-streaming, unordered
and ordered delivery, quick association setup and tear-down, better
attack resilience).  Of all these, taking advantage of multi-streaming,
in particular, requires changes in the application and the API.  An
application cannot take advantage of multi-streaming without specifying
on send what stream to which to place a message.  Also on the receive
side, the application might need to know (or specify) what stream it
came from (or is to be read from).

> would like to see those adaptation layers in-kernel, but that is beside 

Well, it is a general purpose transport protocol.  One that has clear
and specific advantages over other available protocols, but also has its
own limitations, artificial or otherwise.

Get real: any application that is passing records across TCP is already
performing its own segmentation and reassembly, and, of course, even has
to detect its own record boundaries.  The same application running over
UDP cannot exceed a rather restrictive maximum message size.  Does this
mean that UDP and TCP are not general purpose transport protocols in
your view?

If the application uses UDP it is already restricting message size to a
maximum.  If the application uses TCP it already segments and
reassembles large records, and must detect record boundaries within the
data stream.  Doing either is certainly not a limitation for SCTP in
comparison to UDP or TCP.

OTOH, ISO protocols do not require application segmentation, reassembly
or record boundary detection and yet support partial writes and reads.
Therefore, SCTP is only limited when compared to ISO protocols, but then
so are TCP and UDP.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 6 Jan 2006 22:48:46 -0700

Kacheong,

Kacheong Poon wrote:                      (Sat, 07 Jan 2006 11:40:17)
> issue.  I don't think we should mix implementation and

No, this one is definitely a protocol limitation.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Michael Tuexen <[email-masked]>
Date: Sun, 8 Jan 2006 10:43:56 +0100

Dear all,

I would like to look at a specific case suggested by Kacheong.
Assume that we want to use SCTP for an application which has
a maximum message size M and we want to use the 1-to-M model.

The code for the receiver would provide a buffer of size M and
call a receive function in a loop until the MSG_EOR flag is set
or the partial delivery is aborted. Since no messages are interleaved,
one can arrange that the whole message is in the provided buffer
after the loop.

The problem I see here is that if partial delivery happens an
attacker can just send some data chunks containing M-1 bytes
and never the data chunk with the end bit set. The application
might get the starting of the packet, but never receives it
completely and can not determine that this is an attacker because
a message of length M is possible. So it can not abort the association.
How can this DOS attack be avoided?

The SCTP_FRAGMENT_INTERLEAVE option would help here, but makes the
receiving code a bit more complex. I need for each incomplete message
a separate buffer of size M and an additional one for reading. After
I read the fragment, I need to decide for which message it is and copy
to to the right buffer at the right offset.
This is more complex, but using the option the DOS attack can be 
avoided.

But what are you going to do, if this M does not exist (like in the case
of SIGTRAN protocols). You do have a size of messages which you need to 
support
and which is large enough to carry all important stuff. But there are 
other
messages (BEAT, for example) which might be larger.
So you can use Kacheong method of just aborting the association. But the
alternative would be (and I think it is a good one) to send back an 
application
error message. This would require that I can receive a arbitrary large 
message
without blocking other associations. The only way to handle this, is the
SCTP_FRAGMENT_INTERLEAVE option. Or am I missing something?

Best regards
Michael

---

From: Mark Butler <[email-masked]>
Date: Sun, 08 Jan 2006 17:55:13 -0700

Comments inline....

>interfere with the specification.
First of all, the exact semantics of SO_RCVBUF and SO_SNDBUF are not 
available to be specified at the protocol level because both options are 
specified with SOL_SOCKET, not IPPROTO_SCTP or whatever.  Both BSD and 
Linux silently clip SO_RCVBUF and SO_SNDBUF at the socket layer, by 
design.  Applications should not depend on getting a specific buffer 
size, but should get as much as possible when they request more than 
they are allowed.  Otherwise every application would have to do a binary 
search to reach the maximum buffer size.  Of course there is 
communication between the socket layer and the protocol stack - it is 
just the size actually granted, not what was requested.

>It is difficult, but not doable.  The implementation needs to
RFC2960 recommends that space be reserved for each association, which 
can be difficult to accomplish with a small shared buffer.  However, as 
far as the API is concerned SO_RCVBUF and SO_SNDBUF are not available to 
be re-defined on a lower level.  SOL_SOCKET again.  Per association 
buffers should be specified with a different option.  At some point you 
run out of memory - you can't guarantee fixed association buffers for an 
arbitrary number of associations.  Systems that do socket level 
accounting just impose a different limit than implementations with a 
system level pool.

>64K bytes.  I consider this a bug in the implementation.
SO_RCVBUF is generally interpreted as a limit, not a reservation.  Let's 
say you have 10MB of system buffer space, and you set per association 
buffering to 1 MB - are you really going to start rejecting connections 
after the first ten or are you going to overcommit?  Most operating 
systems overcommit socket buffers, which leaves the possibility that 
packets will be dropped no matter what the advertised window is.   If 
overcommitting were not a net gain, no one would do it.

>size.  Will the app then send a 200KB message just to trigger
Partial delivery is either locked or unlocked, regardless of whether 
there is any impact.  In order for partial delivery to be avoided in 
your scenario, the implementation must actually _reserve_ space for each 
association, not just set a limit. If you have one thousand peers, are 
you going to reserve 200MB of memory so that partial delivery is 
guaranteed not be incurred with 128KB messages?    If not, stalling is 
actually _more_ likely because of early partial delivery on a non-shared 
buffer implementation, than one with the equivalent usage of a shared 
buffer.

>issue.  I don't think we should mix implementation and
Changing the semantics of SOL_SOCKET options is not just an 
implementation issue.  But more to the point, specifying critical 
semantics that cannot be widely implemented is a prescription for 
irrelevance.

>SCTP by making
> some very minor changes.
There is no reason to add an SCTP support to a TCP application, unless 
considerable advantage can be gained by doing so.  For many 
applications, those advantages are hard to come by because you cannot 
easily control where the message boundaries are.  So you either get too 
many where they are unnecessary, or you get them in the wrong places. 

To provide a significant advantage HTTP over SCTP would need MSG_EOR 
control on writes, an SCTP_CORK option, SCTP_PARTIAL_DELIVERY_POINT or 
equivalent,  stream specific sockets, and a protocol extension for 
transmit interleaving across streams.  One-to-many implementations would 
need SCTP_FRAGMENT_INTERLEAVE as well.

>using SCTP for that.  I believe this is applicable to most UDP apps.
That is not what I said.  Reliable delivery is commonly implemented over 
UDP to avoid head of line blocking, or to implement partial 
reliability.  NFS, ONC RPC, and Tibco Rendezvous are typical examples.

PR-SCTP is preferable to DCCP whenever you need a mixture of reliable 
and unreliable data. DCCP doesn't do that.  All blocking on the 
unreliable PR-SCTP streams can be eliminated by using  unordered 
messages with an appropriate time to live.

>message delivery locking.  
>
Not with one-to-many sockets, unless your operating system never 
overcommits socket buffers.  Are there any modern ones that do not? I 
doubt it.  Kernel space servers generally use the same protocol stack, 
and similar interfaces, so the same restrictions would normally apply 
there.  There is not a lot of reason to add features to the stack for 
kernel space clients and not expose them to user space as well.

>Is the above claim made because of certain
>experience with a specific implementation?  
>
General engineering principles.  Worst case, not normal case design.

>And how can a "truncated
>message" delivery mechanism help at all? 
>
Again SOCK_DGRAM is only thing POSIX considers reliable or useful enough 
to standardize for one-to-many sockets. AF_UNIX sockets use truncated 
reads with both SOCK_DGRAM (one-to-many) and SOCK_SEQPACKET (one-to-one, 
where implemented), in addition to the traditional SOCK_STREAM mode. 

There is no standard for arbitrarily sized messages over a one-to-many 
socket, nor any existing protocols that support it other than SCTP, as 
far as I know.  SCTP_FRAGMENT_INTERLEAVE and SCTP_PARTIAL_DELIVERY_POINT 
or the equivalent are practical necessities if there is no limit to the 
size of well formed messages.

> Does a NFS server want
Absolutely.  It allows proper error handling.  Better than letting buggy 
clients or attackers consume a large shared buffer before the server has 
a clue. The only other alternative is have the stack abort associations 
when oversize messages arrive, or to have an SCTP_PARTIAL_DELIVERY_POINT 
appropriately low, so the application can do it.  

NFS runs over UDP,  and somehow does not have a problem with messages 
getting truncated. If I am running with 8K read/write sizes, the read 
buffer will be comparable, and anything larger will be truncated.  But 
with the API as specified, an illegitimate message may not be delivered 
until the receive buffer is almost full, which is wasteful, unnecessary 
and a potentially serious vulnerability. 

- Mark

---

From: Mark Butler <[email-masked]>
Date: Sun, 08 Jan 2006 19:32:09 -0700

>went from UDP to TCP and will likely land on SCTP.
Okay - that is a good example.

>Another is iSCSI and, of course, DDP.
Sure there are specifications - but who actually runs iSCSI or DDP over 
SCTP yet, let alone has an implementation in wide distribution?  I 
believe both are pretty much at the testing stage.

>connectionless datagram protocol.  SCTP is a connection oriented
DCCP, the IETF anointed replacement for UDP, is connection oriented.  
There are several reasons why - number one is congestion control,  
number two is reliable control, number three is middlebox traversal.

Most common UDP applications are implicitly connection oriented.  The 
notable exception is DNS.  Virtually everything else could be handled 
nicely using SCTP or DCCP.

>of preserving) record boundaries and TCP cannot.  
>
Adding record boundaries in convenient places doesn't break anything.

>Partial writes to SCTP
>ignored).  
However, presumably one supports SCTP to get better semantics, which 
means controlling record boundaries in SCTP mode, which you can't do, 
and which causes efficiency problems even if you don't care.

>Multiple stream interleaving is not an issue, because when
A proper socket API should provide stream specific sockets to support a 
bundle of SOCK_STREAM style connections on the same association.

>>It can't be used to multi-stream large messages because there is no
Again, presumably one would support SCTP to take advantage of 
capabilities it provides (or could easily provide) that TCP does not.  
Otherwise what is the point? 

>the API, it is a limitation of the protocol.
I said "primarily".  The protocol limitiation could be taken care of 
with a trivial extension - an INIT / INIT ACK parameter signalling 
support for interleaved messages (e.g. when SCTP_FRAGMENT_INTERLEAVE is 
enabled or if the implementation does direct placement where possible).

>interleaving any messages.
I would prefer an offset, for out of order direct placement.  By the way 
who uses the ppid field and for what? 

>editors of the protocol will not change it.
A negotiated interleave option would work around that problem.

>came from (or is to be read from).
Stream specific sockets would fix the stream specification problem quite 
nicely.  All the other stuff can share a code base with TCP with only 
trivial changes when in SCTP mode.

>own limitations, artificial or otherwise.
It could be, if the socket API were equally general.

>mean that UDP and TCP are not general purpose transport protocols in
No, not in the sense I mean here. 

>data stream.  
Record boundaries yes, SAR no, on TCP that is. 

>Doing either is certainly not a limitation for SCTP in
DECnet supports partial writes and arbitrary record boundaries too.  
Doing it only on reads but not on writes is rather odd.

 - Mark

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sun, 8 Jan 2006 20:26:57 -0700

Mark,

Mark Butler wrote:                                                                    (Sun, 08 Jan 2006 19:32:09)
>    and which causes efficiency problems even if you don't care.

No, see RFC 3257.  API semantics is not one of the stated advantages of SCTP.

>    A  proper socket API should provide stream specific sockets to support
>    a bundle of SOCK_STREAM style connections on the same association.

Why?  Where is it necessary?

>    Otherwise what is the point?

SCTP does not support multistreaming of large messages either.

>    I  would  prefer an offset, for out of order direct placement.  By the
>    way who uses the ppid field and for what?

SIGTRAN UAs, to identify the protocol.

>    A negotiated interleave option would work around that problem.

Not in the protocol.

>    Stream  specific  sockets  would  fix the stream specification problem
>    quite nicely.  All the other stuff can share a code base with TCP with
>    only trivial changes when in SCTP mode.

There is no such thing as a stream socket.  The protocol does not support one
either.

>    It could be, if the socket API were equally general.

A protocol and an interface are two different things.  The protocol is general
purpose, the API is what you make of it.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Mark Butler <[email-masked]>
Date: Sun, 08 Jan 2006 21:47:02 -0700

>No, see RFC 3257.  API semantics is not one of the stated advantages of SCTP.
That is a non sequitur.  An advantage does not need to be endorsed to be 
desirable.   Furthermore, SCTP is only available to most through the 
available socket API.  The standard socket API might as well be the 
protocol, for most practical purposes.  It is a common metonymy.

>Why?  Where is it necessary?
Not necessary just desirable.  iSCSI, DDP, HTTP, NFS,  and SMB often 
establish multiple parallel TCP connections. The advantage of running 
them as streams on the same association is lower setup latency.  If the 
protocol is stateless like HTTP or LDAP, you can eliminate connection 
establishment delay completely for follow-on requests.   Another example 
is OpenSSH, which internally multiplexes multiple flow controlled 
streams over a single TCP connection. Running each as an SCTP stream on 
the same association would perform better without the adverse 
authentication overhead of multiple TCP connections.

>SCTP does not support multistreaming of large messages either.
It could and it should.

>SIGTRAN UAs, to identify the protocol.
Well, I think it is dead weight.  Headers like that should be done at 
the ULP level, once per message, not once per fragment.    It would be 
nice if an application could say, I don't need ppids, you may use the 
slot for other purposes (like direct placement).

>>   A negotiated interleave option would work around that problem.
No kidding. One would add it to the protocol, like PR-SCTP.  Twenty 
lines of code, tops.

>There is no such thing as a stream socket.  The protocol does not support one
Yet, that is.  The protocol has nothing to do with whether there are 
stream level sockets or not. You simply serialize access to the 
association appropriately.

 - Mark

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sun, 8 Jan 2006 22:57:43 -0700

Mark,

Mark Butler wrote:                                                    (Sun, 08 Jan 2006 21:47:02)
>        (Sun, 08 Jan 2006 19:32:09)
>   
> 
> Partial writes to SCTP
>    That  is a non sequitur.  An advantage does not need to be endorsed to
>    be  desirable.    Furthermore,  SCTP is only available to most through
>    the  available  socket  API.  The standard socket API might as well be
>    the protocol, for most practical purposes.  It is a common metonymy.

The draft API is not a standard of any kind.

>    a bundle of SOCK_STREAM style connections on the same association.

You are free to use whatever API you feel meets your needs.

>    them  as  streams  on the same association is lower setup latency.  If
>    the  protocol  is  stateless  like  HTTP  or  LDAP,  you can eliminate
>    connection  establishment  delay  completely  for  follow-on requests.
>    Another example is OpenSSH, which internally multiplexes multiple flow
>    controlled  streams  over  a single TCP connection. Running each as an
>    SCTP  stream  on the same association would perform better without the
>    adverse authentication overhead of multiple TCP connections.
> 
>   
> 

Then request the protocol be changed, not just the API.

>    slot for other purposes (like direct placement).

That was never the purpose of the PPID.

>    No  kidding.  One  would add it to the protocol, like PR-SCTP.  Twenty

Then request the protocol be changed.  See how far you get.

>    association appropriately.

Streams are unidirectional.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Randall Stewart <[email-masked]>
Date: Mon, 09 Jan 2006 06:31:17 -0500

> just the size actually granted, not what was requested.

This made me curious..

So here is a little test program I wrote:
#include <stdio.h>
#include <sys/types.h>
#include <sys/errno.h>
#include <sys/socket.h>
#include <netinet/in.h>

main()
{
         int ret;
         int fd;
         int optval, optlen;

         fd = socket(AF_INET, SOCK_STREAM, IPPROTO_SCTP);
         if(fd == -1) {
                 printf("No sctp support error %d\n", errno);
                 exit(1);
         }
         optval = 0x70000000;
         if(setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &optval, optlen) != 0){
                 printf("err:%d could not set rcvbuf to %x 
bytes\n",errno, optval
);
         }
}

Here is the result
bash-stewlap: ./test
err:55 could not set rcvbuf to 70000000 bytes

Now I am running on a FreeBSD 7.0 system.. don't know if 5 or 6 have
the same error.. I suspect so..

55 is ENOBUFS

R

---

From: Randall Stewart <[email-masked]>
Date: Mon, 09 Jan 2006 06:56:00 -0500

Michael:

I have read over this thread now... and I think
Kacheong is not objectioning to SCTP_FRAGMENT_INTERLEAVE
(correct me if I am wrong Kacheong).

What I am unclear of is what option Kacheong is proposing
or not for controlling when partial delivery is not
allowed.

Kacheong... do you propose an option like:

SCTP_DONT_PARTIAL_DELIVER_BELOW

(the name is probably bad).. but the idea being
the app can call this and it tells SCTP that
it MUST NOT partial deliver message below a
set threshold...

Is this what you propose Kacheon? Or am I mis-understanding
and you are proposing just to have a fixed limit
against the rwnd...

???

R

Michael Tuexen wrote:
> This is more complex, but using the option the DOS attack can be avoided.
> 
> and which is large enough to carry all important stuff. But there are other

---

From: Mark Butler <[email-masked]>
Date: Mon, 09 Jan 2006 09:40:51 -0700

Randall Stewart wrote:

>> available to be specified at the protocol level because both options 
>> are specified with SOL_SOCKET, not IPPROTO_SCTP or whatever.  Both 
>> BSD and Linux silently clip SO_RCVBUF and SO_SNDBUF at the socket 
>> layer, by design.  Applications should not depend on getting a 
>> specific buffer size, but should get as much as possible when they 
>> request more than they are allowed.  Otherwise every application 
>> would have to do a binary search to reach the maximum buffer size.  
>> Of course there is communication between the socket layer and the 
>> protocol stack - it is just the size actually granted, not what was 
>> requested.
>
>
>
> This made me curious..
>
Comments in the Linux 2.6 code indicate that they were duplicating BSD 
behavior.  It must have changed in the past few years.

 - Mark

---

From: Mark Butler <[email-masked]>
Date: Mon, 09 Jan 2006 10:03:41 -0700

>The draft API is not a standard of any kind.
Which dictionary do you draw that conclusion from?  As if there were no 
such thing as a de-facto standard.

>You are free to use whatever API you feel meets your needs.
Sorry, I didn't know that.

>Then request the protocol be changed, not just the API.
The base protocol does not have to be changed - an extension would need 
to be documented.

>That was never the purpose of the PPID.
So what?  Has the PPID slot an eternally ordained purpose that requires 
it not be used for things it was not intended for?

>Streams are unidirectional.
You keep talking about terms as if they were timeless Platonic objects 
with one and only one fixed and unchangable meaning.  There are other 
types of streams besides SCTP protocol level streams.  Most of the world 
uses bidirectional streams.  There is nothing stopping someone from 
mapping a normal bidirectional stream to a pair of lower level ones.  
DDP/SCTP does this for example. 

 - Mark

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 9 Jan 2006 11:45:34 -0700

Mark,

I look forward to commenting on your extension draft.  When
will the first version be available?

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Mark Butler <[email-masked]>
Date: Mon, 09 Jan 2006 12:49:18 -0700

>will the first version be available?
The socket API extensions have priority. 

 - Mark

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 9 Jan 2006 15:26:39 -0700

Mark,

It is rather pointless for the API to support it if the protocol does not.

--brian

Mark Butler wrote:                                                    (Mon, 09 Jan 2006 12:49:18)

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Randall Stewart <[email-masked]>
Date: Tue, 10 Jan 2006 07:13:02 -0500

Mark:

A few comments.. in-line...

Mark Butler wrote:
>> (Sun, 08 Jan 2006 21:47:02)
>>  
>>
>>> Mark Butler 
>>> wrote:                                                             

Brian is correct here.. the api document will NOT
have any IETF standards track designation... that
being said we all know that some documents, though
not standards, end up being de-facto as you suggest.

I am not sure that the api document will ever reach that
level.... maybe.. there are at least few implementations
that one can write code to and use moving the src from
one machine to another (solaris/linux w+lk-sctp/BSD+kame/HP/IOS)
but it is not global. Many implemenations that have
been at bakeoff's have had had their own private interface
that allowed different things :-D

Even IOS was this way until recently... But thats
another story...

Time will be the teller of if this reaches "de-facto"
standard or not :-D

> Sorry, I didn't know that.
Many do use different API's but I think that a large number
of app developers are not wanting to go build their
own interface to an SCTP stack.. but you are free to
do this if you so desire :-D

> The base protocol does not have to be changed - an extension would need 

Extensions are welcome for comment and review.. now has to
two specific points in this discussion

PPID field and the idea of a fragement/offset.. some
history so to speak..

1) Some of the original versions of the spec (MDTP I know for
    sure and maybe even some of the SCTP ones) had a of and part
    field. Two 16 bit fields part for the part number.. and of
    for how many chunks were in this message. That was stripped
    away when all fragmentation was taken out of SCTP. It was
    later then, re-added when folks realized that they might
    have XML type signalling messages that. Now I did, at that
    time, want to read-add the part/of fields.. but as in many
    things we do not always get what we want when we do
    standards.... in the end I was just happy that fragmentation
    was back in the protocol ;-D

2) The PPID field was added by the IESG more or less. They
    wanted a way to, in the future, try to seperate the
    port number from being a de-multiplexor and a protocol
    indicator... I argued strongly against this, felling that
    32 bits per message was a LOT to waste for a "future" field
    that I did not think could be used as was desired. However
    again we don't always get what we want.. and actually
    ethereal (and possibly other network equipment) actually
    use this field so that they can display contents based
    on assigned PPID format... which is probably a good
    thing...

> it not be used for things it was not intended for?

I am not to sure one can use the PPID field for anything
but what it was designed for..

>> Then request the protocol be changed.  See how far you get.

Adding a protocol extension is something that anyone can do by
writting a draft and maybe it will proceed. The big thing
here is I would suggest creating a "new" data type chunk
that could have this different format that you desire. That
way old implementations that do not use this "new" format
could still talk to "new" ones :-D

Of course don't think that if you submit any idea it
will not get changed/re-arranged/ and generally reshaped
by the WG.. this is a good thing (more or less :-D)

R

---

From: Mark Butler <[email-masked]>
Date: Tue, 10 Jan 2006 07:39:49 -0700

>    on assigned PPID format... which is probably a good

Is there a PPID registry somewhere then?  The DCCP specification takes 
the same attitude - it is a protocol violation not to specify a service 
code.  You can't connect without a matching service code, you can't 
listen without specifying one. Zero is not allowed.   But the service 
code is only specified at connect time, not every packet.

You can't demux using chunk level PPIDs in the current scheme unless 
everything is handled by the same process.  A traditional socket API 
certainly can't handle system level associations without another layer 
of protocol support.  And if you have system level associations, then 
the port fields are redundant.  So presumably the PPID should have been 
added as a parameter to the INIT chunk, in a manner similar to DCCP, to 
serve the intended purpose.

However, my observation is that the PPID information only needs to be 
carried for the first fragment.  So subsequent fragments could use it as 
a fragment offset field without breaking ULPs that depend on it (with 
protocol level negotiation of course).

> could still talk to "new" ones :-D

I certainly wouldn't suggest breaking interoperability.  There would 
have to be negotiation. 

> by the WG.. this is a good thing (more or less :-D)

Of course.

 - Mark

---

From: Michael Tuexen <[email-masked]>
Date: Tue, 10 Jan 2006 17:37:53 +0100

Hi Mark,

see my comments in-line.

Best regards
Michael

On Jan 10, 2006, at 15:39 Uhr, Mark Butler wrote:

> the same attitude - it is a protocol violation not to specify a 
> service code.  You can't connect without a matching service code, you 
> can't listen without specifying one. Zero is not allowed.   But the 
> service code is only specified at connect time, not every packet.
In contrast to Randy I really like the PPID. There is a registry at
http://www.iana.org/assignments/sctp-parameters
It is very useful to be able to specify the ULP per DATA chunk. First 
of all, tracers
like ethereal can find the ULP without looking at the port numbers. 
Second, you can
have multiple ULP in one association. This is used, for example, in 
ASAP, an RSerPool protocol.
> the port fields are redundant.  So presumably the PPID should have 
> been added as a parameter to the INIT chunk, in a manner similar to 
> DCCP, to serve the intended purpose.
Why can't you demux? You just can not wait for it.
> carried for the first fragment.  So subsequent fragments could use it 
> as a fragment offset field without breaking ULPs that depend on it 
> (with protocol level negotiation of course).
Your observation is correct. But I think it is a very bad idea to use 
this field with
multiple semantics.

If you want to have a DATA chunk with segment offset or number 
information, define
a new chunk type. This would also allow implementations not supporting 
your extension
to react appropriate.

---

From: Randall Stewart <[email-masked]>
Date: Wed, 11 Jan 2006 06:30:41 -0500

Michael:

A comment or two :-D

Michael Tuexen wrote:
> It is very useful to be able to specify the ULP per DATA chunk. First of 
> all, tracers

Well.. I was giving a history lesson on SCTP development... as
to how I feel about the field now.. I must admit it HAS grown
on me .. I might not be has pleased with the field as you.. but
I really do see its relevance for sniffers and such.. So even
though I did not like it orginally.. I am ok with it now... :-D

> Your observation is correct. But I think it is a very bad idea to use 

I agree here.. If mark wants a fragment/offset type data chunk
then he needs to write a draft and submit it (hint hint mark).
Then he can use the supported chunk types field (in the INIT
and INIT-ACK.. this is in addip) to be able to recognize
that his peer supports his new data chunk... That will
allow interoperability  and no problems :-D

R

---

From: Mark Butler <[email-masked]>
Date: Wed, 11 Jan 2006 07:24:49 -0700

Hello Michael - comments below:

Michael Tuexen wrote:

>> certainly can't handle system level associations without another 
>> layer of protocol support.  And if you have system level 
>> associations, then the port fields are redundant.  So presumably the 
>> PPID should have been added as a parameter to the INIT chunk, in a 
>> manner similar to DCCP, to serve the intended purpose.
>

My point is simply that PPID cannot be used to demultiplex the way the 
port number is used, with different processes using PPID bound sockets, 
without additional support. Hypothetically, such support would have to 
include advertisement / verification of PPID capability on an 
association, a PPID to port mapping scheme, and so on.

In practice, PPID standardization is useful for network analyzers, but 
which PPIDs are used has little other practical consequence outside the 
scope of two communicating processes.  The fact that an arbitrary number 
of associated PPIDs may be used by the same ULP makes PPID a poor choice 
for a service code.  A service code would normally be associated with 
one ULP, so that you could look up connection port / address information 
by service code, in DNS or wherever.   This is not a bad thing, I am 
just saying that PPIDs and DCCP-like service codes are different - 
service codes are used at the connection / association level and PPIDs 
are a step lower than that.

An association that handles more than one ULP is really implementing a 
higher level Meta-ULP, with a set of conventions on which lower level 
protocols are supported, on which streams, how each session is initiated 
/ terminated if necessary, and so on.   ULP use of the PPID field is 
convenient because it is already there, but normally each layer should 
add / strip its own headers.  Not that the PPID is going anywhere...

> a new chunk type. This would also allow implementations not supporting 

Nothing would happen without negotiation, so that wouldn't be a 
problem.  I am not suggesting changing the header length or layout.  A 
supporting implementation would use the B flag to determine whether the 
PPID slot contained a PPID or an offset.  DCCP uses the X flag to make a 
much more radical change  - 24 bit vs. 48 bit serial number - it changes 
the header length, and changes bit field alignment accordingly.

There would need to be two advertised capabilities - one would indicate 
"I understand message offsets", the other would indicate "I can handle 
message interleaving".  The first would allow a receiver to do direct 
placement of incoming data, including careful parallel checksum / 
placement like some TCP implementations do.  

Both together would allow arbitrary message interleaving on the same 
association, e.g. for applications that set SCTP_MESSAGE_INTERLEAVE.  
Notably, this would allow small high priority messages to be delivered 
while larger messages are pending.

 - Mark

---

From: Mark Butler <[email-masked]>
Date: Wed, 11 Jan 2006 07:46:52 -0700

> and you are proposing just to have a fixed limit
I think an option like that would solve the problem.  Doing it based on 
the receive buffer size is problematic for a variety of reasons, 
especially shared buffers.

However, why don't we just use SCTP_PARTIAL_DELIVERY_POINT and define it 
with similar semantics, such that partial deliveries must not occur 
below the specified threshold, but is requested when the threshold is 
reached.  That way applications can get early notification of 
illegitimately sized messages and take the appropriate action.

Is that reasonable?

- Mark

---

From: Randall Stewart <[email-masked]>
Date: Wed, 11 Jan 2006 11:45:07 -0500

> illegitimately sized messages and take the appropriate action.

I am ok with it..

Kacheong?

If you agree I will work on getting this and the FRAGMENT wording
together..

R