Subject: [hybi] Now from the experience: utf-8/binary frame type is a BAD idea

From: Iñaki Baz Castillo <[email-masked]>
Date: Tue, 1 Nov 2011 13:43:28 +0100

Hi, I'm implementing SIP over WebSocket as I defined in:
  http://tools.ietf.org/html/draft-ibc-rtcweb-sip-websocket-00

Initially I mandated WS frames of type UTF-8 (as SIP
requests/responses use UTF-8 or ASCII) and I was happy. But now I've
realized that SIP protocol (RFC 3261), whose requests/responses are
very similar to HTTP ones, allows binary bodies in certain cases.

So now I must change the spec and mandate WS frames of type binary
*just* for the case in which a JavaScript implementation of SIP over
WebSocket receives a *valid* SIP request/response with binary body.

It's clear for me that the WS frame encoding indication is a bad idea.
IMHO the encoding should be defined by each WebSocket subprotocol by
leaving the WS frame layer not aware of it.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: John Tamplin <[email-masked]>
Date: Tue, 1 Nov 2011 09:40:03 -0400

> leaving the WS frame layer not aware of it.

If you are sending binary data, then you have to send it in a binary frame
-- you can't pretend it is UTF8 text.

Having a separate text frame type was discussed over and over on this
group, including by you.  Please accept the consensus rather than
continuing to argue against it.

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Tue, 1 Nov 2011 14:52:34 +0100

2011/11/1 John Tamplin <[email-masked]>:
> Having a separate text frame type was discussed over and over on this group,
> including by you.  Please accept the consensus rather than continuing to
> argue against it.

I know. I just meant that, given my current experience, I don't think
it's a good idea.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: John Tamplin <[email-masked]>
Date: Tue, 1 Nov 2011 10:19:44 -0400

> I know. I just meant that, given my current experience, I don't think

It seems really simple -- if you might need to send arbitrary binary data
in the layer on top of WebSockets, you use a WebSocket binary frame for it.
 If you only need text, you use a text frame and let the browser do the
conversion to its internal representation for you in native code rather
than JS.

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Tue, 1 Nov 2011 15:30:06 +0100

2011/11/1 John Tamplin <[email-masked]>:
> It seems really simple -- if you might need to send arbitrary binary data in
> the layer on top of WebSockets, you use a WebSocket binary frame for it.  If
> you only need text, you use a text frame and let the browser do the
> conversion to its internal representation for you in native code rather than
> JS.

The problem is that, in my case, 99.9999% of WebSocket messages my JS
will receive contain UTF-8 text, but I must be ready to receive
binary.
The server is just a SIP proxy and it does not inspect the message in
order to deliver it within utf-8 frames or binary frames, so it must
always use binary frames.

The JL client does a message parsing when a WS message is received.
Since this WS message must be binary I need convert it into a String
before parsing it, and AFAIK ArrayBuffer to String is not
straightforward in JavaScript.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: John Tamplin <[email-masked]>
Date: Tue, 1 Nov 2011 10:51:00 -0400

> straightforward in JavaScript.

Right, which is *exactly* why there is a text frame type in the first
place.  You are suggesting inflicting that pain on everyone by having only
binary frames, when the 99% use case is going to be exchanging something
like JSON.

An alternative you might consider is allowing pure-text frames to be sent
as text, and then sending binary data in binary frames.  You know, what
they were intended for.

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Tue, 1 Nov 2011 15:59:32 +0100

2011/11/1 John Tamplin <[email-masked]>:
> Right, which is *exactly* why there is a text frame type in the first place.
>  You are suggesting inflicting that pain on everyone by having only binary
> frames, when the 99% use case is going to be exchanging something like JSON.
> An alternative you might consider is allowing pure-text frames to be sent as
> text, and then sending binary data in binary frames.  You know, what they
> were intended for.

That would require my WebSocket server to inspect incoming SIP
messages having as destination a WebSocket client. It must check
whether the SIP message contains (or not) pure UTF-8 data and send it
in WS utf-8 frames or binay. It would be not so hard.

Thanks.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Joe Mason <[email-masked]>
Date: Tue, 1 Nov 2011 11:47:29 -0400

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Iñaki Baz Castillo
> 

Sounds like you're complaining about the design of SIP, not WebSockets.

Joe

---------------------------------------------------------------------
This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 02 Nov 2011 10:15:32 +0900

> in WS utf-8 frames or binay. It would be not so hard.

Then I think you should do it. I know almost nothing about SIP, but I 
guess you can also make headers and body two separate messages, and can 
always use UTF-8 for the headers, and only check the bodies (or maybe 
only a subset of bodies, because the headers may tell you whether 
there's (a chance of) binary.

Anyway, I don't think it's WebSockets' fault, and I don't think it's 
SIPs fault, but when gatewaying from one protocol to another, some work 
may be needed in the areas where these protocols use different concepts.

Regards,    Martin.

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Wed, 2 Nov 2011 09:45:01 +0100

2011/11/2 "Martin J. Dürst" <[email-masked]>:
> Then I think you should do it. I know almost nothing about SIP, but I guess
> you can also make headers and body two separate messages, and can always use
> UTF-8 for the headers, and only check the bodies (or maybe only a subset of
> bodies, because the headers may tell you whether there's (a chance of)
> binary.

Well, IMHO it would make things more complex. But indeed could be a solution.

> Anyway, I don't think it's WebSockets' fault, and I don't think it's SIPs
> fault

I don't agree. There is nothing wrong in allowing a signaling protocol
to carry both text and binary bodies. This is not a problem for common
SIP clients.

IMHO the WS frame type is something a bit artificial which just fits
well with JavaScrtip given JavaScript limitations (converting from
ArrayBuffer to String is hard).

Regards.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 04 Nov 2011 17:17:35 +0900

Hello Iñaki,

Sorry to be late with my reply.

On 2011/11/02 17:45, Iñaki Baz Castillo wrote:
> 2011/11/2 "Martin J. Dürst"<[email-masked]>:

> to carry both text and binary bodies.

Yes. That's why I'm saying I don't think it's SIPs problem (assuming 
that the textual data is adequately labeled, e.g. with a MIME Media type 
and a 'charset' parameter).

> This is not a problem for common
> SIP clients.

Of course not. SIP clients are written to work with SIP.

> ArrayBuffer to String is hard).

It looks artificial from a SIP viewpoint. But distinguishing between 
binary and textual data isn't something artificial; it's something very 
important to avoid major problems.

Although things have improved recently, there were lots and lots of 
problems based on the fact that textual data was sent essentially as 
binary data without knowing how it was encoded. Given you have an "ñ" in 
your name, I'm sure you know enough cases where that got garbled.

WebSockets tries to solve this problem in a way that differs from the 
standard approach used in email, HTTP, and so on. That's why at first 
sight, it may look somewhat unusual. And that's why there may be a need 
for some additional effort when converting from another protocol in a 
gateway, because textual data and binary data have to be separated.

But that's not because of JavaScript. Most modern programming languages 
use a single encoding for all their text internally, and separate text 
and binary. The only exception I know about is Ruby, and even in Ruby 
it's much easier to write programs that use a single encoding (UTF-8) 
for all their text than to write programs where each chunk of text is in 
a different encoding.

Regards,   Martin.

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Fri, 4 Nov 2011 10:37:49 +0100

2011/11/4 "Martin J. Dürst" <[email-masked]>:
> Yes. That's why I'm saying I don't think it's SIPs problem (assuming that
> the textual data is adequately labeled, e.g. with a MIME Media type and a
> 'charset' parameter).

Sorry, I understood "I do think it's SIPs fault" :)

> It looks artificial from a SIP viewpoint. But distinguishing between binary
> and textual data isn't something artificial; it's something very important
> to avoid major problems.

> Although things have improved recently, there were lots and lots of problems
> based on the fact that textual data was sent essentially as binary data
> without knowing how it was encoded. Given you have an "ñ" in your name, I'm
> sure you know enough cases where that got garbled.

Too much... :(

> standard approach used in email, HTTP, and so on. That's why at first sight,
> it may look somewhat unusual. And that's why there may be a need for some
> additional effort when converting from another protocol in a gateway,
> because textual data and binary data have to be separated.
>
> But that's not because of JavaScript. Most modern programming languages use
> a single encoding for all their text internally, and separate text and
> binary. The only exception I know about is Ruby, and even in Ruby it's much
> easier to write programs that use a single encoding (UTF-8) for all their
> text than to write programs where each chunk of text is in a different
> encoding.

Thanks. Finally I "solved" my issue by inspecting the SIP message
before sending it via WebSocket to the web browser. The websocket
server is written in Ruby, so basically I do (~pseudocode):

if ( sip_msg.force_encoding(::Encoding::UTF_8).valid_encoding? )
  send_text_frame( sip_msg )
else
  send_binary_frame( sip_msg )
end

So it generates a text or binary frame depending on the content of the
SIP message, and I'm done :)

Now I must deal in client side (JS) as the SIP parser assumes that the
received data is a String. But when a binary message arrives it can be
Blob or ArrayBuffer (seteable via the WebSocket API). Not sure if I
can convert a Blob or ArrayBuffer object containing non valid UTF-8
symbols into a JS String, but that is another battle :)

Thanks a lot.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: John Tamplin <[email-masked]>
Date: Fri, 4 Nov 2011 12:01:12 -0400

> symbols into a JS String, but that is another battle :)

You don't want to store binary data in Strings -- that is the whole point
behind ArrayBuffer.

AFAIK, you have to do the UTF8 encode/decode yourself (via getUInt8 etc) --
it seems like DataView should have some utilities for extracting/inserting
strings into the underlying ArrayBuffer.

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Sat, 5 Nov 2011 14:49:45 +0100

> strings into the underlying ArrayBuffer.

Thanks, but I think that my case is the opposite:

I've a SIP parser in JavaScript. The parser expects a JS string as
argument. Such string would be the WS text message received.
But in case of WS binary message, I wouls get a Blob or an ArrayBuffer
object (depending on WebSocket#binary_type setting).

So what I need is to covert the received Blob or ArrayBuffer into a
JavaScript String so I can pass it to the parser function. I must
examine how to achieve that, but this is my personal battle :)

Thanks a lot.

-- 
Iñaki Baz Castillo
<[email-masked]>