Subject: [ietf-types] The application/www-form-urlencoded format

From: Bjoern Hoehrmann <[email-masked]>
Date: Sat, 25 Sep 2010 23:14:39 +0200

Hi,

  http://tools.ietf.org/html/draft-hoehrmann-urlencoded -- the draft de-
scribes the application/www-form-urlencoded format, a variant of the
application/x-www-form-urlencoded format first described in RFC 1866.
RFC 1866 recommended that implementations also accept ";" as separator,
which the draft permits and prefers, and RFC 1866 failed to define the
character encoding to use, the draft addresses that by mandating UTF-8,
which is universally recommended these days. As the ampersand will be
handled as it would be for the RFC 1866 format, I believe the formats
are similar enough that using the very similar name is justified. [1]

The draft probably still has some rough edges in the prose but the
format is not going to change. I believe it addresses the feedback I
got since the first draft published four years ago; public feedback at
http://lists.w3.org/Archives/Public/www-archive/2006Sep/thread.html#msg30

The template is as follows. Looking at RFC 4288 again, it seems instead
of "8bit" the encoding considerations should be "binary".

   Type name:               application
   Subtype name:            www-form-urlencoded
   Required parameters:     none
   Optional parameters:     none

      Note: The media type does not have a 'charset' parameter, it
      is incorrect specify one and to associate any significance to
      it if specified. The character encoding is always UTF-8. The
      Unicode encoding form signature is not supported; a leading
      U+FEFF character will be considered part of a <name>.

   Encoding considerations: 8bit

   Security considerations: See section 9.
   Interoperability considerations:
      None, except as noted in other sections of this document.

   Published specification: RFC XXXX
   Applications that use this media type:
      Systems that interchange data sets of name-value pairs.

   Additional information:

      Magic number(s):             n/a
      File extension(s):           n/a
      Macintosh file type code(s): TEXT
      Fragment identifiers:        n/a

   Person & email address to contact for further information:
      See Author's Address section.

   Intended usage:          COMMON
   Restrictions on usage:   n/a
   Author:                  See Author's Address section.
   Change controller:       The IESG.

[1] The regular expression to match both names is slightly simpler if
    they only differ in the "x-", and it seems fitting to standardize
    an "x-" type by removing the "x-", but if there is a good argument
    why using similar names is a bad idea, I am also quite open to name
    it application/name-value-pairs or something like that instead.

Let me know if anyone sees any other problem. Thanks,
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: "Anne van Kesteren" <[email-masked]>
Date: Sun, 26 Sep 2010 11:22:40 +0200

> application/x-www-form-urlencoded format first described in RFC 1866.

I think it is unfortunate it still allows encoding in various ways. So  
while things could be more readable as you pointed out in the past user  
agents are still allowed to obscure most everything.

> http://lists.w3.org/Archives/Public/www-archive/2006Sep/thread.html#msg30

The bug about + seems to be still be there. Escapes are first decoded and  
then + is replaced with U+0020. Also application/x-www-form-urlencoded is  
on its way of being standardized as part of HTML5 now.

>       U+FEFF character will be considered part of a <name>.

Most other such formats ignore a leading U+FEFF.

>     it application/name-value-pairs or something like that instead.

Well, application/x-www-form-urlencoded is not going away anytime soon. It  
is what you get for <form> by default. That some consider it non-standard  
is just semantics. Having said that I do not mind dropping the x- for an  
improved version of that format.

-- 
Anne van Kesteren
http://annevankesteren.nl/

---

From: "Stephen D. Williams" <[email-masked]>
Date: Sun, 26 Sep 2010 12:36:27 -0700

On 9/26/10 2:22 AM, Anne van Kesteren wrote:
> On Sat, 25 Sep 2010 23:14:39 +0200, Bjoern Hoehrmann <[email-masked]> wrote:

What the character encoding errors are at the end of section 5 needs to be explained, at least for 2 cases.
>
> I think it is unfortunate it still allows encoding in various ways. So while things could be more readable as you pointed out in 
> the past user agents are still allowed to obscure most everything.

It should be the case that only what has to be encoded is encoded.  This is required for any signature canonicalization anyway.  It 
could be the case that a more restricted situation exists, although I don't see it for form submission.
> ...
> The bug about + seems to be still be there. Escapes are first decoded and then + is replaced with U+0020. Also 
> application/x-www-form-urlencoded is on its way of being standardized as part of HTML5 now.

I was just debugging an Oauth implementation (customized RFC 5849) that has this kind of mistake, plus they used the older set of 
unreserved characters rather than those defined in RFC 3986:
unreserved = ALPHA / DIGIT / "-" / "." / "_" / "~"

Some specs allowed more:
RFC 2396: Section 2.3:
unreserved = alphanum | mark
mark = "-" | "_" | "." | "!" | "~" | "*" | "'" | "(" | ")"

Java's urlencoder seems broken as it is almost RFC 3986 but with '*' instead of '~':
http://download.oracle.com/javase/1.4.2/docs/api/java/net/URLEncoder.html
# The special characters "|.|", "|-|", "|*|", and "|_|" remain the same.
# The space character "| |" is converted into a plus sign "|+|".

For canonicalization purposes, RFC 5849 says in sction 3.6:
This method is different from the encoding scheme used by the
"application/x-www-form-urlencoded" content-type (for example, it
encodes space characters as "%20" and not using the "+" character).
It MAY be different from the percent-encoding functions provided by
web-development frameworks (e.g., encode different characters, use
lowercase hexadecimal characters).

They use uppercase hex and ' '->'%20 rather than the option of ' '->'+'.

If ' '->'+' is to be used, it has to be done after percent encoding so that '+' gets percent encoded before ' '->'+'. At this point 
however, most people can read "%20" as space almost as fast as '+'.  On the other hand, '+' is more compact and is still more 
readable.  A spec should deterministically choose one or the other so that signatures can be done without further agreement.  
Signing a form is a common need.

While these specs have different purposes, it is often the case (as with OAuth), that key/value pairs can show up in HTTP headers, 
POST bodies that may be www-form-urlencoded or something like XML, in the URL as a query string, along with being canonicalized for 
HMAC signature.  Unifying the operations reduces confusion and coding.

> Most other such formats ignore a leading U+FEFF.

U+FEFF, the BOM, is only needed where the encoding can be something other than UTF-8.  If there is no choice, it isn't needed and 
probably shouldn't be present unless the situation might change.  If UTF-16 can be used, then it is needed.

> Well, application/x-www-form-urlencoded is not going away anytime soon. It is what you get for <form> by default. That some 
> consider it non-standard is just semantics. Having said that I do not mind dropping the x- for an improved version of that format.
>
>

sdw

-- 
Stephen D. Williams [email-masked] [email-masked] LinkedIn: http://sdw.st/in
V:650-450-UNIX (8649) V:866.SDW.UNIX V:703.371.9362 F:703.995.0407
AIM:sdw Skype:StephenDWilliams Yahoo:sdwlignet Resume: http://sdw.st/gres
Personal: http://sdw.st facebook.com/sdwlig twitter.com/scienteer

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Sun, 26 Sep 2010 22:19:40 +0200

>agents are still allowed to obscure most everything.

I do think that encoder implementers can make reasonable choices about
that. If an implementer decides it's best to escape, say, the zero-width
space character because it's invisible then I see nothing wrong with
that. If another implementer decides to not escape it, that's fine too.

>on its way of being standardized as part of HTML5 now.

I don't think there is anything that "HTML5" could standardize about it,
other than define how form submission using that type should be imple-
mented in HTML implementations. As for the +, the text refers to the ab-
stract syntax tree you get when parsing a string using the grammar, so
an escaped plus sign is no instance of `plus`. I take it that's one of
the text's rough edges I should smooth out.

>Most other such formats ignore a leading U+FEFF.

I can't think of any format that's always UTF-8 encoded yet allows for
it, but anyway, treating it as part of a name is simpler than ignoring
it, I think people are more likely to implement that correctly than if
I were to require recognizing it.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Sun, 26 Sep 2010 22:32:42 +0200

>be explained, at least for 2 cases.

I believe the first encodes a pair of surrogate code points, the second
a character beyond U+10FFFF, the first is an overlong sequence, then it
is a truncated sequence, and finally you have an illegal starter with
ASCII after it. They are all prohibited by the definition of UTF-8. I do
not think that kind of technical detail would be well-placed there, but
I could rephrase so it refers to UTF-8 directly instead of calling it
"character encoding errors" though that's the same thing here.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: "Anne van Kesteren" <[email-masked]>
Date: Sun, 26 Sep 2010 22:40:37 +0200

> that. If another implementer decides to not escape it, that's fine too.

I suppose. And I guess if we ever decide to implement any of this we could  
make more specific requirements in the forms section of HTML for that  
class of implementors.

After having thought about it some more I am doubting this would be a very  
useful addition though. The benefit over application/x-www-form-urlencoded  
seems marginal. Not high enough to warrant the cost.

> the text's rough edges I should smooth out.

I guess. I'm not very good at reading grammar / prose combinations. (I  
thought that after the escape was handled you would have instances of  
plus.)

> I were to require recognizing it.

text/event-stream and text/cache-manifest.

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Sun, 26 Sep 2010 23:02:51 +0200

* Martin J. Dürst wrote:
>Can you give some rough information on where 
>application/www-form-urlencoded is implemented or implementation is planned?

Well, that depends on what you would consider an implementation. You can
use it in the `enctype` attribute of the form element in a HTML document
right now (if you use "get" as the method and UTF-8 for the document)
and it will work in most browsers, except they use "&" as separator. The
label as such isn't used anywhere that I know of yet. I do not know of
anyone other than me planning to support the label.

>Also, in many cases, an identifier without x- denotes just the same as 
>the identifier with the x-. I think it would be good if the registration 
>template contained at least a small reminder and pointer to make clear 
>that it's not exactly the same as the x- variant.

As I understand it, IANA does not copy the template to its web site, and
I don't know of anyone else using the templates out of context, and the
form does not really have a field where this could go, so I am reluctant
to do that.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Sun, 26 Sep 2010 23:52:17 +0200

>make more specific requirements in the forms section of HTML for that  

That would be reasonable, although I would expect the rules for this
type and the "x-" type to be the same except for the separator.

>seems marginal. Not high enough to warrant the cost.

My interest is in having a proper reference for the format for use in
other protocols, where application/x-www-form-urlencoded already works
supporting application/www-form-urlencoded is indeed not very important.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Ian Hickson <[email-masked]>
Date: Sun, 26 Sep 2010 22:11:01 +0000 (UTC)

> supporting application/www-form-urlencoded is indeed not very important.

May I suggest using a different name in that case? Having two so similarly 
named types in active use would be quite confusing.

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Mon, 27 Sep 2010 01:30:38 +0200

>named types in active use would be quite confusing.

The names are as different as the formats are, if I changed the name to,
say, application/name-value-pairs, then there would probably be someone
saying it's confusing to have so different names even though the actual
differences in the format are trivial if not almost meaningless in prac-
tise. So long as I lack a convincing argument to counter that I'll stick
with the current name.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Ian Hickson <[email-masked]>
Date: Mon, 27 Sep 2010 00:55:47 +0000 (UTC)

> tise. So long as I lack a convincing argument to counter that I'll stick

The problem is that since they aren't compatible, yet have so similar 
names, it's quite likely that people will get the names wrong and end up 
with interoperability problems. This would be _especially_ confusing if 
any software ends up implementing both.

---

From: Keith Moore <[email-masked]>
Date: Sun, 26 Sep 2010 21:16:29 -0400

> named types in active use would be quite confusing.

I was thinking the same thing.

Keith

---

From: "Martin J. Dürst" <[email-masked]>
Date: Sun, 26 Sep 2010 14:25:31 +0900

Hello Björn,

Regards,   Martin.

On 2010/09/26 6:14, Bjoern Hoehrmann wrote:
>        U+FEFF character will be considered part of a<name>.
>
>     Person&  email address to contact for further information:

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:[email-masked]

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 27 Sep 2010 16:15:32 +0900

> anyone other than me planning to support the label.

Ok. That implies that if you use another character encoding for the 
document, it will fail, which isn't helpful at all. It also means that 
you can get exactly the same, with the above preconditions, if you use 
the x- variant.

> form does not really have a field where this could go, so I am reluctant

That's okay then. But the draft should be very explicit about it.

Regards,   Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 27 Sep 2010 16:18:48 +0900

+1.    Regards,   Martin.

On 2010/09/27 10:16, Keith Moore wrote:
> _______________________________________________
> ietf-types mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-types
>
>

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 27 Sep 2010 16:23:10 +0900

> text/event-stream and text/cache-manifest.

Ignoring a leading U+FEFF makes sense for formats that are in most ways 
close to files. That's the case for most MIME types. But this one is 
very clearly different; it's more like a micro-level format for a single 
line of data composed of many fields, than a file format. In that case, 
ignoring a leading U+FEFF seems to make very little sense. Also, while a 
large number of servers these days can work with UTF-8 easily, I 
wouldn't know of any server infrastructure that ingnored an U+FEFF in 
this case.

Regards,    Martin.

---

From: "Stephen D. Williams" <[email-masked]>
Date: Mon, 27 Sep 2010 00:47:14 -0700

On 9/26/10 1:32 PM, Bjoern Hoehrmann wrote:
>>> On Sat, 25 Sep 2010 23:14:39 +0200, Bjoern Hoehrmann<[email-masked]>  wrote:

In real life, say in Java, processing such data would throw an exception in the String library decoding methods.  That's fine, 
however it is not really about the www-form-urlencoded level format/parsing, but the next level down, UTF-8.  When I read that 
section, I expected to see data malformed at that syntax level: "%HA%##", etc.  You can simply say that data that is not valid UTF-8 
produces undefined results (i.e. exceptions, etc.).  That's true of the format itself.

Stephen