Subject: Re: Differences between Kerbeors revisions and PKINIT

From: Jeffrey Altman <[email-masked]>
Date: Sat, 5 Aug 2000 11:32:43 EDT

Back to the question of the security risk of UTF-8 there have been
several threads on the Linux-UTF8 list archived at:

  http://mail.nl.linux.org/linux-utf8/

The thread is called "Re: utf-8 encoding scheme", "Re: UTF-8 and
security", and "Substituting malformed UTF-8 sequences in a decoder".

Most of the issues relate to wording in the Unicode Standard relating
to conversions between UCS2/UCS4 and UTF-8 which state that use of
non-normative forms for character representaton is a "shall not" not a
"must not".  In particular, the following quote is fairly scary:

  "When converting from UTF-8 to Unicode values, however,
  implementations do not need to check that the shortest encoding is
  being used, which simplifies the conversion algorithm."

This would appear to be in conflict with the UTF-8 RFC.  But what it
means is that authors of software that interoperate with us may very
well not use the shortest possible representation for encoding data as
UTF8.  And attackers most definitely will attempt to take advantage of
this.

Markus Kuhn has probably said it best:

"Remember that UTF-8 came with a promise: ASCII compatibility.

"UTF-8 implements certainly what I call ASCII compatibility of the first
kind (also known as file-system safety):

  "ASCII bytes represent only ASCII characters (and are not parts of  
  other characters).

"But many decoder authors fail to provide what UTF-8 should also
provide, namely what I like to call ASCII compatibility of the second kind:

  "ASCII characters can only be represented by ASCII bytes (and not by
  combinations of other bytes)."

It is the lack of checking for shortest forms in the decoders that
make UTF8 a potential security risk.

Markus Scherer has some interesting ideas on how to correct the
problems that are probably outside the scope of this discussion but  
for completeness:

"maybe it is time for the unicode standard and wg2 to consider
restricting the valid utf-8 byte sequences to "regular" ones. this
would result in, for example, declaring bytes 0xc0, 0xc1, and
0xf8..0xff as entirely unused in utf-8 (0xfe & 0xff are already) and
to restrict the length of byte sequences to 4. wg2 may insist on
keeping the "regular" 5- and 6-byte forms to cover ucs-4 entirely,
though, even though those will never be used."

He refers to this modified for of UTF8 as "utf-8c1", the secure
version of UTF8.

The important thing to remember is that it is crucial for any UTF8
input string to be tested for compliance before using it in a trusted
manner. 

- Jeff

                  Jeffrey Altman * Sr.Software Designer
                 The Kermit Project * Columbia University
               612 West 115th St * New York, NY * 10025 * USA
     http://www.kermit-project.org/ * [email-masked]