Subject: UTF8 vs. Punycode

From: Russ Housley <[email-masked]>
Date: Mon, 13 Aug 2007 15:35:10 -0400

This issue was raised by my review of
http://www.ietf.org/internet-drafts/draft-ietf-smime-bfibecms-03.txt

However, I think that the issue goes beyond this document.  The 
decision made here ought to set a precedent.

My original comment was:

>1) Section 2 defines EmailIdentitySchema as a UTF8String.  The text says:
>
>       E-mail addresses that contain non-ASCII
>       characters MUST be encoded using punycode [RFC3492].
>
>Therefore, the result of the encoding should always be ASCII.  Why 
>is an UTF8 String needed?

I think that the use of punycode in an IA5String was the correct 
approach.  This approach uses the ToASCII algorithm to generate an 
ASCII string from the Unicode string.  The punycode can then be 
operated on using the usual subroutines to handle copy, compare, and such.

The alternative is to use UTF8, which cannot be operated on using the 
usual subroutines to handle copy, compare, and such.

I understand that the authors are receiving conflicting advice on 
this point from Jim Schaad and myself.  I have sent email to Jim to 
try and understand his point of view, but Jim has not responded.  The 
text in the most revision of this document was updated to use UTF8, 
without requiring punycode.  I think this was the wrong way to 
resolve this one.  I would prefer to retain the use of punycode and 
carry the email address in an IA5 string.  The reason that I prefer 
this solution is that traditional character comparison routines can be used.

What do others think?

Russ

---

From: "Timothy J. Miller" <[email-masked]>
Date: Mon, 13 Aug 2007 15:30:06 -0500

> decision made here ought to set a precedent.

What's the risk of *not* forcing to ASCII via punycode?  Leave aside  
flaws in unicode handling routines for the moment.  In the IBE  
context, a sender would derive an "incorrect" public key (more  
correctly, a different public key) and the recipient would be unable  
to decrypt the message.

-- Tim

---

From: Russ Housley <[email-masked]>
Date: Mon, 13 Aug 2007 16:55:44 -0400

Tim:

I think that the string comparison routines are the key point.  The 
recipient must recognize their own email address in the recipient 
info structure.  This is an exact match comparison (as opposed to the 
more complicated ones for sorting).  If the recipient does not locate 
the correct entry in the recipient info structure, then decryption is 
not possible.

Russ

---

From: Simon Josefsson <[email-masked]>
Date: Tue, 14 Aug 2007 11:30:17 +0200

> correctly, a different public key) and the recipient would be unable

One risk is that the specification cannot use Unicode code points from a
newer Unicode version than IDNA ToASCII supports, right now that means
Unicode 3.2.  Since some time, we have Unicode 5.0, which includes many
important code points for a variety of languages.  Newer versions of
Unicode will be released in the future.  Having to update this
specification for every IDNA/ToUnicode release seems sub-optimal to me.

I believe it is better to teach protocols how to deal with non-ASCII
data, rather than relying on IDNA idiosyncrasies in every IETF protocol.

The choice to remain with ASCII has been made for the DNS protocol,
where it makes some sense due to backwards compatibility reasons, but
that does not mean we have to make the same choice in every IETF
protocol.  Some IETF protocols can easily negotiate support for UTF-8 on
both sides, and using UTF-8 rather than Punycode seems more robust and
like better engineering to me.

/Simon

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 14 Aug 2007 07:25:22 -0700

At 11:30 AM +0200 8/14/07, Simon Josefsson wrote:
>Unicode 3.2.

That is not necessarily true. The current version of IDNA supports 
Unicode version 3.2. A future version of IDNA may support later 
versions of Unicode.

>Since some time, we have Unicode 5.0, which includes many
>important code points for a variety of languages.

Not to get into a flame-war here, but I think that "important" is a 
gross overstatement. There are a few minor scripts and personal name 
characters that are not included in Unicode 3.2, but there has been 
essentially no public pressure on the IETF to update them.

>Newer versions of

Fully agree. It is the responsibility of the IETF to make sure that 
is not needed.

>data, rather than relying on IDNA idiosyncrasies in every IETF protocol.

We disagree here, particularly for security protocols.

>protocol.

This makes no sense here. The protocol in question is representing 
email addresses. The right side of an email address is a domain name.

>Some IETF protocols can easily negotiate support for UTF-8 on

Un-normalized fails miserably when exact matching is needed, such as 
it is in IBE.

--Paul Hoffman, Director
--VPN Consortium

---

From: Simon Josefsson <[email-masked]>
Date: Wed, 15 Aug 2007 00:05:41 +0200

> Unicode version 3.2. A future version of IDNA may support later

I believe the assumption here is that the document would reference the
current IDNA version.  Or are you suggesting that this document should
wait until the IDNAbis effort is done?

> Un-normalized fails miserably when exact matching is needed, such as

Sending UTF-8 does not preclude normalizing from happening when it is
needed.

/Simon

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 14 Aug 2007 15:38:11 -0700

>wait until the IDNAbis effort is done?

The former. But that does not change what I said: a later version of 
IDNA that has the additional characters you were talking about could 
be replaced in the spec with near-zero effort.

>Sending UTF-8 does not preclude normalizing from happening when it is

Ah, I misunderstood your proposal. I thought you meant "send UTF-8", 
not "do some normalization that is not specified here, then send 
UTF-8". It would be good if you could give your complete proposal so 
people can compare them fully.

--Paul Hoffman, Director
--VPN Consortium

---

From: "Turner, Sean P." <[email-masked]>
Date: Thu, 30 Aug 2007 13:47:39 -0400

Did this issue get resolved? 

>-----Original Message-----
>From: [email-masked] 
>[mailto:[email-masked]] On Behalf Of Paul Hoffman
>Sent: Tuesday, August 14, 2007 6:38 PM
>To: Simon Josefsson
>Cc: Timothy J. Miller; Russ Housley; [email-masked]; 
>[email-masked]; [email-masked]
>Subject: Re: UTF8 vs. Punycode
>
>
>>>>One risk is that the specification cannot use Unicode code points 
>>>>from a newer Unicode version than IDNA ToASCII supports, right now 
>>>>that means Unicode 3.2.
>>>
>The former. But that does not change what I said: a later 
>version of IDNA that has the additional characters you were 
>talking about could be replaced in the spec with near-zero effort.
>
>>>>both sides, and using UTF-8 rather than Punycode seems more robust 
>>>>and like better engineering to me.
>>>
>Ah, I misunderstood your proposal. I thought you meant "send 
>UTF-8", not "do some normalization that is not specified here, 
>then send UTF-8". It would be good if you could give your 
>complete proposal so people can compare them fully.
>
>--Paul Hoffman, Director
>--VPN Consortium
>

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 30 Aug 2007 11:46:29 -0700

>Did this issue get resolved?

Simon did not propose a full alternative that we could consider. 
Barring that, I would still advocate making the simple change to an 
IA5String.

--Paul Hoffman, Director
--VPN Consortium