Subject: Re: [idn] URL encoding in html page

From: Dan Oscarsson <[email-masked]>
Date: Tue, 26 Mar 2002 08:39:37 +0100

Adam M. Costello wrote:

>> As an implementor of a web server, how do I know how to interpret the
>> "host:" field in the HTTP header?  If I see something that looks like
>> ACE, I suppose I would need to decode it to the original UTF-8 to
>> compare with my virtual domain definitions.
>
>No, you wouldn't.  If you don't know anything about IDNs, then in your
>mind all host names are ASCII, so you would do an ASCII comparison
>between the Host: field (which is ASCII) and your virtual domain
>definitions (which are also ASCII), and that would work just fine.

This is only in the mind of a IETF hacker.

In the mind of the common man, a host name is a name composed of
letters, digits and some more characters. Any letters, not just English
letters.
The limitation to English letters is an artificial limit and accepted by
common man. When entering host names, it does not matter where, common
man
will use native character set using any letters used in native language.

The same applies to URLs or URIs. The can contain any letter. Today
lots of people use non-ASCII letters in URLs, embedd them in links in
HTML,
enter them in browsers. Quite often they work as expected.
Common man cannot understand why IETF does not take the easy route and
change its artficial and incomprehensible limits in character allowed.
Just update the current RFC to allow any character outside the ASCII
range.

It time everybody face the facts: if you define an RFC restricting
things
to ASCII where is is natural to allow any letter, people will ignore the
RFC and use any letter anyway. There is so much talk about backward
compatibility - but you only think about ASCII. There is a lot of 
inofficial use of non-ASCII in URLs, host names and other places.
We must support them also.

   Dan

---

From: "David Leung \(Neteka Inc.\)" <[email-masked]>
Date: Tue, 26 Mar 2002 11:43:11 -0500

> This is only in the mind of a IETF hacker.

Totally agree!! If we use ACE as the URL links then all web designers in the
world needs to be retrained for ACE conversion... school has to start
offering a course on ACE compression and decompress in their Comp Sci or
Engineering program : ))

> Just update the current RFC to allow any character outside the ASCII

RFC should be updated according to new things, so if they said RFC doesn't
allow URL/URI with non-ascii, so how about IPv6 back before IPv6 came out, I
dont think URL/URI link can allow IPv6 representation as well...

> things to ASCII where is is natural to allow any letter, people will
ignore the
> We must support them also.

As far as i know a lot of Chinese, Japanese or Korean websites are using CJK
in their URI for directories... If you look at big5.com, they have been
offering Chinese URI in the directory level for a long time!

David Leung
Chief Technology Officer
Neteka Inc.
T: (416) 971-4302
http://w!.neteka.com

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 26 Mar 2002 23:40:02 +0000

"David Leung (Neteka Inc.)" <[email-masked]> wrote:

> If we use ACE as the URL links then all web designers in the world
> needs to be retrained for ACE conversion.

Either the web designer uses an HTML editor, in which case the editor
should know the HTML syntax rules and convert to/from the local charset
as needed, or the web designer uses a text editor, in which case the web
designer is taking responsibility for knowing and obeying the HTML/URI
syntax rules, one of which is that href and src attributes contain only
ASCII characters.

Maybe future HTML/URI specs will allow non-ASCII characters in href and
src attributes, but it's not obvious how to do that without breaking
deployed browsers, and that discussion is for another forum.

AMC

---

From: "David Leung \(Neteka Inc.\)" <[email-masked]>
Date: Tue, 26 Mar 2002 19:01:17 -0500

> syntax rules, one of which is that href and src attributes contain only

So now IDN is a larger scope than we expected, not just browser software
needs to upgrade, even html editor like Dreamweaver, etc needs to upgrade...
interesting idea... I am wondering how many companys will be affected by
this if even HTML editors or text editors need to upgrade, and what about if
they are using an editor and really wants to but something like "bq--xxxx"
on the page, should the editor shows the local encoding on the interface and
keep the ACE in the backend or people are not able to put anything that
looks like an ACE domain on homepage anymore...

> deployed browsers, and that discussion is for another forum.

I guess this is simiilar to say that if some one put a IPv6 link on the page
and will break some browsers that follows old specs, upgrading an RFC
document so that URI can contains non-ASCII is not that hard of a thing...
Hmm... also I am still wondering what kind of variable definition will treat
7bits and 8bits differently and will cause a overflow... : )

---

From: "Adam M. Costello" <[email-masked]>
Date: Wed, 27 Mar 2002 00:57:15 +0000

"David Leung (Neteka Inc.)" <[email-masked]> wrote:

> So now IDN is a larger scope than we expected, not just browser
> software needs to upgrade, even html editor like Dreamweaver, etc
> needs to upgrade...

It should not be surprising that any software that deals with domain
names as such (as opposed to text in general) might need to be upgraded
in order to allow characters in domain names that used to be forbidden.

> I am wondering how many companys will be affected by this if even HTML
> editors or text editors need to upgrade,

HTML editors need to upgrade if they want to fully support IDNs.  Text
editors don't, because they don't deal with domain names as such, they
just deal with text.

> and what about if they are using an editor and really wants to put
> something like "bq--xxxx" on the page, should the editor show the
> local encoding on the interface and keep the ACE in the backend or
> people are not able to put anything that looks like an ACE domain on
> homepage anymore...

An HTML editor knows HTML syntax, so it knows the difference between
the host field of a URI in the src attribute of an IMG tag, versus text
in the content of a P element.  The former is subject to strict syntax
rules because it's a protocol element, while the latter is just text,
regardless of whether it happens to contain substrings that look like
domain names.

AMC

---

From: "Edmon Chung" <[email-masked]>
Date: Wed, 27 Mar 2002 08:31:47 -0500

----- Original Message -----
From: "Adam M. Costello" <[email-masked]>
> in order to allow characters in domain names that used to be forbidden.
(the keyword in this extract is "ANY".  ANY software that deal with domain
names in a "human friendly form" will need to be upgraded.  That means
almost all applications that use domain names isnt it? cause domain names
were supposedly devised to be a "human friendly" name)

This is a revealing point you have pointed out Adam.  While you and some ACE
"everywhere" advocates maintain that ACE "everywhere" requires the least
upgrade, this seems to jump back out at yourselves that afterall, it is not
as small a job as it was advertised.

This and that together, if we are going to need an overhaul what is the
point of going back to ASCII and not move forward to a more advanced schema?

Edmon

---

From: "David Leung \(Neteka Inc.\)" <[email-masked]>
Date: Wed, 27 Mar 2002 12:16:43 -0500

> were supposedly devised to be a "human friendly" name)
Edmon... I think it means every "human readable" UI has to be downgrade not
upgrade... we should have something readable that converts to something
unreadable so that web-designer dont know what they are putting in as URL
: )

> point of going back to ASCII and not move forward to a more advanced
schema?
I always think of IDN this way, we need to upgrade software anyways but
there is two approach...
ACE or UTF8...

ACE will work with "EXISTING" systems but ALL client end(browser, MUA, etc)
and SOME server(Apache, etc) end software actually needs to upgrade in order
to be able to recognize IDN and convert it to ACE back and forth, unless we
want to say that www.ietf.org that we are using nowadays is IDN already : )

On the other hand, we see a lot of applications is moving towards the
support of UTF8/Unicode on the client side, and this is going to be the
future anyways... so why not just upgrade the server software, with one
server software upgraded it will serve many many users... I dont understand
why no one see this simple math : )

---

From: "James Seng/Personal" <[email-masked]>
Date: Thu, 28 Mar 2002 07:37:05 +0800

> unreadable so that web-designer dont know what they are putting in as URL

UTF-8 does not imply "human readable". It means it is encoded in UTF-8.
Dont continue confuse folk display and encoding.

But I agree that ACE is one form of "downgrading" but the target is for
machine readability, not human.

> > as small a job as it was advertised.

In IETF, when face with two proposals, we do engineering trade-off.

Therefore, no one actually say it is "a small a job". Most say IDNA is a
easier task.

> why no one see this simple math : )

You point out the cruial pros & cons over the ACE vs UTF-8. The differ in
opinion is the last part with you dont understand.

My take? If it were up to me, why stick to UTF-8? I'll say lets do one
single upgrade to 128bit, lets called it UTF-128, just in case.

-James Seng

---

From: "Edmon Chung" <[email-masked]>
Date: Wed, 27 Mar 2002 20:33:07 -0500

Its good to here from you James saying that it is smarter to upgrade to UTF8
than to downgrade to ACE.
Although we are not requiring your UTF128 just yet, but I absolutely agree
that the IDN protocol should provision for such upgrades.  Using IDN/EDNS
with versioning, we can provision this.
Glad you are back on the right track James!
Edmon

----- Original Message -----
From: "James Seng/Personal" <[email-masked]>
> Dont continue confuse folk display and encoding.

No one is trying to confuse.  What we mean is that all application that
displays domain names intended for human reading will have to be changed.
And this is a very large number of applications.
> > > This is a revealing point you have pointed out Adam.  While you and
some
> > ACE
> > > upgrade, this seems to jump back out at yourselves that afterall, it
is
> > not

---

From: "James Seng" <[email-masked]>
Date: Thu, 28 Mar 2002 09:45:33 +0800

Edmon,

Please dont put words into my mouth.

> than to downgrade to ACE.

This is not what I say.

> that the IDN protocol should provision for such upgrades.

I did not say this either.

> Using IDN/EDNS

I have not say this either.

> Glad you are back on the right track James!

I am sorry to see you are on the wrong track.

-James Seng

---

From: Martin Duerst <[email-masked]>
Date: Thu, 28 Mar 2002 16:38:23 +0900

>regardless of whether it happens to contain substrings that look like

HTML defines href or source as CDATA, which means just about
'anything goes'. And as I have said in another mail, I don't
think that HTML editors do a lot of checks. Checks may occur
as part of link checking, but that's not on the syntax level,
where http://www.xyz.no-such-tld would be allowed.

Regards,    Martin.

#-#-#  Martin J. Du"rst, I18N Activity Lead, World Wide Web Consortium
#-#-#  mailto:[email-masked]   http://www.w3.org/People/D%C3%BCrst

---

From: Martin Duerst <[email-masked]>
Date: Thu, 28 Mar 2002 16:28:01 +0900

At 23:40 02/03/26 +0000, Adam M. Costello wrote:
>as needed,

I don't know exactly what you mean by 'convert to/from'. Many
HTML editors these days work in Unicode internally, and convert
to whatever encoding the author chooses on input/output.
This conversion is independent of HTML syntax, except for
the use (or not) of numeric character references (&#xHHHH;,...),
and the <meta> charset information.

Some HTML editors may work in a local encoding throughout,
and then they don't do any conversion.

You seem to imply that the HTML editor is checking or should check
the URI syntax, and that they could be upgraded to add a legacy
encoding->Unicode->ACE conversion.

First, I'm not aware of HTML editors actually doing such syntax
checks. Second, there is the very fundamental problem that they
won't be able to do such checks. If you enter hppt:something,
should the editor tell you this is an error? It has no idea
whether hppt is a legal URI scheme or not. Similarly, even if
the HTML editor does the very limited checks that RFC 2396
allows, this doesn't get it very far. For example, while
http: uses generic URI syntax, mailto: uses opaque syntax,
so there is no general way to know where in an URI there is
a domain name. This includes the case that a domain name
is sent as a parameter.

>or the web designer uses a text editor, in which case the web

HTML 4 already says what a browser should do if it finds an
non-ASCII character in an URI. Please see
http://www.w3.org/TR/REC-html40/appendix/notes.html#h-B.2.1
Appendix B.2.1: Non-ASCII characters in URI attribute values
(this is supported by all major browsers in newer versions)

Further information, e.g. on other W3C specs, see also:
http://www.w3.org/International/O-URL-and-ident.html

Of course this doesn't solve the problem of how to handle
IDNs in URIs automatically, but it provides a clear direction.

Regards,    Martin.

---

From: "Edmon Chung" <[email-masked]>
Date: Thu, 28 Mar 2002 09:02:35 -0500

Hi James,
I am sorry if I offended you in any way shape or form with my little playful
note.
However, please explain further your intentions on the comments if not that
you agree ACE is downgrade, UTF8 is a slight upgrade and your UTF128 is a
great upgrade?
Edmon

---

From: "James Seng/Personal" <[email-masked]>
Date: Fri, 29 Mar 2002 04:10:12 +0800

I meant exactly what I said: ACE is a "downgrade" for machine read.
Downgrade here have the same meaning as base64/qp is used to "downgrade"
mail when 8BITMIME is not supported. It does not mean it is better or worst.

UTF-8 is another encoding of Unicode and plainly just UTF-8. I did not make
any comment to say UTF-8 is better or worst. You read too much.

I didnt say that "UTF-128" is better than "UTF-8" is better than ACE either.
UTF-128 is just plainly UTF-128. You jumped too quickly into conclusion.

If we going to have a flag to change everything to 8-bit clean, why stop at
8-bit? 8-bit does not even have the basic bit to support ISO10646. You need
some encoding mechanism to translate ISO10646 to 8-bit, aka, UTF-8.

So why not make all 128bit (or more) clean?

-James Seng

----- Original Message -----
From: "Edmon Chung" <[email-masked]>
To: "James Seng/Personal" <[email-masked]>; "David Leung (Neteka Inc.)"
<[email-masked]>; "IETF idn working group" <[email-masked]>
Sent: Thursday, March 28, 2002 10:02 PM
Subject: Re: [idn] URL encoding in html page

---

From: "David Leung \(Neteka Inc.\)" <[email-masked]>
Date: Thu, 28 Mar 2002 16:06:41 -0500

> any comment to say UTF-8 is better or worst. You read too much.

UTF8 was there for the first place is to make the transition from 8 bits to
16/32 bits Unicode smoother, so the existing 8 bits system will be able to
use it. Same as base64 is to downgrade the 8bits into ASCII to make the
transition smoother...

So should we say that a good design should have fallback(downgrade) for
TRANSITIONAL period only. Then a good IDN design should be able to use ACE
as the fallback for old system, BUT SHOULD also be able to step forward to
use 8/16/32 bits or even more bits when needed.

So we should not say that ACE should be a long term solution for IDN, it
should ONLY be a TRANSITION solution that allows  the LONG TERM solution of
using UTF8 or 16/32 bits Unicode to work.

> So why not make all 128bit (or more) clean?
More bits is good, but when we plan for things more than we need, then it
should be considered to be a waste of resource. So why do we need 128bits
now(i dont think the combined total of characters in all languages in the
world would require that much, not unless we want to include scripts from
oter planets : > ), whereas we need 8/16/32 bits for Unicode, so why not
design a system able to accept ACE as a fallback and also 8/16/32 bits? If
you can justify why designing a system that can handle ASCII as a fallback
and can automatically support 8/16/32 bits Unicode is not a good design,
then I think my thinking is wrong.

---

From: "Adam M. Costello" <[email-masked]>
Date: Thu, 28 Mar 2002 23:52:01 +0000

Martin Duerst <[email-masked]> wrote:

> HTML defines href or source as CDATA, which means just about 'anything
> goes'.

href and src are defined in the DTD as %URI; which in turn is defined as
CDATA, but section 6.4 of the HTML spec says:

    This specification uses the term URI as defined in [URI]

    URIs are represented in the DTD by the parameter entity %URI;.

[URI] links to RFC-2396.  So the href and src values are indeed
constrained by RFC-2396, even though this constraint cannot be expressed
in the DTD.

> > Either the web designer uses an HTML editor, in which case the
> > editor should know the HTML syntax rules and convert to/from the
> > local charset as needed,
>
> I don't know exactly what you mean by 'convert to/from'.

I mean that HTML editors should know that only a certain set of ASCII
characters are allowed in the href and src attributes, and if the user
tries to put other characters in there, the HTML editor should convert
the string to something that conforms to the HTML spec (if it knows how
to do such a conversion without changing the intended meaning) or at
least it should alert the user that they are violating the HTML spec.

Wouldn't you agree that an HTML editor that puts non-ASCII characters
into href and src attributes, in plain violation of the HTML spec,
without even alerting the user, is not quite living up to the name "HTML
editor"?  Because the thing it's editing isn't really HTML anymore.

> encoding->Unicode->ACE conversion.

Regardless of whether today's HTML editors already check URI syntax,
they could be upgraded to convert IRIs input by the user into URIs in
the document, and that conversion could involve ToASCII.

> If you enter hppt:something, should the editor tell you this is an
> error?

No, because it's not an error.  A warning might be helpful, or might be
annoying, just like spell-checking.

> Similarly, even if the HTML editor does the very limited checks that
> RFC 2396 allows, this doesn't get it very far. For example, while
> http: uses generic URI syntax, mailto: uses opaque syntax, so there is
> no general way to know where in an URI there is a domain name. This
> includes the case that a domain name is sent as a parameter.

True, the editor can't be expected to know the syntax for every URI
scheme.  For the ones it does know, it can fully check the syntax.  For
any URI that begins with / or scheme:/, it can check the generic URI
syntax (which is quite detailed).  For opaque URIs with unknown schemes,
it can at least check for characters disallowed in all URIs.

I think that gets you pretty far in practice.  The vast majority of
opaque URIs could also be handled if the editor knows just a few schemes
(mailto, and maybe news and telnet).

> HTML 4 already says what a browser should do if it finds a

It says:

    Although URIs do not contain non-ASCII values (see [URI], section   
    2.1) authors sometimes specify them in attribute values expecting   
    URIs (i.e., defined with %URI; in the DTD).  For instance, the      
    following href value is illegal:                                    
                                                                                
    <A href="http://foo.org/Håkon">...</A>

    We recommend that user agents adopt the following convention for
    handling non-ASCII characters in such cases:

     1. Represent each character in UTF-8 (see [RFC2279]) as one or more
        bytes.

     2. Escape these bytes with the URI escaping mechanism (i.e.,       
        by converting each byte to %HH, where HH is the hexadecimal     
        notation of the byte value).                                    

    This procedure results in a syntactically legal URI (as defined in
    [RFC1738], section 2.2 or [RFC2141], section 2) that is independent
    of the character encoding to which the HTML document carrying the
    URI may have been transcoded.

RFC-1738 section 2.2 seems to say that %HH escapes are allowed in the
host part of a URI, but the BNF in section 5 clearly prohibits %HH
escapes in the host part.  And RFC-2396, which is a normative reference
of the HTML spec and which updates RFC-1738, is unambiguous in its
prohibition of %HH escapes in the host part.

Therefore, it is not true that "this procedure results in a
syntactically legal URI".  I suggest that the procedure be ammended so
that it does result in a syntactically legal URI.  That would not be
a serious change in the HTML spec, because the procedure doesn't even
apply to valid HTML, it applies only to invalid HTML.

AMC

---

From: "Bruce Thomson" <[email-masked]>
Date: Fri, 29 Mar 2002 11:57:03 +0900

> So why not make all 128bit (or more) clean?

Granted, such 128-bit characters might be useful. This would allow various
organizations to define their own characters sets. Space in the global
character encoding space would be allocated in the same way as
IP address space is allocated under IPv6.

But to conserve file space, it would probably be best to allow intermixing of
128-bit characters with ASCI text. UTF-8 continues to be the way to do
this, since it just a compression scheme that does not really depend on the fact
that Unicode is currently limited to 32 bits. It could just as easily be extended
to work with much larger character sets.

Bruce

---

From: "Sun Guonian" <[email-masked]>
Date: Fri, 29 Mar 2002 11:15:11 +0800

James Seng/Personal wrote:

...
> The question is really why 8/16/32 bit Unicode is better than 5bit (ACE)?

I think the former is based on character and the latter is based on string.
Maybe it is a advantage of the former for design many Applications.

> 
> 8 is not neccessary better than 5 just like 32 is not neccessary better than
> 8. It is all engineering trade off.
> 
> So what is the advantage of moving towards more bits? And how much more is
> enough?
> 
> ps: I am not agreeing or disagreeing with a transition. I reserved my
> judgement later.
> 
> -James Seng

-- 
******************************************************
* China Internet Network Information Center (CNNIC)  *
*   Sun Guonian                     No.4, S.4 Street *
*                                   Zhong Guan Cun   *
* Phone: +86-10-62619750 ext. 3016  Haidian District *
* Email: [email-masked]           Beijing          *
* http://www.cnnic.net.cn/          P.O.Box: 100080  *
* �й�����������Ϣ����    �����                     *
******************************************************

---

From: "Bruce Thomson" <[email-masked]>
Date: Fri, 29 Mar 2002 12:40:41 +0900

> The question is really why 8/16/32 bit Unicode is better than 5bit (ACE)?

ACE and UTF-8 are just compression algorithms that squeeze larger
Unicode characters. ACE is more efficient than UTF, although more
complex.

But the claim to fame that UTF-8 has is that it is a standard that
idn can reference, and it is coming into widespread use elsewhere.

So moving to UTF-8 long term seems like such an obvious choice
is surprises me that it even gets debated.

Of course, Punycode is a nice compression method... Maybe we should
take it to the Unicode forum and suggest that it be the next version of
UTF-8?

But if that doesn't happen, it should logically be phased out.

Bruce

---

From: "James Seng/Personal" <[email-masked]>
Date: Fri, 29 Mar 2002 10:42:24 +0800

> use it. Same as base64 is to downgrade the 8bits into ASCII to make the

Agree. Spend more time to think of this and why arguments on ACE vs UTF-8 is
really irrelevant except religious prejudice.

> use 8/16/32 bits or even more bits when needed.

Maybe.

> using UTF8 or 16/32 bits Unicode to work.

The best way is to write a draft, explain how you wish to transit from the
IDNA to a long term solution (The closest I seen is UDNS). Then gather the
rough consensus so we can move it forward.

IMHO, the problem of moving away from ACE to a long term solution is a
smaller one compared to the problem of what is the long term solution.

> should be considered to be a waste of resource.

Read: IPv6.

> So why do we need 128bits

How many new han ideograph are created every day? How many ancient scripts
not encoded yet (e.g. Linear B)?

Or how about a new 128bit character table whereby allocation can be
allocated to 2^64 different locale, each locale get 64bit of space to put
their own encoding? (I know, sound much like IPv6).

Are you in the position to predict the future?

My best *guess* is ISO10646 will get slowly get adoption over the next 20
years. I *guess* UTF-8 will get more popular in the next 10 years. I *guess*
compressed bitstream (i.e. 1bit) ISO10646 will see some more adoption in
certain place where space is a constraint. I also *guess* there will be some
initative to do locale-based table (vs script based) but they will be faced
with huge barrier. But I could only guess...

-James Seng

---

From: "Doug Ewell" <[email-masked]>
Date: Thu, 28 Mar 2002 21:21:35 -0800

> Are you in the position to predict the future?

How far into the future?  Linear B is planned for Unicode 4.0 -- look
for it about 18 months from now -- and I hate to say this, but Linear B
fits into a single 256-character block.  And the number of other ancient
scripts that experts know enough about to encode is not that great.
Sorry, but there simply aren't MILLIONS of characters waiting to be
encoded.

-Doug Ewell
 Fullerton, California

---

From: "Doug Ewell" <[email-masked]>
Date: Thu, 28 Mar 2002 21:23:19 -0800

David Leung (Neteka Inc.) <[email-masked]> wrote:

> More bits is good, but when we plan for things more than we need,
> then it should be considered to be a waste of resource. So why do
> we need 128bits now(i dont think the combined total of characters
> in all languages in the world would require that much, not unless
> we want to include scripts from oter planets : > ), whereas we need
> 8/16/32 bits for Unicode, so why not design a system able to accept
> ACE as a fallback and also 8/16/32 bits? If you can justify why
> designing a system that can handle ASCII as a fallback and can
> automatically support 8/16/32 bits Unicode is not a good design,

Is my memory faulty, or wasn't it the CJK users who complained the
loudest that UTF-8 was unfairly discriminatory because it required 3
bytes for each CJK character, and that some sort of compression (such as
now provided by Punycode) was essential to the success of IDN?

-Doug Ewell
 Fullerton, California

---

From: "Doug Ewell" <[email-masked]>
Date: Thu, 28 Mar 2002 21:24:01 -0800

Bruce Thomson <[email-masked]> wrote:

> But to conserve file space, it would probably be best to allow
> intermixing of 128-bit characters with ASCI text. UTF-8 continues
> to be the way to do this, since it just a compression scheme that
> does not really depend on the fact that Unicode is currently
> limited to 32 bits. It could just as easily be extended to work
> with much larger character sets.

This is not even close to true.  UTF-8 is very much dependent on the
32-bit architecture of Unicode, and in fact is constrained to 31-bit
code points.  A quick check of the "10xxxxxx 10xxxxxx..." chart in RFC
2279, or in the Unicode Standard or ISO/IEC 10646, will confirm that.

And the word "currently," as used to refer to either the 21-bit or the
32-bit limit of Unicode/10646, is being used way too cavalierly.
Unicode is not going to be expanded beyond U+10FFFD, and nobody can
think of a non-whimsical reason why it should be.

-Doug Ewell
 Fullerton, California

---

From: "David Leung \(Neteka Inc.\)" <[email-masked]>
Date: Fri, 29 Mar 2002 02:19:56 -0500

>Maybe it is a advantage of the former for design many Applications.

Totally agree the advantage of the former is to be able to have existing
software or software that plans to have Unicode support to be able to work
without performing EXTRA ACE conversion!!

---

From: "James Seng" <[email-masked]>
Date: Fri, 29 Mar 2002 15:17:08 +0800

Nameprep?

----- Original Message -----
From: "David Leung (Neteka Inc.)" <[email-masked]>
To: <[email-masked]>; "James Seng/Personal" <[email-masked]>
Cc: "Edmon Chung" <[email-masked]>; "IETF idn working group"
<[email-masked]>
Sent: Friday, March 29, 2002 3:19 PM
Subject: Re: [idn] URL encoding in html page

---

From: "David Leung \(Neteka Inc.\)" <[email-masked]>
Date: Fri, 29 Mar 2002 02:25:16 -0500

> But if that doesn't happen, it should logically be phased out.

That's right if ACE is such a good "compression and encoding" scheme for
Unicode, I think Unicode consortium should adopts it as the new UTF-P or
UTF:-P, if they already have UTF8 that is widely use nowadays and also will
be more use in the future, why not use UTF8 as a long term, and re-inventing
the wheel now.

---

From: "David Leung \(Neteka Inc.\)" <[email-masked]>
Date: Fri, 29 Mar 2002 02:38:31 -0500

> rough consensus so we can move it forward.
Sure we happy to come up with a draft to talk about the transition from IDNA
to UDNS(or something similar) and submit it... : )

> > should be considered to be a waste of resource.
So you mean there will be more characters that we will use than IP addresses
in the world that everyone knows is running out with IPv4...

> not encoded yet (e.g. Linear B)?
True, those han ideograph are mostly slang words or swear words too, it
would be nice if we can have all the words formed with the combination of
the "door" in chinese to be able to input on computers... haha : )

> Are you in the position to predict the future?
I did not try to predict the future, (i think you just did below this :>)...
i think you misunderstand my last posting, i only said for now we DONT NEED
128bits to encode ALL characters in the world, USING more than NEEDED is a
waste of resources, so we should not say IPv6 is a waste of resource because
there is already a need there with IPv4 almost already ran out!!

> with huge barrier. But I could only guess...
I hope there is some analysis behind these "predictions" : )

> The question is really why 8/16/32 bit Unicode is better than 5bit (ACE)?
Some existing software is already supporting this, so require less upgrade.
The software that are newly released are supporting mostly 8/16bits version
of Unicode, like newer OSes, browsers, etc... and I am not saying ACE is
bad... I am saying ACE and UTF8 or UCS should be able to co-exists in a
system, so transition will be minimal and that should be a good design.

---

From: "David Leung \(Neteka Inc.\)" <[email-masked]>
Date: Fri, 29 Mar 2002 02:41:45 -0500

> to work with much larger character sets.

Agrees, the design of how UTF-8 encodes and decodes can be expanded to
support more bits... so that's why UTF8 should be reasonable to be the long
term solution for designing i18n applications, including IDN...

---

From: "David Leung \(Neteka Inc.\)" <[email-masked]>
Date: Fri, 29 Mar 2002 02:46:55 -0500

> Sorry, but there simply aren't MILLIONS of characters waiting to be

But if we are talking about the future where aliens from other planets join
us for the discussion at IETF, then maybe we need to encode all the
characters from their planets too!! Also there are billions of stars and
planets out there, who knows : )

---

From: "David Leung \(Neteka Inc.\)" <[email-masked]>
Date: Fri, 29 Mar 2002 02:51:59 -0500

> now provided by Punycode) was essential to the success of IDN?

Dont judge all CJK users to be the same person, I guess you really have
faulty memory or even memory leak, that wasnt me!! Also dont view all CJK
people to be the same group, this maybe kind of discrimination?!

The guy who said that was Soobok Lee(he is Korean and I am Chinese) or
someone else!!  I am the one that keep saying UTF8 or 16 bit is good and ACE
should only be transitional...!!

David Leung
Chief Technology Officer
Neteka Inc.
T: (416) 971-4302
http://w!.neteka.com

---

From: "James Seng" <[email-masked]>
Date: Fri, 29 Mar 2002 16:08:20 +0800

> Unicode characters. ACE is more efficient than UTF, although more

UTF-8 does not have any compression.

> idn can reference, and it is coming into widespread use elsewhere.

Yes, agree it is "coming into widespread'. Until it is widespread, it is
only "Coming Soon". or "Coming Soon to your favourite OS RSN(tm)".

> is surprises me that it even gets debated.

Any technical decision have to based on well-founded technical reasons.

The transcoding can be bypassed if everything is in UTF-8 is a good argument
I can accept. To use UTF-8 just because everyone else is *going to* do so
does not sound too conviencing. And "UTF-8 is obvious" is not an argument I
can buy.

> But if that doesn't happen, it should logically be phased out.

Lets bring base64 and qp over to Unicode and also %-escaping and other
adaptive mechanism out there, then phased them all out! Lets phase out all
other local encoding too. The world should only have one and only one
encoding! Long live UTF-8, for all fonts, for input, for all protocols, all
systems, all software! This the utopia dream of all I18N engineers (myself
included).

But enough dreaming. Lets get back to ugly reality. *sigh*

-James Seng

---

From: "Soobok Lee" <[email-masked]>
Date: Fri, 29 Mar 2002 17:31:24 +0900

----- Original Message ----- 
From: "Doug Ewell" <[email-masked]>
> now provided by Punycode) was essential to the success of IDN?
 
Hmm.. *I*  had proposed to kill DUDE and choose ACE-Z (code simplicity does not matter so much).
*This WG* decided to drop DUDE and chose ACE-Z. 

this is the list of supporters of ACE-Z. there are many non-CJK folks here.

a.. Adam M. Costello [email-masked] 
a.. David C Lawrence [email-masked] 
a.. Makoto Ishisone [email-masked] 
a.. Rick H Wesson [email-masked] 
a.. Scott Hollenbeck [email-masked] 
a.. Soobok Lee [email-masked] 
a.. Yoshiro YONEYA [email-masked] 
a.. Yves Arrouye [email-masked] 
a.. James Seng [email-masked] 
a.. Bright Fulton [email-masked] 
a.. Yasuhiro Morishita [email-masked] 
a.. Zhu Yu [email-masked] 
a.. Martin Duerst [email-masked]

---

From: "James Seng" <[email-masked]>
Date: Fri, 29 Mar 2002 16:46:51 +0800

> to UDNS(or something similar) and submit it... : )

This is definately more productive :-) Please do so.

> in the world that everyone knows is running out with IPv4...

The lesson on IPv4/6 is once upon a time, someone sclaim 32 bits is all we
need for IP.

> the "door" in chinese to be able to input on computers... haha : )

Not all ideograph are slang or swear like in Cantonese.

There are about 500+ han ideograph used in Singapore alone that is not
inside ISO10646. Of which only 25 is in IRG review. Most of them cannot even
be justified under normal process because it is character "created" for some
person name.

Fortune Teller: "Your fire element is too strong and your water element is
lacking. Your bad luck comes from the unbalance Yi-Ching. Lets add some
water, 3 dot (water kangxi) to this character of your name". Bingo! A new
character is born.

But alas, I am just rattling...lets drop this.

> I hope there is some analysis behind these "predictions" : )

I am guessing, not predicting. My guess about future is as good as anyone
else, which is nothing.

> Some existing software is already supporting this, so require less
upgrade.

Some software support UTF-8. No software support Nameprep-UTF-8. Software
that deals with IDN using UTF-8 still have to upgrade. But lets not jump
ahead. I'll wait for your draft.

-James Seng

---

From: Keld Jørn Simonsen <[email-masked]>
Date: Fri, 29 Mar 2002 11:15:14 +0100

> is surprises me that it even gets debated.

I think the UTF-8 is the way to go forward too, and I actuallt think
this is also IESG policy, viz RFC 2277 and RFC 2130. The UTF-8 RFC 2279 is
the only standards track RFC on charsets for the same reason. Citing
from RFC 2277, the IESG policy on charactyer sets and language:

   "Protocols MUST be able to use the UTF-8 charset, which consists of
   the ISO 10646 coded character set combined with the UTF-8 character
   encoding scheme, as defined in [10646] Annex R (published in
   Amendment 2), for all text."

I dont mind having an ASCII fallback as we also have it in email, but
going into the other encoding forms of ISO 10646 is discouraged by
IESG, as they do not want to see a lot of encodings for the same
character set, with the possible problems of interoperability.
So UCS-2, UCS-4, UTF-16, UTF-16-LE, UTF-16-BE etc are discouraged.

Best regards
Keld Simonsen

---

From: Sampo Syreeni <[email-masked]>
Date: Fri, 29 Mar 2002 13:16:18 +0200 (EET)

On Fri, 29 Mar 2002, James Seng wrote:

>> So you mean there will be more characters that we will use than IP
>> addresses in the world that everyone knows is running out with IPv4...
>

That's a bit different -- nobody's supposed to remember all of the IP's
out there. Someone *is*, when it comes to characters. Besides, not too
many characters are out there, considering that the largest group, Chinese
ideographs, has largely been encoded in Unicode already. (True, there are
esoteric ones. Now, after Unicode 3.1 with its CJK Unified Ideographs
Extension B, that group is rapidly shrinking, too.)

>inside ISO10646. Of which only 25 is in IRG review. Most of them cannot
>even be justified under normal process because it is character "created"
>for some person name.

Then we might argue that such characters are analogous to corporate logos,
and *shouldn't* in fact be encoded. Not before the glyphs are in
widespread use and some sort of semantic distinction has arisen between
them and their progenitors.

>Fortune Teller: "Your fire element is too strong and your water element
>is lacking. Your bad luck comes from the unbalance Yi-Ching. Lets add
>some water, 3 dot (water kangxi) to this character of your name". Bingo!
>A new character is born.

No. A new *glyph* for the same *character* is born. If the meaning does
not change, I would view this as a glyph variant. From there on, it's the
person's responsibility to lobby for the inclusion of this character in
fonts as an alternative representation. No need for standards
organizations to get involved...

>Some software support UTF-8. No software support Nameprep-UTF-8.
>Software that deals with IDN using UTF-8 still have to upgrade. But lets
>not jump ahead. I'll wait for your draft.

I too think going directly to UTF-8 would be more elegant than using ACE.
However, it is not a panacea, either -- AFAICS, all of the problems with
ACE and ideographs comes from the way these characters are used. Not ACE.
So this trouble won't be going *anywhere* if we jump to UTF-8. The only
thing one would be buying with such a move would be elegance and, perhaps,
unlimited host name lengths.

Sampo Syreeni, aka decoy - mailto:[email-masked], tel:+358-50-5756111
student/math+cs/helsinki university, http://www.iki.fi/~decoy/front
openpgp: 050985C2/025E D175 ABE5 027C 9494 EEB0 E090 8BA9 0509 85C2

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 29 Mar 2002 06:41:17 -0800

The consortium does not encourage the development of further
general-purpose UTFs. Endlessly multiplying such formats just leads to
confusion.

As to compression formats, once ACE is already defined by an RFC it is
better to simply reference that in any Unicode documentation that
needs to. Keeping the definition in one place is easier than having
two definitions that require synchronization.

Mark
—————

Γνῶθι σαυτόν — Θαλῆς
[For transliteration, see http://oss.software.ibm.com/cgi-bin/icu/tr]

http://www.macchiato.com

----- Original Message -----
From: "David Leung (Neteka Inc.)" <[email-masked]>
To: "Bruce Thomson" <[email-masked]>; "IETF idn working group"
<[email-masked]>
Sent: Thursday, March 28, 2002 23:25
Subject: Re: [idn] URL encoding in html page

> > > The question is really why 8/16/32 bit Unicode is better than
5bit
> (ACE)?

---

From: "Doug Ewell" <[email-masked]>
Date: Fri, 29 Mar 2002 06:45:32 -0800

> planets out there, who knows : )

This is what I mean about listening only to "non-whimsical" arguments
about the number of characters that need to be encoded.  If Earth does
get a visit from space aliens, I suspect character encoding will not be
the first problem that needs to be solved.

Please, let's not cloud the real issues of character encoding.

-Doug Ewell
 Fullerton, California

---

From: "Doug Ewell" <[email-masked]>
Date: Fri, 29 Mar 2002 07:03:15 -0800

> be more use in the future, why not use UTF8 as a long term, and
re-inventing

Punycode is not the invention of Unicode, despite the name.  Unicode
already has a perfectly good Standard Compression Scheme (see
http://www.unicode.org/unicode/reports/tr6/).  But it uses non-ASCII
bytes, and so SCSU-encoded domain names would probably not pass cleanly
in a world that expects ASCII-only, any more than UTF-8 would.

-Doug Ewell
 Fullerton, California

---

From: "Doug Ewell" <[email-masked]>
Date: Fri, 29 Mar 2002 07:07:22 -0800

> term solution for designing i18n applications, including IDN...

Please, everybody, read the definition of UTF-8 before making these
claims that it supports an encoding structure wider than 31 bits.  (And
please read the statements of WG2 and Unicode concerning expansion
beyond U+10FFFD.)

UTF-8 seems to have taken on a mystical quality whereby it can now
provide food for the starving, shelter for the homeless, etc.

-Doug Ewell
 Fullerton, California