Subject: [idn] Re: I don't want to be facing 8-bit bugs in 2013

From: Robert Elz <[email-masked]>
Date: Wed, 20 Mar 2002 14:21:12 +0700

Date:        Wed, 20 Mar 2002 14:32:41 +0859 ()
    From:        Masataka Ohta <[email-masked]>
    Message-ID:  <[email-masked]>

  | IDNA does _not_ work, because Unicode does not work in International
  | context.

This argument is bogus, and always has been.   If (and where) unicode
is defective, the right thing to do is to fix unicode.

That is, it isn't the principle of a single encoding of all characters
that anyone is objecting to here, it is that some specific characters
have been implemented incorrectly ("merged" with others) as I understand it.

I'm not competent to decide how important this problem is, and this is
not the forum to debate it anyway (so please don't reply just to tell me
how significant the problem is, nor why).   Do that with whoever maintains
unicode.

If you can't get enough of the unicode experts to agree that there's a
problem that needs fixing, then by definition, there isn't.  That's just
the same way as the IETF works (whether the unicode group actually work
this way or not - if not, they should...) - that a few people believe
something is broken is irrelevant if they can't demonstrate that well
enough to sway others to agree with them.

So, stop arguing against unicode (10646) - just fix any problems it has.

kre

---

From: Masataka Ohta <[email-masked]>
Date: Tue, 19 Mar 2002 23:22:45 -0800

id OAA00306; Wed, 20 Mar 2002 14:32:42 +0900
Subject: [idn] Re: I don't want to be facing 8-bit bugs in 2013
In-Reply-To: <[email-masked]> from "D. J. Bernstein" at
 "Mar 20, 2002 04:32:50 am"
To: "D. J. Bernstein" <[email-masked]>
Date: Wed, 20 Mar 2002 14:32:41 +0859 ()
CC: [email-masked], [email-masked], [email-masked], [email-masked]
X-Mailer: ELM [version 2.4ME+ PL68 (25)]
Sender: [email-masked]
Precedence: bulk

D. J. Bernstein;

> Paul Robinson writes:
> > You tell him that although it's gobbledygook to people without greek
> > alphabet support, it will still work. It's not convenient, but it WILL
> > work. Guaranteed.
>
> False. IDNA does _not_ work. IDNA causes interoperability failures.

IDNA does _not_ work, because Unicode does not work in International
context.

> People who say that IDN is purely a DNS issue are confused.

It's purely a cultural issue.

> In fact, the cost of fixing UTF-8 displays is much _smaller_ than the
> cost of fixing IDNA displays. UTF-8 has been around for many years, has
> built up incredible momentum (as illustrated by RFC 2277), and already
> works in a huge number of programs.

In international context, it is technically impossible to properly
display Unicode characters.

There is no implementation exist.

While some implementations work in some localized context, local
character set serves better for the context.

Masataka Ohta

---

From: Masataka Ohta <[email-masked]>
Date: Thu, 21 Mar 2002 00:57:18 +0859 ()

Kre;

> is defective, the right thing to do is to fix unicode.

Unicode is not usable in international context.

There is no unicode implementaion work in international context.

Unicode is usable in some local context.

There is some unicode implementaion work in local contexts.

However, the context information must be supplied out of band.

And, the out of band information is equivalent to "charset"
information, regardless of whether you call it "charset" or
not.

> So, stop arguing against unicode (10646) - just fix any problems it has.

Fix is to supply context information out of band to specify which
Unicode-based local character set to use.

With MIME, it is doable by using different charset names for
different local character set.

See, for example, RFC1815.

As for IDN, it can't just say "use charset of utf-7" or "use charset
of utf-8".

IDN can say "for Japanese, use charset of utf-7-japanese".

Or, if you insist not to distinguish different local character sets by
MIME charset names, IDN can say "use charset of utf-7, but, for
Japanese, use Japan local version of utf-7" and somehow specify
how a name is used for Japanese.

Anyway, with the fix, there is no reason to prefer Unicode-based
local character sets, which is not widely used today, than existing
local character sets already used world wide.

						Masataka Ohta

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 20 Mar 2002 12:29:10 -0600

dropping a few mailing lists off the CC:

> Unicode is usable in some local context.

Agreed. Note that "some" is changing to "many" as time goes on.
> However, the context information must be supplied out of band.

Agreed.
> information, regardless of whether you call it "charset" or

Do not agree.
for most values of what we currently have registered as "charset",  it is 
not sufficient to identify the context.

Therefore, depending on "charset" to identify context is not only useless, 
but actively harmful.
My opinion, which I stated in RFC 1766, and have found no reason to change.
The IETF rough consensus seemed to agree at the time.

                 Harald

---

From: Sampo Syreeni <[email-masked]>
Date: Wed, 20 Mar 2002 21:16:07 +0200 (EET)

>Unicode is not usable in international context.

Come, now. My understanding has been that that's precisely the context
where Unicode is supposed to be used. Why have Unicode in the first place
if not for multilingual text?

>There is no unicode implementaion work in international context.

How's that? I would tend to see current browser and office suite work as
such, at the very least.

>Unicode is usable in some local context.

If I understand correctly, you're opposed to Unification and the external
protocols for language tagging it necessitates if we insist on absolute
typographical correctness of text with unified ideographs. HTML and
XML/XHTML provide precisely such tagging, as do Unicode language tag
characters in the context of protocols with no external language
indication facilities. Also I seem to remember that East Asian Unicode
text is legible even when printed in a font not designed for the
particular "local context", as you put it. Where's the problem?

>There is some unicode implementaion work in local contexts.

Well, considering that UTF-8 is the encoding of choice for some past, much
present and all future IETF and W3C work, and that Microsoft's products
seem to be heading for UTF-16, I'd say that is a colossal understatement.

>However, the context information must be supplied out of band.

Not must, but rather must provided that the text is meant for human
consumption *and* exact typography of variant characters is a requirement.
Unicode was never meant to solve the latter part -- it does not encode
font information, for instance.

>And, the out of band information is equivalent to "charset" information,
>regardless of whether you call it "charset" or not.

Absolutely not. Unicode's characters are perfectly well defined even if
they are not correctly printed. What we're indicating here are differences
between languages and preferred renderings of a given piece of Unicode
text.

You should think about this in the context of rendering to speech, perhaps
that will help you see the fine distinctions involved. After all, one
cannot even *attempt* such renderings from text written in pure Latin-1
without external language indication. Why should graphic rendering be any
different?

>Unicode-based local character set to use.

No. The fix is to indicate the language the document is in, or perhaps
encode font information explicitly. Provided such meticulous attention to
the appearance of the text is warranted in a given application, anyway.

>See, for example, RFC1815.

Yes, you seem to have objected to Unicode, before. The trouble is, not a
whole lot of people agree with RFC1815. For a reason, I daresay.

>As for IDN, it can't just say "use charset of utf-7" or "use charset of
>utf-8".

Of course you can. UTF-8 and UTF-7 are bona fide character sets -- the
characters they define are unique and well-defined. You're confusing
renderings of characters with characters themselves, a common mistake with
Unicode. (But of course you already know that.)

>Anyway, with the fix, there is no reason to prefer Unicode-based local
>character sets, which is not widely used today, than existing local
>character sets already used world wide.

Of course there is -- my local character set cannot represent Arabic,
Japanese, English and Finnish, with correct punctuation and other
typographic pedantries, in the same document. Hell, it cannot do that even
in separate documents, unless I use Unicode. The latter on the other hand
works perfectly, with no loss of information.

Your line of thinking is what has lead to e.g. i-mode employing JIS or
Latin-1, and making the current incarnation of that technology useless to
Central European, Chinese, African, Indian, and a probably a whole lot of
other, user communities. Just think about what something like this would
do in an IDNA context, and you'll understand why Unicode is a Good Idea.

Besides, if you look at a Chinese user typing in the name of a Japanese
site, I would say unification makes the procedure considerably more
forgiving. Don't you?

I'd say local variants will full support are Bad, a unified coding with
local profiles is Good.

Sampo Syreeni, aka decoy - mailto:[email-masked], tel:+358-50-5756111
student/math+cs/helsinki university, http://www.iki.fi/~decoy/front
openpgp: 050985C2/025E D175 ABE5 027C 9494 EEB0 E090 8BA9 0509 85C2

---

From: Masataka Ohta <[email-masked]>
Date: Thu, 21 Mar 2002 06:37:15 +0859 ()

Harald;

> Agreed. Note that "some" is changing to "many" as time goes on.

Irreleant, because some contexts are not compatible.

The point here is that there can be no universal context.

> > However, the context information must be supplied out of band.

Then, how can you provide the information with IDN?

> not sufficient to identify the context.

That is a problem of current and past charset reviewers including you.

> Therefore, depending on "charset" to identify context is not only useless, 

Agreed. ISO 2022 escape sequence is the way to go.

> My opinion, which I stated in RFC 1766, and have found no reason to change.

It was already denied by real world examples.

						Masataka Ohta

---

From: "Adam M. Costello" <[email-masked]>
Date: Thu, 21 Mar 2002 02:54:57 +0000

> Unicode is not usable in international context.

I think you mean that Unicode doesn't work very well for displaying
free-form text, unless you have more meta-information about the text,
and you may be right.

But for text used as identifiers, Unicode is extremely useful, for
testing whether two identifiers are the same.

Suppose there are 100 charsets in use around the world.  Many of them
have characters in common, so two identifiers that use different
charsets might still contain the same characters in the same order.
To test for that, do we write 10,000 comparison algorithms to handle
all possible combinations?  Of course not.  We write 100 transcoding
algorithms to convert each charset to Unicode, plus one normalization
algorithm for Unicode.

AMC

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 20 Mar 2002 19:18:14 -0800

> Unicode is not usable in international context.
...

It would not be worth replying to these threadworn and repeated
assertions by Mr. Ohta, except that some members of this list may not
be that familiar with Unicode. Clearly Unicode is being used
successfully in a huge variety of products in international contexts.

For more information on the CJK repertoire (which is the part Mr. Ohta
objects to), see http://www.unicode.org/unicode/faq/han_cjk.html.

Mark
—————

Γνῶθι σαυτόν — Θαλῆς
[For transliteration, see http://oss.software.ibm.com/cgi-bin/icu/tr]

http://www.macchiato.com

----- Original Message -----
From: "Masataka Ohta" <[email-masked]>
To: "Robert Elz" <[email-masked]>
Cc: <[email-masked]>; <[email-masked]>; <[email-masked]>;
<[email-masked]>
Sent: Wednesday, March 20, 2002 07:58
Subject: [idn] Re: I don't want to be facing 8-bit bugs in 2013

---

From: Masataka Ohta <[email-masked]>
Date: Thu, 21 Mar 2002 16:47:58 +0859 ()

Mark;

As my mailer says;

	[Charset utf-8 unsupported, skipping...]

you should learn that Unicode is not usable in international context
of IETF.

> successfully in a huge variety of products in international contexts.

That is not so totally untrue. There do exist some Unicode products
valuable in international contexts, because they are useful in some (or
many) local contexts.

However, as Harald and I agreed, such products need local
context information OOB to work properly.

Then, the problem is that some local contexts are mutually exclusive.

There is no instance of Unicode-based-local-character-set
internationally useful.

Some product using unicode may be usable in multiple local context,
if the context information is given OOB, and the product is valuable
in international market.

However, unicode, still, is not usable in international context unless
OOB information, which, thanks to poor charset reviewing, essentaily is
yet another charset designaiton, is attached.

						Masataka Ohta

---

From: Alan Barrett <[email-masked]>
Date: Thu, 21 Mar 2002 14:33:37 +0200

On Thu, 21 Mar 2002, Masataka Ohta wrote:
> Unicode is not usable in international context. [...] Unicode is
> usable in some local context. [...] However, the context information
> must be supplied out of band.

Let me see if I can understand this argument about Unicode and local
context.  I am an English speaker who can't tell the difference between
the Chinese character that appears as the second character of the
Chinese word for the city that I call "Beijing", and the Japanese
character that appears as the second character of the Japanese word
for the city that I call "Tokyo".  I believe that (as used in the city
names) both characters mean something like the English word "capital".

Say there's a Chinese character that looks (to uneducated western eyes)
like a box with three legs and a hat, and a Japanese character that
looks (to uneducated western eyes) like a box with three legs and a hat.
Say the Chinese character looks slightly different from the Japanese
character, but a Chinese person can easily recognise the Japanese
character and understand its meaning in context, and a Japanese person
can easily recognise the Chinese character and understand its meaning in
context.

As far as I understand, Unicode would say that these are not two
different characters, but just different display forms of the same
unified character (or whatever the correct technical terms are).
Display software would have to have out of band knowledge to help it
choose between the Chinese and Japanese display forms.

As far as I understand, absence of out of band knowledge could lead to
the hypothetical Unicode character <CJK character that looks a bit like
a box with three legs and a hat> being displayed as if it were <Chinese
character that looks a bit like a box with three legs and a hat>, even
if the author's intent was to display <Japanese character that looks a
bit like a box with three legs and a hat>.

As far as I understand, Masataka Ohta considers this to be a fatal flaw
in Unicode.  I hope he will correct me if I have misunderstood his
objection.

I don't know enough to tell whether the difference between corresponding
Chinese and Japanese characters is analogous to a font difference or
a spelling difference, but the "ignorant westerner can't tell the
difference" test biases me towards the "font difference" side.  If they
are analogous to spelling differences, then I would say that unifying
the different characters was probably an error in Unicode, but that IDN
should not try to undo that unification.  Either way, I think that IDN
should document the potential problem but not try to fix it.

(In contrast, I think I have learned enough by following the past <n>
months of discussion to tell that the differences between Traditional
and Simplified Chinese are analogous to spelling differences, and so IDN
should not try to unify them.)

--apb (Alan Barrett)

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 21 Mar 2002 09:51:56 -0600

> context information OOB to work properly.

apologies for indicating too strong an agreement....
we agree that for many functions that users would like software to perform 
(sorting being an obvious example), context information is needed to work 
properly.
> There is no instance of Unicode-based-local-character-set

I assert that for many functions (such as non-high-quality text display) in 
many locales, having information in Unicode-based character sets are useful 
without OOB information. Thus, Unicode is internationally useful - but we 
are deluding ourselves if we claim that it solves ALL problems.

> yet another charset designaiton, is attached.

We disagree on this point.
I assert that the fact that charset does not give the required information 
is by DESIGN of the charset mechanism, not by "poor review".
As a citizen of a continent where > 10 nations use the same character set 
(ISO 8859-1), but where locale requirements vary across the continent, I 
believe that separation of those concerns was a correct engineering 
decision.

Just to make it clearer where Ohta and I agree, and where we disagree.....

                           Harald

---

From: Robert Elz <[email-masked]>
Date: Fri, 22 Mar 2002 00:39:56 +0700

Date:        Thu, 21 Mar 2002 00:57:18 +0859 ()
    From:        Masataka Ohta <[email-masked]>
    Message-ID:  <[email-masked]>

Otha-san

  | Anyway, with the fix, there is no reason to prefer Unicode-based
  | local character sets, which is not widely used today, than existing
  | local character sets already used world wide.

Let's assume that local char sets, and an explicit indication of which
to use is adequate for this purpose (as Harald has said, and I agree,
that's not sufficient for all purposes, but for stuff like domain names,
file names, etc, I suspect it is).

Then, let's take all the MIME names of those charsets, and number them,
0, 1, 2, 3, ... (as many as it takes), that's a 1::1 mapping, after all
the mime charset names are in ascii, we wouldn't want to keep that.

Then, rather than labelling whole names with a charset, we'll label
every individual character, so if, by some strange chance, ascii (or 8859-1)
happened to be allocated number 0, then 'A' would be 0-65.
This way we can mix and match names with characters from any random
character set (it may be a little less efficient than one label per name
but that's OK, assume we'll pay that price for the flexibility).

No, we'll just list all the possible characters in all the char sets,
with their char set numbers attached, in one big table.

What we have at that point is (essentially) 10646 - unicode.   Just up
above you said this works (throwing in some redundant labelling cannot
cause it to stop working, nor can altering the labels from ascii to binary 
numerals).

Of course, a large fraction of the char sets that we just listed in a big
table contain ascii as a subset,so we end up with dozens of versions of
A (it's in 8859-1 8859-2 ... tis-620 (Thai) ...).  All of those A's are
in all respects identical, keeping them will do no more than cause
interoperability problems.   So let's go through and squash all the duplicates.
and then renumber everything to be a bit more compact (the renumbering is
a 1::1 operation that alters nothing important).

Having done that, we have exactly 10646 (or can have, if we picked the
right character sets for our initial big list of everything, gave them
the appropriate char set numbers, and compressed the number spaces in
just the right way).

Again, you said above, this works.

The only place in all of this where there's any possibility of a problem is
in the "squash all the duplicates" - if some characters that are duplicates 
aren't removed, or some that aren't are treated as if they are.  If this
happened, then there'd be a bug in the actual character tables (too many, or
too few) - but this doesn't alter the principle of the thing.

If there is a bug like this (which I am not able to judge) then someone
should get it fixed.

Whether or not there is a bug, the unicode/10646 approach is clearly the most
flexible way, and is perfectly adequate for, labelling things using
whatever characters anyone wants to use - internationally or locally.

There is simply no way to rationally claim that a local char set, plus label,
is adequate and unicode is not.

kre

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 21 Mar 2002 09:51:56 -0600

Harald