Subject: Re: [idn] WG last call summary

From: Randy Bush <[email-masked]>
Date: Fri, 15 Mar 2002 13:07:15 -0800

> It's possible to eliminate the interoperability failures by upgrading
> _everything_, or by ugprading _nothing_. The problem is the transition.

so, it sounds like the problem is not the proposed standard in question,
but the lack of a transition plan without holes.  shades of ipv6!

so it sounds as if the wg is missing a work item.

randy

ps: i find your arrogant mail robots sufficiently annoying and abusive
    of my time that i will no longer be corresponding with you.  oddly,
    we all think we're more important than everybody else.  of course,
    none of us are.

---

From: Dave Crocker <[email-masked]>
Date: Fri, 15 Mar 2002 13:59:55 -0800

>so it sounds as if the wg is missing a work item.

Well, no, not really.

The example given had nothing to do with Internet protocols, but rather 
with how to transition the software on a computer system to support a 
richer set of characters.  That is, of course, an interesting problem, but 
it is not part of IETF scope.

The transition issues for IDNA that actually pertain to Internet protocol 
behavior have been described repeatedly and are identical with the kind of 
changes that were required to adopt MIME.

d/
----------
Dave Crocker <mailto:[email-masked]>
TribalWise, Inc. <http://www.tribalwise.com>
tel +1.408.246.8253; fax +1.408.850.1850

---

From: "Eric A. Hall" <[email-masked]>
Date: Fri, 15 Mar 2002 18:46:05 -0600

Dave Crocker wrote:

> The transition issues for IDNA that actually pertain to Internet
> protocol behavior have been described repeatedly and are identical
> with the kind of changes that were required to adopt MIME.

I can see why you'd want to say that, but it's a poor analogy.

MIME imposed structure on unstructured data by using a subset of values,
and specifically limiting itself to particular technologies. I mean, even
RFC2047 only allowed some particular kinds of data (mostly unstructured)
to be extended, and marked structured data as off-limits. Conversely, IDNA
modifies structured data-elements, and applies to all technologies.
Resistance is futile, you will be transliterated.

A much better analogy for IDNs in general is IPv6, where every
participating protocol, data-element and application that uses the
previous data-format has to be modified if it wants to use the new
data-format. But since IDNA seeks to avoid these costs, in the end, the
protocols, data-elements and applications don't actually obtain any real
benefits from internationalization; it's all just a facade, taped onto the
surface of the Internet, sometimes in harmful ways given the artifacts of
mandatory transliteration. In this regard, the best analogy would be what
IPv6 *DIDN'T* do: pass IPv6 inside of IPv4 options and call the work
finished. That's pretty much what IDNA does.

As for breakage, it will undoubtedly occur, including in those places that
Dan mentions, but it will also happen inside protocols and namespaces.
Message-IDs will absolutely get corrupted and reinserted into the
messaging system, Kerberos and NetBIOS will get broken and then get panic
fixes, etc.

> richer set of characters.  That is, of course, an interesting problem,
> but it is not part of IETF scope.

Dan's argument is actually the easiest to deal with, since it can be
addressed with the judicious and cautious use of "MAY" along with the
appropriate warnings.

Extending all known data-structures which happen to make use of DNS domain
names as key elements is the larger problem, and it is certainly within
the scope of the IETF.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: "Adam M. Costello" <[email-masked]>
Date: Sat, 16 Mar 2002 01:51:47 +0000

Once again I have removed [email-masked], [email-masked], and [email-masked]
from the Cc list.

"Eric A. Hall" <[email-masked]> wrote:

> IDNA modifies structured data-elements

IDNA is built around the core principle of *not* modifying existing
structured data elements.  IDNA requires that all domain names in
existing structured data elements continue to be ASCII-only.  Non-ASCII
domain names may appear only in new structured data elements and in
unstructured text and displays.

AMC

---

From: "D. J. Bernstein" <[email-masked]>
Date: 16 Mar 2002 02:21:34 -0000

> but the lack of a transition plan without holes.  shades of ipv6!

No.

IDNA is being proposed as something that new sites can use _right now_.
In contrast, nobody is telling sites that they can rely on IPv6 for
complete Internet service today.

More importantly, IDNA is a set of changes without a long-term strategy.
In contrast, specifying the desired final architecture is an explicit
part of the IPv6 project; there are many documents along these lines,
separate from the transition documents.

There certainly isn't IDN WG consensus on IDNA as the long-term IDN
solution. In fact, many of us suspect that there's consensus on the
following statement, which implies that IDNA is merely temporary:

   The long-term IDN solution will encode Unicode characters as UTF-8 on
   the wire.

The IDN WG chairs have repeatedly refused to run a straw poll on this
statement, never mind actually settling all the other important details
of what we're really trying to achieve!

If we were faced with a document proposing IDNA as a long-term solution,
I'd agree that the interoperability issues were only with the transition
plan. It _is_ possible to make a world of multiple character encodings
work correctly, even though it's remarkably bad software engineering.

Of course, the lack of interoperability is only one of IDNA's problems.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

---

From: Keith Moore <[email-masked]>
Date: Fri, 15 Mar 2002 22:00:37 -0500

> to be extended, and marked structured data as off-limits. 

more precisely, RFC 2047 was intended only for human-readable text, 
because at the time we hadn't figured out how to solve the various 
problems associated with having multiple representations of 
machine-readable text.

today we have a much better idea for how to do that.

I agree with Dave that IDNA is very similar to things we did with MIME
in both the header and the message body, and I think the experience
with MIME will be a good predictor of experience with IDNA.  

Keith

---

From: "Eric A. Hall" <[email-masked]>
Date: Fri, 15 Mar 2002 21:54:27 -0600

> today we have a much better idea for how to do that.

At least IDNA only has one conversion form. Still, the point is valid that
something as precise and scope-limited as 2047 only worked well within the
scope it was intended for, namely presentation of unstructured header
fields within a specific service. I mean, I don't know of any email
clients that perform 2047 encoding on search string input, do you?

> in both the header and the message body

The notion of transliterate-for-display-so-we-don't-upgrade is similar,
but the scopes are completely different. If you are using a 2047-capable
email client, copy-n-paste between From: and To: is practically guaranteed
to work out. But all IDN approaches introduce copy-n-paste between web
browsers, email clients, WHOIS clients, ad nauseum. That also introduces
all kinds of other issues, like, "is that i18n URL I copied actually a
legal URL? [not today, no]" That's why this is closer to IPv6 than MIME.

Just to be clear, my position on this subject remains that IDNA is a
necessary and useful transfer encoding for legacy systems that need access
to i18n domain name data, but that the mandatory transliteration of all
domain names as any kind of solution to the problem is just reckless.
These problems won't go away until they are accepted and designed for.

---

From: Keith Moore <[email-masked]>
Date: Fri, 15 Mar 2002 23:50:03 -0500

> clients that perform 2047 encoding on search string input, do you?

No, but that would be a really lousy way to implement searching for 
non-ASCII text in message headers.   If I were implementing such 
a search function, I'd build an index of decoded and canonicalized 
header text rather than trying to match the query string against 
the 2047 encoding.
  
> browsers, email clients, WHOIS clients, ad nauseum. 

Copy and paste is also an issue for 2047.  The only difference is that
since 2047 doesn't apply to addresses, but only to human-readable text,
a transliteration bug is less likely to cause your mail to go to the 
wrong address.  But such bugs do sometimes cause names or subjects to 
be garbled.

But there are copy-and-paste issues even for ASCII URLs, due to %-encoding
(if you're copying from a poorly "decoded" form of the URL) and line 
wrapping.  Perhaps in hindsight URLs could have been designed differently.
But with so many people writing so many implementations of things that
handle URLs, subtleties tend to get overlooked by large numbers of 
implementors.  That's also the worst problem with doing IDNs, but it would
exist no matter what scheme were chosen.  

Keith

---

From: Dave Crocker <[email-masked]>
Date: Sat, 16 Mar 2002 05:54:58 -0800

>non-ASCII text in message headers.   If I were implementing such

Indeed.

The reference to searching nicely underscores that this entire thread 
pertains to local implementation issues, rather than having anything at all 
to do with the DNS.

The DNS does not have searching.

And with respect to the comparison with MIME, IDNA lays a specialized 
character format onto an existing string environment, thereby requiring 
change only for participants that which to enjoy the new feature.  It 
requires no change to the infrastructure and has no impact on 
non-participating DNS entities.  That is the architectural basis for 
comparing IDNA to MIME.

In other words, folks, this entire thread is caused by misunderstandings of 
the difference between protocol and implementation.

d/

----------

---

From: Keith Moore <[email-masked]>
Date: Sat, 16 Mar 2002 10:51:20 -0500

>    The long-term IDN solution will encode Unicode characters as UTF-8 on

I doubt we'd reach consensus on that either, since many of us suspect 
that the following statement is closer to the truth:

	The on-the-wire encoding of IDNs is irrelevant; what matters is the 
 	behavior experienced by users.

Keith

---

From: "Eric A. Hall" <[email-masked]>
Date: Sat, 16 Mar 2002 11:01:53 -0600

>     behavior experienced by users.

Obviously. It's a question of getting there, however.

The final, ultimate argument in this logic-chain is that native
representation of the data in the protocol message is the ultimate design
solution, since it means less implementation work, fewer errors introduced
by wayward codecs, highest reusability by other services, etc. Managed
facades are not long-term solutions to anything, and in fact, tend to
introduce as many problems as they try to fix.

Obviously, direct encapsulation is harder with some services than others.
That doesn't mean that direct encapsulation should not be the ultimate
design target, it only means that some protocols will have to use a patch
(like IDNA) until they can either be extended or replaced.

---

From: "Eric A. Hall" <[email-masked]>
Date: Sat, 16 Mar 2002 11:34:35 -0600

Dave Crocker wrote:

> pertains to local implementation issues,

"Searching" in the context of RFC2047 doesn't just apply to local issues.
Consider that IMAP SEARCH and NNTP XPAT both provide protocol mechanisms
to search server-based data. However, I don't know of any clients that
convert "ë" to "=EB" for those searches.

That's partly a local issue, but it is also a protocol issue, and in the
end it is also a service-wide issue. At some point, somebody has to say
"okay we either mandate back-end conversion to allow for unencoded
searches or we mandate encoded search behavior." Globally, it is a
user-interaction issue, where the facade failed, due to the lack of direct
integration.

This same kind of issue will apply equally to searches for an "ë" that is
embedded into a domain name component of an email address, and which has
been ACE-encoded. There are multiple secondary considerations here as
well. Searching for a single "ë" that has been encoded with ACE is not as
simple as searching for an "=EB" sequence with 2047, for example. I dare
say that substring searches are likely to prove impossible with
ACE-encoded strings.

Anyway, clearly these aren't just local implementation issues.

---

From: Dave Crocker <[email-masked]>
Date: Sat, 16 Mar 2002 09:38:22 -0800

>Obviously. It's a question of getting there, however.

Not necessarily.

UTF-8 is an encoding.  IDNA uses an encoding.  Neither is a "native" 
representation of the semantics.

Given the tiny length of the strings involved, the relative differences 
between UTF-8 and the approach taken in IDNA is essentially irrelevant.

Note that we are approaching the 10th anniversary of MIME, which involves 
much larger strings, and there is still no strong pressure for system-wide 
use of "pure" binary encoding.  It exists and is used, but it is not the 
dominant encoding method, nor are users particularly aware of when it is 
being used.

d/

ps.  after this many years, it has become amusing to watch anyone make a 
firm prediction about the world's technical future.

----------

---

From: "Eric A. Hall" <[email-masked]>
Date: Sat, 16 Mar 2002 11:46:33 -0600

Dave Crocker wrote:

> Neither of them have anything to do with the DNS.

IDNA isn't just about DNS.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Sat, 16 Mar 2002 11:56:01 -0600

>    The long-term IDN solution will encode Unicode characters as UTF-8 on

If this was reformulated as

  The long-term IDN solution will not use DNS

it would have my vote.

---

From: "Eric A. Hall" <[email-masked]>
Date: Sat, 16 Mar 2002 11:45:50 -0600

> representation of the semantics.

If all of the inputs and outputs are using UTF-8, then UTF-8 on-the-wire
is about as close as you can get to a direct representation.

---

From: Dave Crocker <[email-masked]>
Date: Sat, 16 Mar 2002 09:43:43 -0800

At 11:34 AM 3/16/2002 -0600, Eric A. Hall wrote:
>Consider that IMAP SEARCH and NNTP XPAT

d/

----------

---

From: Keith Moore <[email-masked]>
Date: Sat, 16 Mar 2002 16:53:00 -0500

> introduce as many problems as they try to fix.

Every single one of the above arguments is highly dubious.

First, it's naive to assume that UTF-8 will be the native 
representation on everybody's platform - it has a number of nice
properties but it's far from clear that everybody is going to
drop what they're doing now to support it.  Second, the portion of
IDNA that does ASCII encoding is such a trivial bit of code that 
the number of failures introduced by that code will pale in 
comparison to those introduced by the other code needed to handle 
10646 (normalization, etc) which would be needed no matter what 
encoding were used.   Third, your statement about "managed facades"
is a gross over-generalization.  Numerous examples demonstrate 
that transition issues are often paramount in determining whether 
a new technology can succeed.

Simplicity is often a virtue, but IDN is inherently complex - it
reflects the tremendous variety in the world's languages and 
writing systems.  And blind faith in some vague notion of 
cleanliness is a poor substitute for engineering analysis.

Having said that, I'm all for having an alternate method to do
UTF-8 queries on-the-wire, provided it produces consistent
results with the IDNA queries, with similar efficiency and
reliability.  But the need to allow incremental upgrade of 
legacy application components strongly compels IDNA, and the 
incremental benefit of a native UTF-8 query interface beyond 
that of IDNA does not appear to justify the additional complexity.  
Nor does it justify delaying approval of IDNA.  Most of the 
problems that are anticipated with IDNA - including the various
kinds of transcription problems - will occur no matter what 
encoding is used for on-the-wire queries.

Keith

---

From: "D. J. Bernstein" <[email-masked]>
Date: 17 Mar 2002 00:09:25 -0000

Users suffer every day from the current character-encoding mess. Anyone
who ventures beyond ASCII is faced with endless trouble.

There's no theoretical obstacle to making multiple character encodings
work---make sure that there is, implicitly or explicitly, a perfectly
clear character encoding for every byte stream; make sure that every
copy and comparison includes all necessary conversions---but in practice
this is a disaster. It's too much work to specify the encodings. It's
far too much work to program all the necessary conversions.

Look at IDNA. Yet another character encoding. An amazingly unclear
specification of which byte streams are supposed to use that encoding.
Massive redeployment of, at a minimum, every web browser in the world.
And that's just for domain names! Is every worldwide identifier---for
example, mailbox names---supposed to have its own massive upgrade?

UTF-8 offers a way out of this mess. We do _one_ upgrade to make sure
that UTF-8 works everywhere. For example, RFC 2277, IETF Policy on
Character Sets and Languages, requires UTF-8 support in all protocols.
Then we convert all stored data to UTF-8. Then, finally, we can drop
support for the other character encodings.

Do we want programmers in twenty years to be faced with the same mess
that we have today? Or do we want them focusing on positive features for
the users?

>  	behavior experienced by users.

Everything is judged by the user experience, yes, but you are clearly
incorrect in saying that the encoding is irrelevant.

---

From: "Eric A. Hall" <[email-masked]>
Date: Sat, 16 Mar 2002 19:42:13 -0600

Keith Moore wrote:

> representation on everybody's platform

Clarification. I did not say anything about UTF-8 on platforms, but
instead cited native representation in protocol messages. However,
if UTF-8 is the encoding of choice for a particular protocol's data or
message formats, then we know that UTF-8 is also going to be incorporated
into the necessary supporting functions at the participating end-points.
This doesn't mean that the whole box has to use UTF-8. Frankly, I don't
even think that's relevant. Instead, it is a question of whether or not
the related components (searching, as in the previous example) will be
likely to deal with UTF-8, rather than having to selectively graft an
exraneous encoding into select portions of that service in order to
provide simple functionality (as with 2047 and searching, again).

But this is also entirely irrelevant. By your argument that the transfer
encoding is irrelevant, I would like to hear your arguments as to how,
say, using EBCDIC to pass ASCII data around could possibly be seen as
reasonable design. Of course the native encodings are always best. The
fact that most of the apps are heading towards UTF-8 should tell us that
we should be designing for a long-term support infrastructure that
provides the data in the format it is going to be used in. Furthermore,
whenever the remaining services get upgraded or replaced, they should be
able to use something a little better than the best technology that 1968
money can buy.

> Second, the portion of IDNA that does ASCII encoding is such a trivial
> bit of code that the number of failures introduced by that code will
> pale in comparison to those introduced by the other code needed to
> handle 10646 (normalization, etc) which would be needed no matter what
> encoding were used.

Getting new problems in addition to shared problems is hardly an argument
in your favor. You've already conceded that 2047 has some problems with
transliteration goofiness, and that restricting it to unstructured data
limits the real damage that is caused. Are we to believe that extending
structured data with mandatory transliteration will not cause the problems
you thankfully avoided?

> Numerous examples demonstrate that transition issues are often
> paramount in determining whether a new technology can succeed.

I agree that transitional services are important. I also think that the
evidence shows that end-station layering works well when existing formats
are used as carriers for *subsets*, and when it is targeted to a specific
usage. That isn't what's being done here, though. Instead, well-known and
commmonly-used data-types will get *extended* into broader and
incompatible forms by default, and it will happen purposefully and
accidentally. This is not transitional probing, it is knowing that stuff
will break and doing it anyway.

Cripes, why do we have to do it all in a big-bang? Can't we start with the
transfer encoding (no required upgrades for anything), incrementally add
transliteration where we know it will be safe and robust (some upgrades),
and then add UTF-8 for those newer services that can make use of it (some
more upgrades)? What is the problem with this?

> cleanliness is a poor substitute for engineering analysis.

That's almost a fair shot. I do put a bunch of faith into transparent
data-types and structures. Dunno about "blind". ASCII is always best when
it's encoded as ASCII, after all.

> that of IDNA does not appear to justify the additional complexity.

The complexity required for a direct UTF-8 name-resolution service in
conjunction with simple passthru-everywhere is minor in comparison to the
complexity of transliterate-everywhere.

---

From: Keith Moore <[email-masked]>
Date: Sat, 16 Mar 2002 22:20:56 -0500

> provide simple functionality (as with 2047 and searching, again).

again, the effort to get the ASCII encoding right is trivial compared
to the effort required to do string comparisons in a way that's effective 
for searches. 

> reasonable design. 

it would be quite reasonable for applications that already had a 
deeply-wired assumption that character strings were in EBCDIC.
Which is why on BITNET and related networks, the applications 
generally exchanged data in EBCDIC even between ASCII-native hosts.

> Of course the native encodings are always best. 

utter hogwash.  first, there is no single "native" encoding of UCS, 
second; there's no encoding of IDNs that is native to the vast majority 
of deployed applications.  

> The fact that most of the apps are heading towards UTF-8 should tell us 
> that we should be designing for a long-term support infrastructure that
> provides the data in the format it is going to be used in. 

again, that's hogwash, because it completely ignores issues associated 
with transitioning existing apps and/or with maintaining multiple 
interfaces to DNS.

> Furthermore,

that's like saying that now that we have automobiles, we should completely 
rearrange all of those cities that were designed on a walking scale, that
we should change the width of roads so that they're more appropriate for
motor-driven vehicles than horse-driven vehicles, etc.  the fact is that 
once conventions get established, it's often more effecient overall to 
stick with a convention that is sub-optimal than it is to rebuild everything 
to be optimized for current conditions - particularly when those conditions
will probably change again anyway.  (e.g. it's much easier to build mass 
transit for cities designed on a walking scale...commuting by automobile 
sucks!)

if and when the rest of the protocol uses UTF-8, then the worst that 
happens is that there's an extra layer of encoding that happens before 
a DNS query is sent.  it's isolated to a single routine.  it's not 
going to hurt enough to matter.   and having the client do encoding
prior to lookup is less complex than the extra cruft that's required
to take advantage of having two kinds of DNS queries without introducing
a significant new source of failures or delays.

like I said, blind faith in cleanliness is no substitute for analysis.

> in your favor. 

no, but you're ignoring the set of new problems that comes with either
a) providing multiple DNS query interfaces or 
b) failing to provide an ascii-compatible encoding of IDNs that can be
   tolerating by existing apps
but sure enough, if you close your eyes, you can pretend that the room 
is empty.

> will break and doing it anyway.

the alternative is to say that all apps have to be rewritten to take
advantage of IDNs, and the likely result is to fragment existing apps
into >> 2 non-interoperable enclaves.  the IDNA path causes less harm.

> more upgrades)? What is the problem with this?

it completely ignores the forces that drive deployment of upgrades. 

> complexity of transliterate-everywhere.

only if you have the luxury of starting from scratch, before any
applications are written.`

Keith

---

From: "Eric A. Hall" <[email-masked]>
Date: Sat, 16 Mar 2002 23:17:20 -0600

Keith Moore wrote:

> > I would like to hear your arguments as to how, say, using EBCDIC
> > to pass ASCII data around could possibly be seen as reasonable
> > design.
> 

Mapped to the current topic, you are saying that ~if the current apps were
already deeply-wired for IDNA, then it would be reasonable to use. Of
course, I would agree with that, if it were anything resembling the
current situation. Too bad nothing in the known universe makes any use
whatsoever of IDNA or (more importantly) its conversion routines.

> utter hogwash.  first, there is no single "native" encoding of UCS,

You already know that UTF-8 is the implicit default encoding for it here.

> of deployed applications.

Keith, there's no IDNA encodings of IDNs that are native to *any*
applications, either current or scheduled.

[big snip]

> a DNS query is sent.

No, the "worst" that happens is that well-known and widely-used data-types
get extended by a third-party process, and then get reused.

>  and having the client do encoding

There is definitely some transitional pain in getting to direct UTF-8, but
it is non-fatal here, there, and in-between. It doesn't rewrite data, it
doesn't require a forklift upgrade of the entire Internet for rendering
purposes, and the entire Industry would be better off when it was done.
None of that can be said for IDNA transliteration. Well, the pain part
still applies.

> you're ignoring the set of new problems that comes with either

Also necessary for IDNA sooner or later, mark my words.

>    tolerating by existing apps

To repeat myself for the record, I think that the IDNA transfer-encoding
portion is necessary to provide legacy applications with access to
resources in the IDN namespace. What I am specifically arguing against is
transliteration. It is just foolish to expect that well-known and
widely-used data-types can survive being extended beyond their spec and
still function as expected. I have even provided you with evidence proof
to the contrary.

Look, if a private venture were to come out pitching an untested add-on
that rewrote every domain name which appeared on screen, the greybeards
would trot out to make measured statements cautioning against mangling of
well-known datatypes. What's the difference here?

Oh, shit, we *ARE* the greybeards!

---

From: Dave Crocker <[email-masked]>
Date: Sat, 16 Mar 2002 20:08:24 -0800

>is about as close as you can get to a direct representation.

If one plus one were three, the result would be odd.

Non-sequitors are not overly helpful to technical discussions, Eric.

At 12:09 AM 3/17/2002 +0000, D. J. Bernstein wrote:
>work

Nice to see you admit that.

>Massive redeployment of, at a minimum, every web browser in the world.

Oh, rather more packages than that need changing.

However it's nice to note that for any of them to enter the world of 
enhanced DNS characters, they would need changing anyhow.

So IDNA does not affect this statistic, other than not making it worse.  By 
contrast, a UTF-8 based encoding will be expected to break most of the DNS 
clients that have not changed.  This is a very poor transition strategy.

>example, mailbox names---supposed to have its own massive upgrade?

Yes, Dan.  Dealing with an installed base is really a hassle.  However it 
beats the alternative of not dealing with that base.

>UTF-8 offers a way out of this mess.

Well, no.  In fact, it creates a much, much more serious transition 
requirement.  It requires a full-system switchover, rather than permitting 
incremental adoption.

At 12:52 AM 3/17/2002 +0000, D. J. Bernstein wrote:
>Internationalized domain names are a failure if non-ASCII glyphs don't
>appear on the screen. Have you completely lost sight of the goal here?

Transition plans that require everyone to switch over all at once ensure 
that failure, or have YOU lost sight of the goal here, Dan?

The issue is how to accomodate the installed base.  IDNA permits 
incremental adoption.  Anyone who opts in gets benefit.  However no one is 
required to opt in and those who do not opt in are not penalized with 
anything worse than ugly strings.

 From the standpoint of transition schemes for an installed base, this 
represents the ideal.

>The interoperability problems begin when IDNA gobbledygook is converted
>to the local character encoding for display. If the result is copied to
>another program---through pipes, copy-and-paste, whatever---then that
>program's lookups will fail.

Dan, you need to talk to the people responsible for inter-process 
communications within a single system.  That ain't the IETF.

d/

----------

---

From: Dave Crocker <[email-masked]>
Date: Sat, 16 Mar 2002 20:08:24 -0800

d/

----------

---

From: "Adam M. Costello" <[email-masked]>
Date: Sun, 17 Mar 2002 06:58:17 +0000

"Eric A. Hall" <[email-masked]> wrote:

> To repeat myself for the record, I think that the IDNA
> transfer-encoding portion is necessary to provide legacy applications
> with access to resources in the IDN namespace.  What I am specifically
> arguing against is transliteration.

I'm not sure exactly what you mean by transliteration.  In IDNA there
are two transformations that can be applied to labels: ToASCII, which
forces a label to ASCII (performing ACE encoding if necessary), and
ToUnicode, which undoes the ACE encoding if present.

Are you saying that ToASCII is good, and ToUnicode is bad?

I can imagine a world with ToASCII but without ToUnicode.  If a
non-ASCII name came to you via new protocols that support non-ASCII
names directly, then you'd see the human-friendly form.  But if the name
traversed an old protocol (at any point), you'd see the ACE.

I don't see how doing away with ToUnicode would solve any problem.  The
main danger is non-ASCII names getting accidentally fed to old software.
Even without ToUnicode, non-ASCII names would still be out there (being
carried around by the new protocols and new applications), and it would
still be possible for them to get pasted or piped into old programs.

AMC

---

From: Dave Crocker <[email-masked]>
Date: Sat, 16 Mar 2002 20:08:24 -0800

d/

----------

---

From: "Eric A. Hall" <[email-masked]>
Date: Sun, 17 Mar 2002 15:08:49 -0600

> Are you saying that ToASCII is good, and ToUnicode is bad?

I am saying that legacy applications need access to IDN namespaces, but
that modifying well-known and widely-used data-types in order to render
domain names in Unicode form is foolish. We have to separate domain names
from the data-types that also use them; they do not need to be cojoined.

This means that legacy applications, protocols and data-types which use
STD13 names must only be presented with the STD13 form of the IDNs. The
i18n form of those names must only be presented to the applications,
protocols and data-types which can make use of an i18n domain name.

Incorporating this distinction into the current concepts isn't all that
easy because of the cross-breeding of ideas and objectives in the docs.

What I would like to see is for the current IDNA spec to be made into a
codec definition with guidance on implementation. This means deleting (or
re-scoping) section "3. Requirements" and deleting section "6.4 Avoiding
exposing users to the raw ACE encoding", and adding a new section for
"Implementation considerations".

The new text should essentially state that domain names which are used by
applications, protocol messages and data-formats MUST be passed and
displayed in LDH form, except where the governing specification has
explicitly defined an IDN behavior for the affected domain name, and that
the use of ToUnicode is expressly prohibited if the governing
specification has not defined how and where that function will be
deployed. It should also be stated that the "governing specification" will
often be the local software specifications, such as man named.conf, man
ping, or whatever (these will govern domain names which are used as
connection identifiers, and which are not used for protocol messages or
standardized data-types).

> traversed an old protocol (at any point), you'd see the ACE.

That would be true for new protocol messages and/or data-types that had to
traverse the old namespace, yes. There would also be an assumption that
the new messages and/or data-types provided mechanisms for storing the
IDNs in some kind of raw form (eg UTF-8), and that a conversion point was
defined which said "do the conversion here".

> still be possible for them to get pasted or piped into old programs.

If the sanctity of the existing data-types are preserved, that won't break
anything (or as much, anyway), since only the new data-types will be able
to use native IDNs.

---

From: Dave Crocker <[email-masked]>
Date: Sun, 17 Mar 2002 13:57:00 -0800

>I am saying that legacy applications need access to IDN namespaces, but

IDN uses strings that are entirely compatible with the existing DNS 
infrastructure.

Hence, legacy apps will automatically get the access they need.  They need 
no change.

d/

----------

---

From: "Eric A. Hall" <[email-masked]>
Date: Sun, 17 Mar 2002 16:53:00 -0600

"Eric A. Hall" wrote:

> The new text should essentially state that ...

Section 6.2 of draft-hall-dm-idns-00.txt can be partially adapted for this
purpose. Excerpted and adjusted:

    *   Protocol transaction data. If a protocol supports UTF-8
        encoded  internationalized domain names as parameters to
        commands or responses, UTF-8 encoded IDNs SHOULD be
        used wherever they are allowed. If UTF-8 is not
        supported by the protocol, the ACE equivalents of those
        domain names MUST be used. Protocols that wish to
        provide UTF-8 encoded domain names as protocol data
        as extended functionality MUST provide negotiation
        services which clearly distinguish the IDN encoding to
        be used.

    *   Structured data-types. Structured data-types that
        incorporate domain names as key elements (such as URLs
        and email addresses) MUST be processed according to the
        rules which govern those data formats. Applications MUST
        NOT perform ToUnicode conversions against structured
        data-types unless this conversion has been explicitly
        prescribed by the governing specifications. In some
        situations, new data-type formats will be required in
        order for IDNs to be safely encapsulated.

    *   Unstructured data. Domain names that appear within
        application output as unstructured data (such as email
        message bodies or HTML documents that simply reference
        an IDN, but not as structured data), are subject to the
        restrictions imposed by the specifications which govern
        that data, and by the desires of the users who create
        that data. Generally speaking, it is RECOMMENDED that
        applications allow users to enter or view domain names
        in their preferred encoding formats wherever this is
        allowed, but that any conversions which are subsequently
        applied to that data should use UCS as the translation
        intermediary, so that the IDNs are properly converted
        along with the rest of the application data.

> If the sanctity of the existing data-types are preserved, that won't
> break anything (or as much, anyway), since only the new data-types
> will be able to use native IDNs.

clarification: "only the new data-types [or those data-types which have
been extended by the governing bodies] will be able to use native IDNs."

---

From: "Adam M. Costello" <[email-masked]>
Date: Mon, 18 Mar 2002 03:15:27 +0000

"Eric A. Hall" <[email-masked]> wrote:

> domain names which are used by applications, protocol messages and
> data-formats MUST be passed and displayed in LDH form, except where
> the governing specification has explicitly defined an IDN behavior for
> the affected domain name, and that the use of ToUnicode is expressly
> prohibited if the governing specification has not defined how and
> where that function will be deployed.

Let me make sure I'm understanding you correctly.  For a given domain
name slot (protocol element, structured data field, function argument,
etc), the governing specification says what you may write into that
slot, and tells how to interpret what you read out of that slot, but it
does not dictate what you may do with the name after you've read it.
That depends on what you're going to do with the name next.

For example, if an application reads a name from slot X and writes it to
slot Y, then it's the specification of slot Y that determines whether
ToUnicode can be performed along the way, not the specification of slot
X.  If the application reads a name from slot X and displays it, then
it's the specification of the application itself that determines whether
ToUnicode can be performed.

Do we agree on all that?

If so, then your paragraph quoted above is stipulating the same
requirement as rule 1 of IDNA: domain labels written into domain name
slots MUST be in ASCII form unless the specification of the slot
explicitly provides for non-ASCII labels.

> Structured data-types that incorporate domain names as key elements
> (such as URLs and email addresses) MUST be processed according to
> the rules which govern those data formats.  Applications MUST NOT
> perform ToUnicode conversions against structured data-types unless
> this conversion has been explicitly prescribed by the governing
> specifications.

This is another form of rule 1 of IDNA.

> Generally speaking, it is RECOMMENDED that applications allow users
> to enter or view domain names in their preferred encoding formats
> wherever this is allowed...

This looks an awful lot like rule 2 of IDNA.  SHOULD and RECOMMENDED are
equivalent according to RFC 2119.

From the IDNA draft:

    1) Whenever a domain name is put into a generic domain name slot, every
    label MUST contain only ASCII characters.

    2) ACE labels obtained from domain name slots SHOULD be hidden from
    users except when the use of the non-ASCII form would cause problems
    or when the ACE form is explicitly requested.  When requirements 1
    and 2 both apply, requirement 1 takes precedence.

Isn't this very close to what you're saying?

("Generic domain name slot" is defined earlier in the draft, and
includes any structured-data domain name whose specification does not
explicitly provide for IDNs.)

IDNA does not explicitly say something like "when using protocols, data
formats, and interfaces, you MUST follows the rules governing them", but
that goes without saying.

AMC

---

From: [email-masked]
Date: Sun, 17 Mar 2002 21:28:45 -0800

> generally exchanged data in EBCDIC even between ASCII-native hosts.

Damn you Keith.  I had *almost* managed to repress the joyful memories of
trying to debug what happened when a document managed to start off life
as ASCII, get translated to EBCDIC for Bitnet, to/from ASCII again for
an intermediate hop, back to EBCDIC again for another hop, and then back
to ASCII at the tail end of the journey.  And yes, that translate count
is correct, and that document was... umm... interesting. ;)

However, the day-to-day joys of Bitnet ASCII-EBCDIC translation and what
it did to uuencode was one of the *primary* motivating reasons why Base64
happened instead of uuencode.   Let's not lose sight of that debacle. ;)
-- 
Valdis Kletnieks
Computer Systems Senior Engineer
Virginia Tech

---

From: "Eric A. Hall" <[email-masked]>
Date: Sun, 17 Mar 2002 23:38:27 -0600

"Adam M. Costello" wrote:

> etc), the governing specification says

The fields that are used in a message have different behaviors than the
loosely-coupled data-types which appear in multiple messages or in the
message payload.

> what you may write into that slot,

yes for both

> and tells how to interpret what you read out of that slot,

yes for both

> but it does not dictate what you may do with the name after you've
> read it.

True for message fields, false for the loosely-coupled data-types which
are independent of a particular of any particular protocol message (email
addresses, Message-ID, URLs, etc). Also true for connection identifiers.

No.

---

From: "Adam M. Costello" <[email-masked]>
Date: Mon, 18 Mar 2002 06:13:47 +0000

"Eric A. Hall" <[email-masked]> wrote:

> > For a given domain name slot (protocol element, structured data
> > field, function argument, etc), the governing specification...does
> > not dictate what you may do with the name after you've read it.
>
> True for message fields, false for the loosely-coupled data-types
> which are independent of a particular of any particular protocol
> message (email addresses, Message-ID, URLs, etc).

I'm not getting your point at all.  Apparently you are concerned that
there exist scenarios where ToUnicode ought to be forbidden/discouraged,
but the IDNA draft allows/encourages it.  Maybe if you described a
specific example of such a scenario, I'd start to understand.

AMC

---

From: "Eric A. Hall" <[email-masked]>
Date: Mon, 18 Mar 2002 03:12:33 -0600

> specific example of such a scenario, I'd start to understand.

I don't know how to say this any different than I have been saying it. The
ToUnicode step changes well-known and widely-used data-types in such a way
that the data no longer conforms to the rules which govern that data.
These changes are harmful when the extended data-types are reused, whether
by copy-n-paste, program output, direct transcription, or whatever.

An example I have already given is Message-ID. Basic functionality will be
broken if the structure of the well-known and widely-used Message-ID
data-type as defined in STD11 is extended beyond the scope of the
governing spec. The extended, STD11-incompatible form will break on search
inputs that don't allow those values, it will break if the search input
accepts it and passes it to a remote system via an IMAP SEARCH or NNTP
XPAT operation, it will break if a user puts it into a news or http URL
which gets transliterated and then percent-hacked, and it will break if a
user types it into a web URL as a parameter to a server-based search
function. That's just search and fetch, nevermind other problems like
damage to threading that results from an extended Message-ID which is
manually added to See-Also or References, corrupted spam complaints, and
the dozens of other common uses for this well-known, widely-used,
STANDARDIZED data-type.

This does not mean that users with mail systems located in IDN zones
cannot generate Message-IDs from those domains, only that they have to
continue generating them with LDH domains (via ToASCII). In that scenario,
Message-ID would continue to function across all of the services which
utilize the format specified in STD11.

That is the way it MUST be until that data-type is either extended (after
months of rancorous debate, surely), or it is replaced with something
else, or the WG in charge of that standard choose to allow for
transliteration (THEM, NOT US). Note well: the subsequent rancorous
debates are not the problems of this WG anymore than extending the
data-types is our within our scope of responsibility. They are smart
enough to figure out what they can get away with on a case-by-case basis.

Chorus: Implicitly extending all such data-types currently in use on the
Internet (as the current draft does) is willfully causing breakage, and
goes well beyond the authority of this WG.

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 19 Mar 2002 02:07:06 +0000

"Eric A. Hall" <[email-masked]> wrote:

> The ToUnicode step changes well-known and widely-used data-types in
> such a way that the data no longer conforms to the rules which govern
> that data.

ToUnicode does not change any data type.  ToUnicode does not apply to
email address, or URIs, or message-IDs.  It applies to IDNs (including
traditional ASCII domain names).  In order to get any interaction
between a message-ID and ToUnicode, you first need to extract the
domain name from the message-ID.  Then when you apply ToUnicode, you
get an IDN, not an internationalized-message-ID.  If you want to have
a message-ID again, you need to insert the IDN back into a message-ID
structure, which invokes rule 1 and requires the IDN to be forced back
to ASCII (via ToASCII).

That's the story when machines are talking to machines.  When it's
machines talking to humans, it's a different story:

When a mail program displays a message header, it's not necessarily a
real RFC 822 message header on the screen.  The mail program doesn't
promise to speak RFC 822 to the human user.  What's on the screen is a
rendering.  The program might omit certain fields that are required, and
might include non-ASCII characters (converted from RFC 2047 encoding).
It might let me type non-ASCII characters into the Subject line of an
outgoing message.  When I instruct the program to send a message, the
program will take care to send an actual RFC 822 message, doing whatever
encoding is necessary.  I might try to cut & paste the message from the
screen into sendmail, but I wouldn't be very surprised if that didn't
work.  The program might let me type non-ASCII characters into a search
query, and it will find messages with matching headers or matching
bodies, even if they use quoted-printable and base64 encoding.  (Mutt
does all this correctly, by the way.)

> An example I have already given is Message-ID.  Basic functionality
> will be broken if the structure of the well-known and widely-used
> Message-ID data-type as defined in STD11 is extended beyond the scope
> of the governing spec.  The extended, STD11-incompatible form will
> break on search inputs that don't allow those values, it will break
> if the search input accepts it and passes it to a remote system
> via an IMAP SEARCH or NNTP XPAT operation, it will break if a user
> puts it into a news or http URL which gets transliterated and then
> percent-hacked, and it will break if a user types it into a web URL as
> a parameter to a server-based search function.  That's just search and
> fetch, nevermind other problems like damage to threading that results
> from an extended Message-ID which is manually added to See-Also or
> References, corrupted spam complaints, and the dozens of other common
> uses for this well-known, widely-used, STANDARDIZED data-type.

IDNA does not extend the message-ID type.  Message-IDs continue to be
ASCII-only, just as they always have been.  IDNA-aware applications,
as a user-interface courtesy, will usually render them using non-ASCII
characters (when they contain ACE labels), and allow users to enter
them using non-ASCII characters.  But an IDNA-aware program knows that
message-IDs are really ASCII-only, and will apply ToASCII as needed.

Everything you and I have said about message-IDs applies equally to
email addresses (which is not surprising, since message-IDs and email
addresses are syntactically the same).  Email addresses continue to be
ASCII-only, but for email addresses containing ACE labels, it will be
helpful for user interfaces to display/accept non-ASCII characters.  So
any internationalized program that deals with mail and human users is
going to have to deal with the conversion.  If it can cope with email
addresses, then it can cope with message-IDs in exactly the same way.

> Implicitly extending all such data-types currently in use on the
> Internet (as the current draft does)

We have found the crux of our disagreement.  I maintain that IDNA does
*not* extend or alter any data types.  It defines one new data type
(IDN), and defines a correspondence between IDNs and ASCII domain names,
to be used for user interfaces and for gatewaying between old & new
protocols/interfaces.  But it does not extend any existing types.  Any
existing type that contains ASCII domain names continues to contain
ASCII domain names.

AMC

---

From: "Eric A. Hall" <[email-masked]>
Date: Mon, 18 Mar 2002 22:07:31 -0600

"Adam M. Costello" wrote:

> promise to speak RFC 822 to the human user.

822 defines data-types which are used by multiple protocols. Nothing
"speaks" 822. Rather, several distributed functions exchange data which
has been formatted according to the rules specified in 822. In many
places, the data can leave the messaging network entirely and then be
freely reinserted. It is at this juncture that the mandatory conversion
causes the breakage to occur.

When you give somebody your email address, do you give them the
human-friendly rendering of "Adam Costello" or do you give them the 822
email address? Will this change with IDNA? Wouldn't you stick with the
822-compliant form, knowing that the sacrosanct 822 encoding is going to
work? It will only take a couple of retransmissions before you figure out
that the personal-address-exchange protocol only functions with the 822
form. We know that this will happen, experience and evidence tells us so,
why not skip it altogether?

> The program might omit certain fields that are required, and
> encoding is necessary.

2047 avoids structured data-types specifically for the reasons I mention.
When 2047 breaks, delivery still occurs. And when it does break, it is
somewhat noticable.

> *not* extend or alter any data types.

Call it "replaces" if you wish, it is the same effect. The data being
presented to the user is not the Message-ID that has always been presented
in the same manner, those things which look URLs aren't actually URLs
anymore, and so forth. Yep, that is exactly the problem.

It should be pointed out here that removing the mandatory transliteration
gives the collective Internet more options, not fewer options. Removing it
means that those data-types which should be extended (probably email
addresses) will be extended by the authoritative source if they can be
fixed at all, thereby improving all of the uses for that data-type, and
forcing the agents that use those data-types to think about the
ramifications of those changes, and how they can provide the necessary
support (SMTP, IMAP, News, and the associated messaging system parts). I
mean, which would you rather have, vendors doing it different ways because
there is no guidance, or having the vendors get together to decide on a
common way because the standard is being changed?

Conversely, if a data-type cannot be changed, well, thank goodness that
this WG didn't break anything that depends on it. :/

Meanwhile, those data-types which *should not* be extended (such as
Message-ID, among others) will not be extended by either us or the
controlling body, meaning that no unnecessary breakage will ever occur
with those data-types. Cumulatively, this flexibility equates to smoother
deployment in the short run and deeper benefits in the long run. This
should not be considered a set-back.

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 19 Mar 2002 05:11:49 +0000

"Eric A. Hall" <[email-masked]> wrote:

> When you give somebody your email address...  Wouldn't you stick with
> the 822-compliant form, knowing that the sacrosanct 822 encoding is
> going to work?

If I'm giving the address to people familiar with Latin letters, then
I'd stick to the 822 form.  If I'm giving the address to someone very
unfamiliar with Latin letters, so much so that the probability that they
fail to copy the address correctly exceeds the probability that their
mail program cannot handle non-ASCII characters, then it would make
sense to give them non-ASCII characters.

> > I maintain that IDNA does *not* extend or alter any data types.
>
> presented to the user is not the Message-ID that has always been
> presented in the same manner, those things which look URLs aren't
> actually URLs anymore, and so forth.

People commonly pass around things that are not URLs, like
"www.example.com", and yet most browsers transform it into a correct URL
"http://www.example.com/".  It will be a similar story when people start
passing around non-ASCII strings to be entered into browsers.

> It should be pointed out here that removing the mandatory
> transliteration gives the collective Internet more options, not fewer
> options.

I think "mandatory" is an overstatement.  The IDNA draft says "ACE
labels obtained from domain name slots SHOULD be hidden from users
except when the use of the non-ASCII form would cause problems or when
the ACE form is explicitly requested."  I think that leaves sufficient
room for application designers to exercise their own good judgement.

That said, I wouldn't really care if it instead said "ACE labels
obtained from domain name slots MAY be displayed to users as equivalent
non-ACE labels."  (And similar softening of the various recommendations
that follow from it in section 6.)

I don't think such a change is necessary, but I wouldn't object to it.
You'll probably still face resistance from others though.

AMC