Subject: Re: [idn] WG last call summary

From: Randy Bush <[email-masked]>
Date: Fri, 15 Mar 2002 09:29:03 -0800

dan, for those of us out here who are mono-lingual honkeys, easily
confused when we cross a county line, blah blah blan, could you
give one *simple* *concrete* example of the failure mode you fear?
and try not to use adjectives.  thanks.

randy

---

From: "D. J. Bernstein" <[email-masked]>
Date: 15 Mar 2002 20:56:02 -0000

Randy Bush writes:
> could you give one *simple* *concrete* example of the failure mode you
> fear?

I assume that by ``failure mode'' you're referring specifically to
interoperability problems. Here's another example.

The ezmlm-list program takes a local mailing-list name on its command
line and sends the subscriber list to stdout. People use this program in
two ways:

   (1) Displaying the results on the screen, for human consumption. For
       example, the ezmlm-list output is often piped to the standard
       ``less'' pager: ezmlm-list ~/sos | less

   (2) Feeding the results to another program. For example, people have
       shell scripts along the lines of

          #!/bin/sh
          ezmlm-list ~/sos | ( while read sub
          do
            echo ... | qmail-inject -- "$sub"
          done )
       
       to send personalized email to all the subscribers.

Under the IDNA deployment plan, names in the IDNA character encoding
will start showing up in subscription lists immediately. Those names
will be displayed as gobbledygook. The ezmlm-list author will be forced
to have ezmlm-list convert addresses to the local character encoding.

However, if ezmlm-list converts from the IDNA character encoding to the
local character encoding, then the script shown above will break. Mail
will bounce. That's an interoperability disaster.

What is the ezmlm-list author supposed to do? How do we make IDNA work
without encountering this failure?

``Disable the conversion by resetting LANG inside the shell script,''
you say? Yes, we could change the shell script to eliminate the problem.
But that isn't under the control of the ezmlm-list author, and the IDNA
deployment plan doesn't wait for it.

Deliberately screwing up a user's email, and then telling him how to fix
the problem, is not interoperability. It's sabotage.

``Upgrade qmail-inject to support IDNA,'' you say? Yes, I could change
qmail-inject to convert addresses to the IDNA character encoding, and
that would make the script work if the local character set is Unicode
(encoded as UTF-8, presumably). But the IDNA deployment plan doesn't
wait for that either.

There is, of course, no safe order of package upgrades if IDNA is
deployed immediately. For example, addresses are copied from web
browsers to mail programs, and from mail programs back to web browsers.

It's possible to eliminate the interoperability failures by upgrading
_everything_, or by ugprading _nothing_. The problem is the transition.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

---

From: "Adam M. Costello" <[email-masked]>
Date: Fri, 15 Mar 2002 22:57:46 +0000

ATTENTION all participants in this thread:  Please think twice before
you Cc [email-masked], [email-masked], and [email-masked].  They are probably
getting annoyed with us by now.  Once again, I've removed those
addresses from the Cc field.

"D. J. Bernstein" <[email-masked]> wrote:

> line and sends the subscriber list to stdout.

The documentation for ezmlm-list says "ezmlm-list prints...one
address per line."  Clearly this is structured output designed to be
machine-readable, so IDNA rule 1 applies (domain labels in generic
domain name slots must contain only ASCII characters).  The output is
also intended to be human-readable, so IDNA rule 2 also applies (ACE
labels should be hidden from users).  But the IDNA spec says that when
rules 1 and 2 both apply, rule 1 takes precedence.  So ezmlm-list will
not perform ACE-decoding, and the IDNs will appear as ACE gobbledygook.

There are options available to both the user and the ezmlm-list author.
For the user, the output of ezmlm-list is already ill-suited for
display, because it's usually too long.  That's why the user usually
pipes it through a pager.  Similarly, the user can pipe it through a
program that performs ACE-decoding.  This works without any changes to
ezmlm-list.

The ezmlm-list author, meanwhile, can add an option to ezmlm-list that
changes the output format from "one address per line" to "one address
per line possibly containing IDNs using the local charset".  When this
option is used, the output contains internationalized domain name slots,
not generic domain name slots, so IDNA rule 1 no longer applies, and
rule 2 can assert itself, and ezmlm-list will perform ACE-decoding.  The
output format can be selected by a combination of compile-time default,
system-wide config file, user config file, environment variables, and/or
command-line switches.

(Another option available to the ezmlm-list author is to change
the output format without making it selectable, which would be an
incompatible change.  That's probably not a good idea.)

AMC

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Sat, 16 Mar 2002 11:52:02 -0600

> will bounce. That's an interoperability disaster.

under the scenario dan presents:

1) upgrading the qmail-inject program to support encoding will do no harm. 
Anything that will be changed by encoding was an illegal email address 
under the old rules.

2) upgrading both ezmlm-list and qmail-inject does no harm in the scenario 
given.

thus, I think the defensive behaviour for ezmlm-list is to implement a 
switch:

ezmlm-list 
--yes-I-have-really-upgraded-the-thing-that-will-read-this-to-understand-em
ail-addresses-with-utf-8-characters-in-them

given that in the endgame of essentially all programs being upgraded to 
undestand UTF-8 email addresses, this switch will be a bother, it should 
probably be capable of being specified in a configuration file, so that it 
can be left on and negated when needed.

3) contrary to some beliefs, watching gobbledygook is not known to cause 
permanent damage in human beings. So not upgrading anything does not cause 
much harm either.

4) yes, I do agree that specifying this level of behaviour is out of scope 
for the IETF.

                 Harald

---

From: "D. J. Bernstein" <[email-masked]>
Date: 17 Mar 2002 00:52:02 -0000

Harald Tveit Alvestrand writes:
> permanent damage in human beings.

Internationalized domain names are a failure if non-ASCII glyphs don't
appear on the screen. Have you completely lost sight of the goal here?

The interoperability problems begin when IDNA gobbledygook is converted
to the local character encoding for display. If the result is copied to
another program---through pipes, copy-and-paste, whatever---then that
program's lookups will fail. Mail will bounce, web links will fail, etc.

The conversions I'm talking about are specified in the IDNA documents.
They will produce interoperability problems visible on the wire. You are
incorrect in saying that this is outside the IETF's scope.

---

From: Keith Moore <[email-masked]>
Date: Sun, 17 Mar 2002 08:39:07 -0500

> appear on the screen. Have you completely lost sight of the goal here?

by that logic, IDNs are doomed to fail, because regardless of their
representation, few existing apps will be able to properly display them.
similarly, the entire Internet is a failure because it fails to route
packets to networks that don't exist yet.

not everyone accepts your criteria for success.  actually, it appears 
that your criteria for success are incompatible with many others'
criteria for success - in particular, that existing apps be able to
accept IDNs in the spaces that DNS names are currently acceptable.

Keith

---

From: [email-masked]
Date: Sun, 17 Mar 2002 21:27:38 -0800

On Sun, 17 Mar 2002 19:23:11 GMT, Paul Robinson <[email-masked]>
said:

> That's not actually completely true. Badly written software will break. In
> particular throwing up chars to the shell might break things, but that's
> particularly specific to a platform. Pipes won't break. Copy and Paste may
> depending on OS, and 'from where and to where'. Generally, it won't. If it
> does, it's because somebody has written some *really* crusty code that
can't
> handle ASCII outside 0-127. The IETF shouldn't be in the business of
making
> sure everythign works with badly written code.

Hmm.. so you're saying that *ALL* that code out there that double-checked
that
things that claimed (possibly implicitly) to be USASCII were in fact in the
0-127 range are "crusty" code?

Damn.  Sendmail 8.12.3.Beta1 is crusty - it actually bothers checking.

What's wrong with this picture?
--
Valdis Kletnieks
Computer Systems Senior Engineer
Virginia Tech

---

From: "James Seng" <[email-masked]>
Date: Mon, 18 Mar 2002 13:44:11 +0800

> appear on the screen. Have you completely lost sight of the goal here?

Dan,

On 28th May 2001 (oh boy, we been aroud for so long?), I response to you on
the similar discussion: See
http://www.imc.org/idn/mail-archive/msg02789.html

Particularly, it will explain why display of non-ASCII glyphs isnt as simple
as "just use UTF-8 and everything is okay".

-James Seng

---

From: "D. J. Bernstein" <[email-masked]>
Date: 18 Mar 2002 06:20:56 -0000

Let me emphasize that the IDN WG has received public written objections
to IDNA from at least fifteen regular WG participants and _hundreds_ of
other people. Everyone agrees that the objections together accuse IDNA
of causing a tremendous amount of damage, including bounced email, web
link failures, widespread user confusion, and more.

The IDN WG chairs have nevertheless sent IDNA to the IESG. That's
outrageous. It is, within the IETF, the most blatant abuse of power I've
ever seen.

I'm willing to spend some time discussing the technical issues that led
to these objections; but I'd like readers to keep the context in mind.
These specifications should never have been sent to the IESG.

Now, back to the technical discussion.

Keith Moore writes:
> not everyone accepts your criteria for success.

``Internationalized domain names are a failure if non-ASCII glyphs don't
appear on the screen.'' What kind of idiot would disagree with that?
Would you like to explain, Keith, how IDNs can be a success without
non-ASCII glyphs appearing on the screen?

Paul Robinson asks whether I'm saying that _one_ bad display means that
IDNs are a _complete_ failure. Of course not. One bad display means one
failure. One billion bad displays mean one billion failures.

Obviously the IDNA authors understand the desire to put non-ASCII glyphs
on the screen. Why is this obvious? Because the IDNA documents say that
the ad-hoc IDNA character encoding ``SHOULD be hidden from users,'' with
various exceptions. The documents spend quite a bit of time explaining
how to convert strings out of the IDNA character encoding.

The problem is that this conversion creates interoperability failures.
Mail will bounce. Web pages will fail to load. The same failures would
not have occurred with ASCII domain names.

These failures are visible on the wire, and are caused by a proposed
IETF specification. The assertion that they're outside the IETF's scope
is patently absurd.

On Sun, 27 May 2001 21:30:52 +0000, IDNA coauthor Adam Costello claimed
on the IDN WG mailing list that nothing would actually break. On Thu, 19
Jul 2001 04:31:48 +0000, Costello admitted on the IDN WG mailing list
that his claim was wrong. Things will break.

To answer another question from Paul: The problem happens when the
converted data is fed to _any_ network program that isn't aware of IDNA.
It doesn't matter whether that program is an obsolete 7-bit application
or a modern 8-bit-clean application; what matters is that the program
isn't undoing the IDNA conversion.

When I give specific examples of these IDNA interoperability problems,
someone inevitably responds that we can fix those examples by turning
off the initial IDNA conversion. That is a silly response, for two
reasons:

   * Preventing _all_ the interoperability problems means turning off
     conversion for _every_ display subject to IDNA-unaware piping,
     IDNA-unaware copy-and-paste, etc.---and that's a massive failure to
     achieve the IDN goal.

   * The proposal on the table, IDNA, does not disable these
     conversions. It explicitly encourages these conversions.

Anyway, while some IDNA proponents obviously want to slither between
interoperability failures and display failures, Costello has already
admitted that IDNA will cause both types of failures.

---

From: "D. J. Bernstein" <[email-masked]>
Date: 18 Mar 2002 07:08:19 -0000

[email-masked] writes:
> Hmm.. so you're saying that *ALL* that code out there that
> double-checked that things that claimed (possibly implicitly) to be
> USASCII were in fact in the 0-127 range are "crusty" code?

Time for some facts.

Sendmail, by default, does _not_ enforce the 0-127 restriction for mail
message headers. It allows bytes 160-255. Otherwise European users would
be dumping Sendmail even more quickly than they are today; ISO 8859-1
Subject lines are extremely popular.

Sendmail _does_ discard bytes 128-159 in mail message headers, because
it uses those bytes internally for its internal macro handling. Those
bytes aren't used in ISO 8859-1, but they are used in UTF-8. See
http://pi.cr.yp.to for a concrete example.

I sent Allman some email in February 1999 suggesting that he convert

   128 -> 255 160
   129 -> 255 161
   ...
   159 -> 255 191
   255 -> 255 255

with the opposite conversion on output. There have been several
security-fix releases of sendmail since then, so we could have had the
128-159 problem fixed on a huge number of machines. But he ignored the
suggestion. Apparently he doesn't care about international users.

People proposed more than a decade ago that the IETF require 8-bit-clean
mail software. (See, for example, Andre Pirard's ietf-smtp message dated
Tue, 19 Feb 91 12:08:00 +0100.) The only objection to this requirement
was the claim that 8-bit support would take a long time to be deployed.
Paul Vixie said that he had some seven-year-old sendmail binaries, for
example, and concluded ``with near-certainty'' that ``any changes to the
SMTP spec will take at least a decade to reach 90% of the critical
server population.''

In fact, it took less than a decade for every critical server to add
support for 8-bit message bodies, even though the IETF _still_ doesn't
require this. If the SMTP specification had been changed in 1991 to
require transparent 8-bit handling in both the header and the body, we
wouldn't have Sendmail's UTF-8 problems today.

Sendmail's continued data corruption is an embarrassment to the Sendmail
company. The fact that RFC 2821 and RFC 2822 allow this garbage is an
embarrassment to the IETF.

---

From: "D. J. Bernstein" <[email-masked]>
Date: 18 Mar 2002 08:42:27 -0000

James Seng writes:
> Particularly, it will explain why display of non-ASCII glyphs isnt as
> simple as "just use UTF-8 and everything is okay".

Here we go again: IDN WG co-chair James Seng responds to a discussion of
IDNA's flaws by attacking another proposal.

Think about that for a moment. IDNA has received massive objections
because it's horribly destructive. Seng has sent it to the IESG anyway.
How does he defend himself? By claiming that other proposals are worse.

What Seng fails to do is compare IDNA to the status quo. Sure, the
status quo forces sites to stick to ASCII, which is visually unpleasant
for many users. But the status quo also gets the mail delivered, makes
web links work, avoids the chaos and confusion that IDNA would produce,
avoids massive software upgrades and redeployment, etc.

Refusing to consider the status quo as an option is another way of
saying ``We have to _do something_.'' That's a recipe for the Internet
protocol suite to spiral down into a neverending nightmare. That's not
how the IETF works.

More importantly, in the absence of consensus, the status quo wins.
Something is seriously wrong when an internationalization proposal draws
objections from hundreds of Chinese-speaking users, for example.

P.S. If you'd like to see an example of a serious UTF-8 IDN proposal,
try http://cr.yp.to/proto/idnc3.html.

---

From: "James Seng" <[email-masked]>
Date: Mon, 18 Mar 2002 17:19:00 +0800

> IDNA's flaws by attacking another proposal.

Nope. If you read my mail again, you realise that the display problem which
you keep banging on occurs for both UTF-8 & punycode or whatever system you
come up with.

Display problem is problem at a higher level, one that can only be solved by
implementation and deployment, not protocol standardisation. If you
disagree, then explain how protocol can enforce what the display/render
engine do?

> How does he defend himself? By claiming that other proposals are worse.

Nope. The co-chairs believe we have rough consensus within the working group
to move forward these documents. The merits of other proposal(s) are
irrelevant to the status of the ones we move forward.

If and when the other documents have rough consensus, the co-chairs will
move them forward likewise.

> protocol suite to spiral down into a neverending nightmare. That's not

Once again, neither Marc or myself have said "we have to _do something_".
You hear it in your dream (or nightmare). Repeating it again and again does
not make it true, even if you wish it is so.

The IETF works by moving documents which have rough consensus. And that is
what we are doing here.

> objections from hundreds of Chinese-speaking users, for example.

In the absense of rough consensus, yes, status quo wins. But not in this
case.

The objections from the Taiwanese (non-wg members btw) are noted to the
group.
See http://www.imc.org/idn/mail-archive/msg05977.html

None of them provide any useful technical information to the last call.
Neither are the protest within the IETF process as described in RFC2026
Section 6.5.

-James Seng

---

From: Thor Harald Johansen <[email-masked]>
Date: Mon, 18 Mar 2002 03:14:33 -0800

> Particularly, it will explain why display of non-ASCII glyphs isnt as
simple
> as "just use UTF-8 and everything is okay".

I've been silently following this discussion for a while. Why is it that
international domain names can't be implemented with some Unicode
encapsulating 7-bit encoding? I understand that there will be issues of
case sensitivity (for latin alphabets) on servers and incorrect display on
old clients, but how many of you type your URLs in mixed case? The
transition will go largely unnoticed. It'll be the responsibility of the
creator of the domain to make sure his/her site is available, anyway.

Have I missed an important point here, or...?
--
Thor Harald Johansen

---

From: Paul Robinson <[email-masked]>
Date: Mon, 18 Mar 2002 04:53:49 -0800

On Mar 18, [email-masked] wrote:

> Hmm.. so you're saying that *ALL* that code out there that double-checked that

No. I'm saying that if a piece of software gets input that is unexpectedly 
'out of range' and then crashes and burns it is badly written. Sure, 
checking data is a good thing. Not checking it and letting things 'just 
break' is stupid. There are 14-year olds in high school in their second 
class of visual basic programming who know this.

To go back to the original argument, trying to put a stop on a standard
getting through because it breaks a piece of softwre written some time ago
that a small percentage of people use today is a dumb idea. OK, let's take
IDNA. How many people are there in China that would benefit from this
process? Let's say the population is 2 billion, and it would benefit 5%.  
That's 100 million people. How many people would be 'hurt' by the example
given breaking? Are there 100 million people out there who would be
affected? If yes, then the process needs to stop and take that into account.

To argue that it's upsetting the existing user base is also flawed - this is 
not about keeping things cosy for ourselves at a cost to those outside of 
the Latin alphabet. If somebody can show how this process will break things 
for the majority of users (in other words using IE 4 and above, with Outlook 
Express for mail), THEN it would be a good idea to question 
interoperability. I'm not saying that we should be considering IE and 
Outlook Express as de facto standards, just as a benchmark to how many 
people are likely to experience 'breakages' as a result of IDNA things being 
pushed through.
 
> What's wrong with this picture?

There's too much blue in it, and its skewed to the left a little. :-)

-- 
Paul Robinson

---

From: [email-masked]
Date: Mon, 18 Mar 2002 06:08:25 -0800 (PST)

> that a small percentage of people use today is a dumb idea.

I have very little sympathy for software that breaks when given unexpected
input. But this is not the concern. The concern is that there is a bunch of
widely used software that prohibits certain inputs. It does so because the
standards in effect at the time the software was written said those inputs are
illegal. 

> interoperability.

Here's a quote from a message sent by Carl Gutekunst,
<[email-masked]>, to the IDN list on 10-Feb-2000 that directly
addresses this:

 I conducted some experiments with raw 8-bit data in message header fields a
 year or so ago in response to requests from customers in Japan and China.
 There were (and are) some native Japanese and Chinese language E-mail clients
 that quite happily stuff unencoded Shift-JIS, EUC JIS, Big 5, et al into the
 Subject and the phrase fields (but not the route-addr) of addresses.

 The results were similar to what we saw with Just-Send-8 a decade ago. As long
 as the entire end-to-end path had been carefully managed to specifically
 handle these messages, they would be received and displayed correctly. But if
 any SMTP relay or client wasn't party to the bilateral agreement, all kinds of
 damage happened, including RFC2047 conversions, nonsensical character set
 conversions, seemingly random mangling, 8-bit truncation, and non-deliveries.
 (And I'm *not* including untagged charset mismatches, which would not be an
 issue if we used UTF-8 exclusively.)

Carl later stated that he tested using sendmail 8.9.3, UofW Imapd 4.5, and
Netscape Communicator 4.7. The latter dumped core.

I've done similar tests and reached similar conclusions.

In short, this proposal DOES break things, so interoperability has to be
questioned.

				Ned

---

From: Keith Moore <[email-masked]>
Date: Mon, 18 Mar 2002 11:07:15 -0500

> appear on the screen.'' What kind of idiot would disagree with that?

anybody with two working brain cells should be able to figure out
that IDNs won't be able to be displayed with software that was
designed and written before IDNs existed.   even if IDNs were bare
UTF-8, they still wouldn't be able to be input from software
that predated IDNs because such software wouldn't do the appropriate
canonicalization/normalization and wouldn't know how to properly
compare two IDNs for equivalence.

but you know this already.  you just choose to ignore it so that
you can berate people who don't want to rewrite all applications to
suit your whim.

Keith

---

From: Keith Moore <[email-masked]>
Date: Mon, 18 Mar 2002 11:13:09 -0500

> Neither are the protest within the IETF process as described in RFC2026

I have to disagree with this.

first of all, there's no such thing as a wg member.   anybody who
contributes to the discussion is as much a member as anyone else.

second, this is useful technical input, and the problems with
TC/SC equivalence are serious.  even if they can't be fixed 
reasonably with the current DNS protocol, they do have some
bearing on the applicability of IDN.

third, I don't see why you say these are not within the IETF process,
but I find no justification for that statement.  if nothing else,
the spirit of IETF has always been to consider any constructive input,
even if it were not presented in exactly the correct way.

Keith

---

From: Dave Crocker <[email-masked]>
Date: Mon, 18 Mar 2002 10:23:21 -0600

>What Seng fails to do is compare IDNA to the status quo. Sure, the

this has been done many times.  there is no need to repeat just because a 
participant failed to acknowledge or understand it.

d/

----------
Dave Crocker <mailto:[email-masked]>
TribalWise, Inc. <http://www.tribalwise.com>
tel +1.408.246.8253; fax +1.408.850.1850

---

From: "Doug Ewell" <[email-masked]>
Date: Mon, 18 Mar 2002 08:44:10 -0800

[irrelevant cc: addresses snipped]

On Thursday I responded to D.J. "Dan" Bernstein <[email-masked]>:

>> The bottom line, once again, is that at least fifteen regular WG
>> participants and _hundreds_ of other people have written messages
>> to the IDN mailing list in opposition to IDNA.
>
> Identically worded chain letters don't count.  It's the same thing
when
> you write to your congressperson: they throw out the Xeroxed ones
> immediately.

On Sunday, Dan wrote:

> other people.

> objections from hundreds of Chinese-speaking users, for example.

And Dan accuses *others* of not listening?

Let me try again, for what I expect is the last time:

HUNDREDS of people who have been told to write a letter, about an issue
they may know nothing at all about, and who have been handed the exact
text of the letter, such that they merely need to sign their names and
send it, does NOT indicate a legitimate technical concern on the part of
HUNDREDS of people.  It indicates that HUNDREDS of people are capable of
following instructions.

Dan also wrote:

> Everyone agrees that the objections together accuse IDNA

"Everyone," in this case, meaning the hundreds of lemmings who sent the
chain letters.

Once again:  IDNA can *only* cause anything like these problems if a
user manually types an IDN name differently from the way it was
originally presented.  That means the user must use TC when TC is
presented, and SC when SC is presented, just as all of us must currently
do with BiCapitalized web pages.  I agree that this is not the best
solution that the human imagination can admit, but I do NOT agree, nor
does the rest of this list, that this constitutes the sort of Armageddon
that Dan envisions.

Previously I had written:

> Nobody was ever able to explain to me, anyway, how providing Chinese
> domain names in Han characters, but not mapping them between TC and
> SC, could possibly cause more damage, conflict, and chaos than forcing
> the Chinese to use ASCII, as the current system does.

And still nobody has done so, least of all Dan, who seems more intent
than anyone on using this purported "damage, conflict, and chaos" to put
a stop to IDNA.

I no longer have any interest in reading anything Dan has to say, for or
against IDNA.  Reading his posts is like listening to a Charmin' Chattie
doll who only has four or five things to say and no capacity to listen
to others.  I will be deleting all of Dan's future messages on sight
from now on.  Whether the rest of the list chooses to do likewise is
entirely up to them.

-Doug Ewell
 Fullerton, California

---

From: "Doug Ewell" <[email-masked]>
Date: Mon, 18 Mar 2002 08:47:26 -0800

Thor Harald Johansen <[email-masked]> wrote:

> Why is it that
> encapsulating 7-bit encoding?

That's exactly what Punycode is.

-Doug Ewell
 Fullerton, California

---

From: [email-masked]
Date: Mon, 18 Mar 2002 06:08:25 -0800 (PST)

Ned

---

From: "James Seng" <[email-masked]>
Date: Tue, 19 Mar 2002 10:34:59 +0800

> contributes to the discussion is as much a member as anyone else.

I meant to say they are not subscriber of the wg.

(btw, it is not funny to get 353 bounce mails, sorting them out and manually
replying to each of them.)

> bearing on the applicability of IDN.

It is useful technical input on the first time.
By the 353th time, someone got to ask what else are they contributing?

> even if it were not presented in exactly the correct way.

It is a protest and appeal against the last call. The IETF process specify
the exact process to do so in RFC2026 Section 6.5.

-James Seng

---

From: [email-masked]
Date: Mon, 18 Mar 2002 18:38:37 -0800

> > Hmm.. so you're saying that *ALL* that code out there that
double-checked that

> class of visual basic programming who know this.

OK.. Thanks for the clarification. The distinction is important, because:

> standards in effect at the time the software was written said those inputs
are

And let's remember - currently, if it's *not* ascii and *not*
RFC2047-encoded,
it is *illegal* in email headers and *IF* it works, it only does so either
*BY ACCIDENT* or *BY MUTUAL PRIVATE AGREEMENT*.

As such, I have to insist that *any* IDN proposal be required to be
backwards
transparent to the current 2821/2822/2045-2049 standards the same way that
MIME/ESMTP was required to be backwards transparent to the then-existing
821/822
infrastructure.

On the other hand, I *will* accept an IDN proposal that looks as ugly
to non-upgraded software as 2047-encoding does to a non-2047 capable MUA.
--
Valdis Kletnieks
Computer Systems Senior Engineer
Virginia Tech

---

From: Keith Moore <[email-masked]>
Date: Mon, 18 Mar 2002 22:05:40 -0500

> By the 353th time, someone got to ask what else are they contributing?

of course once should be sufficient.   
some people don't realize that we don't do voting.

> the exact process to do so in RFC2026 Section 6.5.

perhaps, but this isn't a justification for ignoring valid concerns.

Keith

---

From: "D. J. Bernstein" <[email-masked]>
Date: 19 Mar 2002 03:45:56 -0000

[email-masked] writes:
> backwards transparent to the current 2821/2822/2045-2049 standards

Putting UTF-8 into email headers right now would trigger a serious
Sendmail bug, so of course I don't encourage anyone to do that.

However, the Sendmail bug should be fixed as soon as possible. Do you
disagree with this?

Furthermore, the IETF specifications that allow 7-bit software should be
fixed as soon as possible. Do you disagree with this?

Or do you want these bugs to continue to plague programmers in 10 years?
20 years? 50 years?

---

From: Mats Dufberg <[email-masked]>
Date: Tue, 19 Mar 2002 05:01:03 +0100 (CET)

> Sendmail bug, so of course I don't encourage anyone to do that.

Putting UTF-8 into the mail header is against the mail protocol, and you
know that. Any bug in sendmail is irrelevant. If your want to change the
mail standard, write a draft. That draft will NOT be processed in the IDN
group.

> However, the Sendmail bug should be fixed as soon as possible. Do you

That discussion should be taken with those working on sendmail, not any
IETF group. Those who are dissatisfied wiht sendmail use something else.

> fixed as soon as possible. Do you disagree with this?

Write a draft.

> Or do you want these bugs to continue to plague programmers in 10 years?

IETF has nothing to do with bugs in programs, just bugs in protocols. You
know that. And how long will you plague us with your complaints about
sendmail? 10 years? 20 years? -- Stick to what is relevant!

Mats

----------------------------------------------------------------------
Mats Dufberg <[email-masked]>
----------------------------------------------------------------------

---

From: "Bruce Thomson" <[email-masked]>
Date: Tue, 19 Mar 2002 13:25:51 +0900

> sendmail? 10 years? 20 years? -- Stick to what is relevant!

No, this WG has come up with a proposal which is targeted specifically
at circumventing bugs in legacy implementations. If it was only a question
of choosing an ideal specification and ignoring bugs, "just send UTF"
would be a clear winner. Dan's comments here are perfectly
relevant.

Or are you saying that IDNA is needed merely because this WG
doesn't have access to the particular protocols that need to be
changed to do this thing right?

Bruce

---

From: Mats Dufberg <[email-masked]>
Date: Tue, 19 Mar 2002 05:50:20 +0100 (CET)

On Mar 19, 2002, 13:25 (+0900) Bruce Thomson <[email-masked]> wrote:

> at circumventing bugs in legacy implementations.

Dan's complain was about the restriction that mail headers only may
contain ASCII (7 bit). Move that to an appropriate wg.

> If it was only a question of choosing an ideal specification and
> ignoring bugs, "just send UTF" would be a clear winner.

I do not agree. There is a specification of a host name, which sets the
limit on DNS (for host names) to a-z, 0-9 and '-'. The ACE solution is
also better because it breaks less systems during transition.

> changed to do this thing right?

We have a reality, and changing to a new model will require a transition.
A good protocol will respect those two factors.

IETF's task is not to repair software bugs or remove limitations set by
the software authors, but when writing new protocol, the IETF has to
respect the reality and to a cerain extent, the effects on currently
installed programs.

If we would create DNS from nothing today, we would do it differently. In
that case, UTF-8 would be a good candidate for the coding scheme. But we
already have DNS, and that fact connot be neglected.

Mats

---

From: "Bruce Thomson" <[email-masked]>
Date: Tue, 19 Mar 2002 14:06:02 +0900

> A good protocol will respect those two factors.
I agree.
 
> respect the reality and to a cerain extent, the effects on currently
Right.

> already have DNS, and that fact connot be neglected.
Perhaps you mean sendmail, not DNS? DNS is already 8-bit clean and works
fine with UTF. Sendmail and various other apps apparently don't, and that's
what Dan was referring to. That's why I couldn't understand why you said
he was off the subject.

He is right that this whole project is about the tradeoffs in costs of fixing things
and costs incurred when things break. Where you disagree is on the resolution
of those tradeoffs.

Bruce

---

From: Mats Dufberg <[email-masked]>
Date: Tue, 19 Mar 2002 06:28:32 +0100 (CET)

On Mar 19, 2002, 14:06 (+0900) Bruce Thomson <[email-masked]> wrote:

> fine with UTF.

No, I'm talking about DNS. -- No, I don't agree with your statement about
DNS. -- Hey, should we repeat the entire discussion aobut DNS, from which
most of us came to the conclusion that ACE was the best solution? :-)

> Sendmail and various other apps apparently don't, and that's what Dan
> was referring to. That's why I couldn't understand why you said he was
> off the subject.

The mail protocol sets a limit to 7 bits (independently of any bugs in
sendmail). Is the mail protocol a topic of this working group?

Mr. Bernstein has a hang on some bugs in sendmail (or are they just
limitations due to the design), and he repeats that i every second mail.
He should discuss those bugs with the sendmail people, and possibly with
those who have installed his own MTA. It does not belong to any of the
IETF mail lists I belong to.

> He is right that this whole project is about the tradeoffs in costs of
> fixing things and costs incurred when things break. Where you disagree
> is on the resolution of those tradeoffs.

I do not agree with your conclusion.

Mats

---

From: "Bruce Thomson" <[email-masked]>
Date: Tue, 19 Mar 2002 14:50:26 +0900

> I do not agree with your conclusion.

Really? I was hoping that we could at least agree on this. OK, let me
analyze the factors that might possibly be relevant to our decision.

1. Cost vs. benefits (including time to market, etc.)
2. The fact that many people want to do it a particular way.
3. Our sense that a particular solution is more elegant or nice.
4. Special interest groups.
5. The structure of the IETF decision-making process that mandates
     that we do things in a way not dictated by 1 through 4.

Have I left out any possibilities here? I am just trying to say that
number 1 is the main determining factor and probably has led to
us being where we are now. And that the final specification will
logically be justifiable in these terms.

Bruce

---

From: Thor Harald Johansen <[email-masked]>
Date: Tue, 19 Mar 2002 12:06:04 +0100 (CET)

> Or do you want these bugs to continue to plague programmers in 10 years?

I'm having trouble understanding why we're still using these old
protocols. Does a standardized transparent (in the "binary" sense of the
word) mail protocol with built-in support for multi-part messages exist?
Then why isn't it being implemented all over the place?

It would make things a whole lot simpler: no UTF-8 or MIME encoding of
international characters would be nessecary (use Unicode), binary
attachments would take up significantly less space and Unicode IDNs could
easily be supported.
-- 
Thor Harald Johansen

---

From: YangWoo Ko <[email-masked]>
Date: Tue, 19 Mar 2002 20:29:11 +0900

[I removed iesg, iab, ...]

(Short term) solution on Internet tends to be widespread as legacy
which will hardly obsoleted without fundamental flaws which prevent 
users from using it anymore.

-- 
/*------------------------------------------------
YangWoo Ko : [email-masked]
We Invent Enterprise Software Solutions
and Make You Secure & Powerful.
------------------------------------------------*/

---

From: Thor Harald Johansen <[email-masked]>
Date: Tue, 19 Mar 2002 13:42:37 +0100 (CET)

> users from using it anymore.

Of course, but such a transition would go unnoticed if done correctly.
Example: A special SMTP handshake wich would be ignored by old
implementations (servers or clients) and acted upon by newer software.
The end user wouldn't notice anything but an improvement in performance on
some servers.

When the special handshake would go into effect, the sockets on both sides
would go from being line-based to becoming transparent. The protocol could
be based on single byte commands, length dwords and such, to allow for
content independent encapsulation with a non-existant encoding overhead.

New servers would use Unicode for internal representation of the plain
text content of messages. Whenever mail would be exchanged with an old
server, one way or the other, the new server would do the nessecary
encoding/decoding and/or encapsulation.

The first servers to be upgraded would barely use the new protocol. The
handshake would simply not generate a reply at any server or client. This
has to be expected and accepted during the transition from one protocol to
another. The effect of the new protocol would be the gradual improvement
in effective bandwidth usage on the Internet.

Another idea would be the addition of compression. One (or the
implementation) could choose to compress individual message parts,
individual messages or the entire content of the inbox of a mail account.
This would also improve bandwidth usage, and have a noticable effect on
servers exchanging large amounts of mail.

I can only see advantages.
-- 
Thor Harald Johansen

---

From: "J. William Semich" <[email-masked]>
Date: Tue, 19 Mar 2002 12:29:21 -0500

Correct for RFC 1035 and the BIND 9 implementation of it at least.

Bill Semich
.NU Domain

On Mar 19, 2002, 14:06 (+0900) Bruce Thomson <[email-masked]> wrote:

>DNS is already 8-bit clean and works fine with UTF.

---

From: Mats Dufberg <[email-masked]>
Date: Tue, 19 Mar 2002 20:16:44 +0100 (CET)

> analyze the factors that might possibly be relevant to our decision.

I was probably too harsh. :-) Yes, we all (?) agree that any solution will

>      that we do things in a way not dictated by 1 through 4.

6. The current use of current protocol.
7. The strictness of current protocol.

Mats

---

From: Mats Dufberg <[email-masked]>
Date: Tue, 19 Mar 2002 20:36:35 +0100 (CET)

> Correct for RFC 1035 and the BIND 9 implementation of it at least.
(...)
> >DNS is already 8-bit clean and works fine with UTF.

If you by DNS only consider the transport in DNS packets, DNS can handle
any 8 bit codes. Only ASCII characters are defined, which means that high
octetts are are undefined.

If you by DNS include host name restrictions, restrictions in other
protocols, current usage and display, the set of allowed characters is far
more restricted.

My conclusion is that DNS is not 8 bit clean. One problem of DNS is that
the defintion of the protocol is strict, and that the usage has not been
reflected into the protocol.

Mats

---

From: John C Klensin <[email-masked]>
Date: Tue, 19 Mar 2002 17:37:48 -0500

> Correct for RFC 1035 and the BIND 9 implementation of it at

Well, we can keep repeating this until it becomes true, water
flies uphill, and other convenient things happen.  However, a
careful reading of RFC 1035 is illuminating:

(i) It requires that all character strings (explicitly including
labels and domain names, not just "host names") be compared in a
case-insensitive manner (section 2.3.3).  As this working group
has discovered multiple times now, case-insensitivity is not
precisely defined for many non-ASCII scripts.  And, in
particular, there is no simple operation (i.e., not requiring
mapping tables) that implements case-insensitive mapping for
arbitrary UTF-8 strings.

(ii) While it provides that "non-alphabetic codes must match
exactly" (end of section 3.1), it would be quite a stretch to
argue that the characters of, say, Greek are "non-alphabetic".

In any event, one can't just put binary information in there
without knowing whether it is to be compared case-insensitively
(whatever that means) or absolutely.   And RFC 1034 strongly
implies that binary information can be used only with new RR
types (or, presumably, with new Classes, not in the existing RR
types and classes.

      john

---

From: "D. J. Bernstein" <[email-masked]>
Date: 20 Mar 2002 01:25:48 -0000

John C Klensin writes:
> flies uphill, and other convenient things happen.

There is no excuse for 8-bit failures under the DNS specifications. More
importantly, there is no evidence of 8-bit failures in real DNS servers.

> As this working group has discovered multiple times now,
> case-insensitivity is not precisely defined for many non-ASCII
> scripts.

RFC 1034 says ``case-insensitive manner, assuming an ASCII character
set.'' The obvious interpretation, and the interpretation universally
adopted by DNS cache implementors, is that (1) bytes 65-90 are treated
identically to bytes 97-122 respectively and (2) there are no other
equivalences.

If the wording isn't clear enough for non-implementors, perhaps it could
be improved, but that wouldn't justify your false claims that DNS has
problems with 8-bit names.