Subject: RE: [idn] Web navigation for IDN resolving

From: Yves Arrouye <[email-masked]>
Date: Mon, 25 Mar 2002 10:40:27 -0800

>      As I  know in Taiwan and China Area, the input Chinese character
> string
> with non-protocol-head in Chinese Domain name form  typed into input  bar
> of
> IE , the browser will intercept these string and convert it to a UTF-8 %nn
> string as one of parameters to auto.search.msn.com , then it will be
> passed
> to Realname's server. The server of Realname will convert the dotted IDN
> domain name to an UTF-8 form and re-post it in the input bar with an error
> reply to hold it in client.
>      The server of Realname never to do dns lookup and not try to  re-
> direct
> the client browser to do dns lookup in replied UTF-8 form.

The DNS lookup happened *before* the string was URL-encoded and passed to
Autosearch, which only happen after a failure. Please refer to
draft-arrouye-idn-ie5-resolution-00.txt. RealNames *does* provide resolution
services for IDNs on a contractual basis, and so far we resolve .com, .org,
.net, and .jp. IDNs typed with these TLDs will resolve using an ACE.

YA

---

From: "Soobok Lee" <[email-masked]>
Date: Tue, 26 Mar 2002 09:59:06 +0900

----- Original Message ----- 
From: "Yves Arrouye" <[email-masked]>
To: "'tsenglm@????.??.tw'" <[email-masked]>; ""Yves Arrouye"" <[email-masked]>; <[email-masked]>
Sent: Tuesday, March 26, 2002 3:40 AM
Subject: RE: [idn] Web navigation for IDN resolving

> Autosearch, which only happen after a failure. 

In MS IE 5.0 or later,
this is true only for these two kinds of url inputs: try dns lookup first

  1)   with  explicit http: prefix

       http://www.<ML>.com
       http://<ML>.com

  2)   without http: prefix

       www.<ML>.com

  In these two cases, IEs try dns lookup for IDN hostname regardless of whether it is in utf8 or legacy encoding.

IE does *not* try dns lookup for this third kind of url input without http: prefix

  3)   hostnames begin with non-ASCII char
       <ML>.com 
       <ML>.<ML>.com

  IN this case, those inputs are treated as invalid inputs and  passed into
       auto.search.msn.com for MSN search/Realnames keyword lookups.

This IE's heuristic may be confusing to someone, but, that is not a error
and just attempts to be loyal to both user expectations and ASCII only URI spec.

IMO, 1) and 2) dns lookup should also have been prohibited and should be passed to search engines
 for IDNA or UTF8 conversions, because in most cases, they are in legacy encodings and make 
 troubles like legacy code confusion/collisions on DNS servers without appropirate conversions.
 
Netscape has different heuristic. Netscape passes any chars(including !?#$,,,) into dns lookups. 
  
Soobok Lee

> Please refer to

---

From: "Bruce Thomson" <[email-masked]>
Date: Tue, 26 Mar 2002 11:05:25 +0900

> IE does *not* try dns lookup for this third kind of url input without http: prefix

With IE 6, I find that it's doing the DNS lookup in all three cases.

Bruce

---

From: "Soobok Lee" <[email-masked]>
Date: Tue, 26 Mar 2002 12:18:45 +0900

----- Original Message -----
From: "Bruce Thomson" <[email-masked]>
> With IE 6, I find that it's doing the DNS lookup in all three cases.

I have not tested with Japanese or English version of MS IE 6.0.
I have only Korean MS IE 6.0 (version 6.0.2600.00), and find it does not try dns lookup for "<ML>.com"
and "<ML>.<ML>.com" and just sends them into auto.search.msn.com. ( I had a debugging session with my name servers ). the
auto.search.msn.com/realnames receives those search queries and converts them
into ACE ones and answers with redirection requests for http://bq--???.mltbd.com.

For other TLDs like "<ML>.cc", the search engine responds with "not found" error  pages.

---

From: [email-masked] <[email-masked]>
Date: Tue, 26 Mar 2002 18:09:33 +0800

Dear Yves Arrouye:
         Thanks your descriptions.   I think this message  from Mr. YangWoo
Ko ([email-masked]) is the key point:
I am still wondering what "looks like a domain name" means

> Autosearch, which only happen after a failure.
          The ASCII Domain Name (even without dot) will be  put to do DNS
lookup first, but IDN domain name will not . The IDN  will be intercepted to
auto.search.msn.com  directly without doing DNS lookup.  That means the IDN
input with native coded string without http://  protocol head will be
interpreted as  "not a legal domain name"  by IE.
          The server of  Realname can not found these IDN in non contract
supporting, so it is an error for Realname or IDN resolving, but the return
error cause the halting of browser will disturb the further regular
processing of these UTF-8 IDN.   That is my understanding of these strange
processing to IDN .

L.M.Tseng

---

From: "Edmon Chung" <[email-masked]>
Date: Tue, 26 Mar 2002 10:18:31 -0500

This is a rather interesting and serious observation.  Does it not therefore
mean that RealNames/MS is effectively hijacking all IDN requests through an
IE browser?

---

From: Dan Ebert <[email-masked]>
Date: 26 Mar 2002 07:53:26 -0800

If you prepend the http:// (i.e. "http://<ML>.cc") you can usually get
past the "not found" error pages.  I have found the sending behaviour to
vary depending both on the version of Windows (98/2000 etc., and the
language) and the version of the browser.  I have also observed that the
UTF-8 sent by IE is not always correct (but still looks like UTF-8).

Dan Ebert.

-- 
Dan Ebert      <[email-masked]>       eNIC Corporation
----------------------------------------------------------
 "Entropy isn't what it used to be."

---

From: "David Leung \(Neteka Inc.\)" <[email-masked]>
Date: Tue, 26 Mar 2002 11:29:32 -0500

From our testing results, we found out the same thing too... and worst of
all the new IE6 will DROP all IDN that is not within the locale set in the
regional setting of the OS.

i.e. if you set the locale to Japanese, ONLY characters contain in the
Japanese range will work and all the others will be DROPPED, try setting the
regional settings to say Korean or some other locale and if you have
"Chinese".tw the new IE will only sent .tw!!

We have try to replace the url.dll and urlmon.dll with the previous IE5 dlls
and it works again, we have pinpoint the functions inside these dlls and
send a bug fix report to MS but they still have not fix it yet... I think
this will prohibit the expand of IDN in China and Taiwan.

David Leung
Chief Technology Officer
Neteka Inc.
T: (416) 971-4302
http://w!.neteka.com
----- Original Message -----
From: "Edmon Chung" <[email-masked]>
To: "[email-masked]" <[email-masked]>; <[email-masked]>;
<[email-masked]>
Sent: Tuesday, March 26, 2002 10:18 AM
Subject: Re: [idn] Web navigation for IDN resolving

> > lookup first, but IDN domain name will not . The IDN  will be
intercepted
> to

---

From: "James Seng/Personal" <[email-masked]>
Date: Wed, 27 Mar 2002 00:20:25 +0800

While it is an interesting observation, the discussion of (right or wrong)
behavior of specific commerical/non-commerical software are better done
outside this wg.

The discussion of the how URL is to be encoded and how Host: field are to be
handled is probably more relevant so lets get back to that.

-James Seng

----- Original Message -----
From: "Edmon Chung" <[email-masked]>
To: "[email-masked]" <[email-masked]>; <[email-masked]>;
<[email-masked]>
Sent: Tuesday, March 26, 2002 11:18 PM
Subject: Re: [idn] Web navigation for IDN resolving

intercepted

---

From: "Adam M. Costello" <[email-masked]>
Date: Wed, 27 Mar 2002 03:30:59 +0000

James Seng/Personal <[email-masked]> wrote:

> The discussion of the how URL is to be encoded and how Host: field are
> to be handled is probably more relevant so lets get back to that.

Okay.  Eventually this message will arrive at the following proposal:

    Proposed repertoire for internationalized *host* labels:  All
    characters in classes L (letter), M (mark), and N (number) are
    allowed, and U+002D (hyphen-minus) is also allowed.  Everything else
    is forbidden.

The IRI proposal (draft-masinter-url-i18n-08) calls for the host labels
to be ASCII LDH only, just like in URIs.

When converting an IRI to a URI, you have to convert the path components
from the local charset to Unicode, then do Unicode normalization, UTF-8
encoding, and %-escaping.  But you don't do anything to the host labels
because they're already LDH.

I suspect that the reason the IRI proponents don't internationalize the
host field is that they don't yet have an official IDN spec to point at.
When they do, I suspect they'll want to revise their proposal so that
the host field can use the local charset.

This raises the question of what characters should be allowed in host
labels.  Since URIs do not allow arbitrary ASCII labels, only host
labels restricted to LDH characters, one would expect, analogously,
that IRIs would not allow arbitrary IDNs containing the exotic symbols
and punctuation allowed by Nameprep, but would allow only host labels
restricted to a selected set of characters,

Which characters should be allowed in internationalized host labels?
This is an interesting question in its own right, and it's possible that
the IESG will demand an answer.

The Unicode character database classifies each character as belonging to
exactly one of the following broad classes:

L: letter
M: mark
N: number
P: punctuation
S: symbol
Z: separator
C: other

We can start by examining which of these classes of ASCII characters are
allowed in ASCII host labels.

L: 52 exist, all are allowed
M:  0 exist
N: 10 exist, all are allowed
P: 23 exist, only hyphen-minus is allowed
S:  9 exist, none are allowed
Z:  1 exists, it is not allowed
C: 33 exist, none are allowed

We can trivially extend these results to form a simple rule covering the
entire Unicode repertoire, except that we have no precedent for class
M.  Since characters in class M tend to be things like diacritics, they
should be allowed.  So the proposed rule is:

All characters in classes L (letter), M (mark), and N (number) are
allowed, and U+002D (hyphen-minus) is also allowed.  Everything else is
forbidden.

Notice that there is no conflict with Nameprep, because Nameprep does
not prohibit any characters in classes L, M, or N.

If we were to adopt this definition of internationalized host name, it
would best be understood as an amendment of ToASCII step 3 (which checks
host name restrictions if applicable), tightening substep 3a from:

         (a) Verify the absence of non-LDH ASCII code points; that is, 
             the absence of 0..2C, 2E..2F, 3A..40, 5B..60, and 7B..7F.

to:

         (a) Verify that the sequence contains only host code points;
             that is, U+002D (hyphen-minus) and code points classified
             as L (letter), M (mark), or N (number).  See appendix ? for
             an enumeration of host code points.

Or maybe the enumeration would go in Nameprep, or in a separate document
that defines internationalized host names.

Getting back to IRIs and URIs:  I propose that conversion of an IRI
to URI involve applying ToASCII to each host label.  This would allow
conversion of any IRI to a URI without changing the syntax of URIs.  In
contrast, the method proposed in draft-ietf-idn-uri-01 would change the
URI syntax.

AMC

---

From: "James Seng" <[email-masked]>
Date: Wed, 27 Mar 2002 16:25:28 +0800

>     allowed, and U+002D (hyphen-minus) is also allowed.  Everything else

Hmm, the last discussion on this topic of hostname have been summaried by
Eric Hall - http://www.imc.org/idn/mail-archive/msg05182.html

If we want to go near this slippery slope, lets start from there.

> because they're already LDH.

Correct.

But I think IRI could be NFC-UTF-8, %-escaped even for host labels. This is
consistent with the CharMod of HTML. When IRI is used and that applications
have to look up the host names, then it can apply Nameprep-Punycode before
it hits the stub resolver.

To be more precies, lets look how browser can implementation IRI with IDNA.

- On the IRL bar -> IRL are likely to be in font encoding. Which font
  encoding would depend on the OS capability to render the font, ranging
  from locale to UTF-8. (This is why the confusing behavior of some
  browsers)

- Internally, the applications would process IRL in NFC-%-escape-UTF-8.

- For DNS resolution, host name would be extract from IRL in
  NFC-%-escape-UTF-8 which will be converted to Nameprep-Punycode
  before doing gethostbyname()

The two issues I could see would be:

1. For proxy look up, does the browser send IRI in NFC-%-escape-UTF-8?
   Or should it send out as NFC-%-escape-UTF-8 except hostname in
   Nameprep-Punycode? Or just UTF-8?

2. For "Host:" field in HTTP negiotation, does it send "Host:" in
   Nameprep-Punycode or does it send in NFC-%-escape-UTF-8? Or just
   UTF-8?

I believe for backward compatibility, browser sending both in
Nameprep-Punycode and not NFC-%-escape-UTF-8.

Am I going off tangent here?

> the IESG will demand an answer.

My believe is what is allowed in host labels is a topic for the zone
administrator to decide. .CN have a different set compared to .SG compared
to .COM compared to say IBM.COM. Each zone administrator have to decide what
to allow and what not to allowed.

> contrast, the method proposed in draft-ietf-idn-uri-01 would change the

I like this idea.

If this is done (i.e. IRI goes thru ToASCII for host label), then we
basically saying for proxy and Host:, you should start from URI not IRI.

-James Seng

---

From: Keld Jørn Simonsen <[email-masked]>
Date: Wed, 27 Mar 2002 10:28:59 +0100

> the IESG will demand an answer.

That is indeed a good question. The same question has been put forward
for programming languages that can use full ISO 10646 (Unicode) in
their variable names, and ISO has developed a list of characters 
that then are recommended for use in identifiers. This generally
consist of letters, digits and characters that can be used in
normal words in any language of the world. The specification is
available in annex A of ISO/IEC TR 10176. Leading Unicode people
participated in creating this list, which is freely available.
Get it via http://www.dkuug.dk/jtc1/sc22/wg20

Kind regards
Keld

---

From: "Kent Karlsson" <[email-masked]>
Date: Wed, 27 Mar 2002 15:54:05 +0100

> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]]On Behalf Of
> Keld Jørn Simonsen
...
> normal words in any language of the world.

I guess you mean 'orthographies of any language'...  But even so,
that is not quite true, since that annex only aims for what 10646
calls "level 2" (some combining characters are not included), not
"level 3" (any combining characters are allowed).  So, e.g. Lithuanian,
just to take one example, is not covered by it.  Java, XML, Unicode rec.,
and IDN 'identifiers' already aim for level 3.

		/kent k

> The specification is

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 27 Mar 2002 07:34:28 -0800

Thanks, Keld. For those interested, there is an updated version of the
recommended characters for programming language identifiers on:

http://www.unicode.org/Public/3.2-Update/DerivedCoreProperties-3.2.0.t
xt

The derivation is listed in comments in that file:

# Derived Property: ID_Start
#  Characters that can start an identifier.
#  Generated from Lu+Ll+Lt+Lm+Lo+Nl

# Derived Property: ID_Continue
#  Characters that can continue an identifier.
#  Generated from: ID_Start + Mn+Mc+Nd+Pc
#  NOTE: Cf characters should be filtered out.

The meaning of the abbreviations is on:

http://www.unicode.org/Public/3.2-Update/UnicodeData-3.2.0.html#Genera
l%20Category

(However, it was long ago agreed that IDN should be more inclusive,
rather than than more strictly follow programming language
restrictions (see the archives). Of course, it is in keeping with the
tenor of this group to endlessly revisit old issues...)

Mark

—————

Γνῶθι σαυτόν — Θαλῆς
[For transliteration, see http://oss.software.ibm.com/cgi-bin/icu/tr]

http://www.macchiato.com

---

From: "Soobok Lee" <[email-masked]>
Date: Thu, 28 Mar 2002 00:29:54 +0900

----- Original Message ----- 
From: "Adam M. Costello" <[email-masked]>
> exactly one of the following broad classes:

May I add this?

  U: unassigned code points.

> C: 33 exist, none are allowed

  U: indefinite, all are allowed .

> allowed, and U+002D (hyphen-minus) is also allowed.  Everything else is
 
 U should be also allowed in addition to L,M,N.
 But in later version of unicode , U may be partitioned into L' ~ C' and smaller U'.

 Soobok Lee

---

From: "Adam M. Costello" <[email-masked]>
Date: Thu, 28 Mar 2002 03:52:19 +0000

This message contains responses to both James Seng and Soobok Lee.

James Seng <[email-masked]> wrote:

> administrator to decide. .CN have a different set compared to .SG
> compared to .COM compared to say IBM.COM.

Zone administrators can always impose their own restrictions, but that
still leaves us with the question of what the IRI spec should say about
what characters are allowed in the host field of IRIs.

The historic precedent is that ASCII punctuation and symbols are allowed
in ASCII *domain* names, but not in ASCII *host* names, and not in the
host field of URIs.

Should IRIs be more loose and allow non-ASCII punctuation and symbols
in the host field (while continuing to disallow ASCII punctutation
and symbols)?  Or should IRIs try to apply an old tradition to a new
situtation, and disallow punctuation and symbols?

>   U: unassigned code points.

I see your motivation.  The classes I listed are all the ones mentioned
in the Unicode character database, but of course the database covers
only assigned code points.  All code points not mentioned in the
database are unassigned, and we could view that as another class.

>  U should be also allowed in addition to L,M,N.

ToASCII and Nameprep already take an input flag indicating whether
unassigned code points are to be allowed or prohibited.  My proposal
wouldn't change that.

AMC