Subject: Re: [idn] URL encoding in html page

From: Dan Oscarsson <[email-masked]>
Date: Wed, 27 Mar 2002 08:47:53 +0100

>From: "Adam M. Costello" <[email-masked]>
Adam M. Costello wrote:

>> So now IDN is a larger scope than we expected, not just browser
>> software needs to upgrade, even html editor like Dreamweaver, etc
>> needs to upgrade...
>
>It should not be surprising that any software that deals with domain
>names as such (as opposed to text in general) might need to be upgraded
>in order to allow characters in domain names that used to be forbidden.

It is the IETF hacker talking again. The common man do not think
non-ASCII
letters are forbidden.

In general you can all expect when soon non-ASCII domain names are
officially
allowed, there will be lots of places where people will enter names
using non-ASCII. They will expect them to be allowed in HTML documents
in
href field and in img src field. It does not matter that some have
decided
to restrict the definition of "URL" to ASCII, people will and already
do,
expect non-ASCII to be allowed in both host and path part of URL, in
native
character set without ACE or %-encoding!! Especially as it works in many
places
already today.

It is important that existing RFCs like the one for the URL, imediately
be
updated to allow non-ASCII letters. And do not use the IETF hacker
language
and call it IRI, for the common man it will be a URL and URI.
People will not ACE encode host names in URLs, people will not
%-encode paths in URLs. They will do like they do today: use native
character
set of the HTML document. Ignoring reality does not work.

   Dan

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Wed, 27 Mar 2002 08:22:46 -0800

>set of the HTML document. Ignoring reality does not work.

Quite true. The proposal to allow anyone to enter host names in URLs 
using any native encoding scheme, certainly ignores the reality that 
every DNS server would have to have equivalents for every name in 
every conceivable encoding scheme.

Similarly, the proposal to allow people to enter host names in URIs 
using only UTF-8 ignores the reality that many people enter text in 
many different encoding schemes and often have no idea what scheme 
they are using at the moment (the text on the screen appears the same 
regardless of the encoding scheme).

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: "David Leung \(Neteka Inc.\)" <[email-masked]>
Date: Wed, 27 Mar 2002 12:04:30 -0500

> It is the IETF hacker talking again. The common man do not think
So I think my Chinese name, or some Japanese, or Korean or any ones' name
that is non-ASCII should be forbidden in the internet world too : >

> character set without ACE or %-encoding!! Especially as it works in many
That's right ACE or % escaped is like making displayable 16bits unicode or
8bits UTF8 falling back to 7bits ASCII, and on the UI of most of the
applications or OS are moving towards the support of Unicode already.
So I think that URL should be displayable and should move towards non-ASCII
and should not move back to 7bits ASCII(actually 5bits ACE)... kind of
stepping backwards to make new things happen : )

> set of the HTML document. Ignoring reality does not work.
I think so too... what do they mean by the RFC only allow URL to have
non-ASCII letters, RFC is written by "HUMANS" and we can update them anytime
if we "WANT" to... think about the creation of IPv6 and all the RFCs that
requires to update when back then there is only IPv4... so should we say
that IPv6 format cannot be used and should somehow come up with an
"Algorithm" to convert the IPv6 into something that looks like IPv4.... : >

---

From: "David Leung \(Neteka Inc.\)" <[email-masked]>
Date: Wed, 27 Mar 2002 13:03:12 -0500

> every conceivable encoding scheme.

I think how the name is being display in the HTML or UI can be a seperate
issue of how it is going to be resolve at the backend, we can have IDNA with
ACE in the name lookup functions instead of the interface where users can
see the ACE... should we treat that as a leakage as well?

> regardless of the encoding scheme).

However this will happen for IDNA as well, IDNA with ACE ASSUMES that all
input are going to be in Unicode, however if user is inputting BIG5 or GB or
any local encoding, you have to specify it on the IDNA client's setting
before it can properly convert that to Unicode and to ACE anyways... so IDNA
and ACE is "ignoring the reality" too!!

But for newer OSes and applications I can see that there is more support for
UTF8 and local encoding is slowly being replaced... I think there should be
support for local encoding for the TRANSITIONAL period to make the
deployment of IDN smoother before users can move toward using UTF8.

---

From: Martin Duerst <[email-masked]>
Date: Thu, 28 Mar 2002 15:28:05 +0900

>and call it IRI, for the common man it will be a URL and URI.

You are referring to
http://www.ietf.org/internet-drafts/draft-masinter-url-i18n-08.txt
Internationalized Resource Identifiers (IRI).

As a co-author, I have to tell you that the draft became quite
a bit easier to write and read when introducing the distinction
between URIs and IRIs. And I expect the same benefit for other
specifications that reference this work. So I wouldn't want
to change back to the terminology used in earlier drafts.

However, I agree with you that in common use, people will most
probably call everything an URI or URL, and that should work
out fine.

Regards,    Martin.

#-#-#  Martin J. Du"rst, I18N Activity Lead, World Wide Web Consortium
#-#-#  mailto:[email-masked]   http://www.w3.org/People/D%C3%BCrst

---

From: "J. William Semich" <[email-masked]>
Date: Thu, 28 Mar 2002 19:05:45 -0500

For anyone on the list who would like to see how Verisign's ML.com domain 
name registrations can work right now in a lightweight URL-based system 
(which is what most users want right now anyway - not "domain names") using 
any local, UTF8 or ACE encoding, I've set up a demonstration of all of the 
multilingual domain names registered by Verisign under <ml>.com up until 
Nov., 2001, which currently resolve as bq--??.mltbd.com in the Verisign 
name servers. These will all work as ML URLs under the .com.nu domain.

So the domain name alingsås.com, which was registered by Register.com for 
one of its customers, and which currently resolves as the ACE domain name 
bq--abqwy2lom5z6k4y.mltbd.com to the Register.com "parked" page, will also 
resolve to the same parked page if the URL http://alingsås.com.nu/ is used, 
whether that URL is encoded as UTF-8 UNICODE or ISO-8859-1. The same holds 
true for the domain name l’esprit.com  which was registered for a customer 
by WorldNames and which points to the WorldNames parked page either as
bq--ed7wygp7mx7xh73q75zp62p7oq.mltbd.com  or as http://l’esprit.com.nu/.

Also try the NetSol customer's URL, http://wörterbuch.com.nu/ . I'm not 
reproducing any Asian, Arabic or similar URLs here for obvious reasons, but 
anyone on the list who knows of an Asian, etc., <ml>.com domain name that 
was registered before November, 2001, should feel free to test the system 
using any local encoding as the URL (not just ACE and not just UTF-8). Note 
that the RealNames/Microsoft "error" based search tool does not get into 
the act with these URLs, since there's no error for it to respond to.

These also all will work for hypertext links on a Web page, using local 
language encoding or UTF-8, and can also be saved as "favorites" in IE.

Bill Semich
.NU Domain

---

From: "Adam M. Costello" <[email-masked]>
Date: Fri, 29 Mar 2002 05:13:28 +0000

"J. William Semich" <[email-masked]> wrote:

> So the domain name alingsås.com, which was registered by Register.com
> for one of its customers, and which currently resolves as the ACE
> domain name bq--abqwy2lom5z6k4y.mltbd.com to the Register.com
> "parked" page, will also resolve to the same parked page if the URL
> http://alingsås.com.nu/ is used, whether that URL is encoded as UTF-8
> UNICODE or ISO-8859-1.

Depending on your resolver.  On my Debian GNU/Linux system, "dig" can
successfully look up alingsås.com.nu, and so can "host" (which warns
that the name is illegal but prints the IP address anyway).  But the
resolver won't have it:

> ping alingsås.com.nu
ping: unknown host alingsås.com.nu

I tried adding "options no-check-names" to /etc/resolv.conf, but that
had no effect.  I then tried changing it to "options no-check-names
debug", but I got no debugging messages.  I don't know what's going on
with the resolver.  I'm using glibc 2.2.5.

By the way, when I tried the URL with various browsers, netscape 4.x and
mozilla passed the host name straight through and ran into the resolver
problem, but w3m followed the unfortunate recommendation in the HTML
spec and converted the URL to http://alings%E5s.com.nu/, and of course
that lookup failed.

> These also all will work for hypertext links on a Web page

That depends on how the browser handles hrefs containing non-ASCII.
This is invalid HTML, so there's no telling what will happen.  The HTML
spec recommends that the browser convert the non-ASCII bytes to %HH
escapes, but I think that makes the host name lookup even more likely to
fail (unless the browser unescapes the host name just before doing the
lookup).

AMC

---

From: "J. William Semich" <[email-masked]>
Date: Fri, 29 Mar 2002 00:45:10 -0500

try the links in various encodings on these pages ....

http://www.worldnames.net/pages/demo.cfm

Bill

---

From: "J. William Semich" <[email-masked]>
Date: Fri, 29 Mar 2002 00:53:48 -0500

This lookup, using the HTML spec "alings%E5s.com.nu", works fine for me 
using IE - the system supports the html encoding just as it supports UTF-8, 
ACE or ISO-8859-X. Not sure what w3m is doing in this case. (see attached 
screen shot)

Bill

At 12:13 AM 3/29/2002, Adam M. Costello wrote:
>  w3m followed the unfortunate recommendation in the HTML
>that lookup failed

---

From: "Adam M. Costello" <[email-masked]>
Date: Fri, 29 Mar 2002 07:05:54 +0000

"J. William Semich" <[email-masked]> wrote:

> This lookup, using the HTML spec "alings%E5s.com.nu", works fine
> for me using IE - the system supports the html encoding just as it
> supports UTF-8, ACE or ISO-8859-X.

Okay, then w3m is failing for the same reason that netscape and mozilla
are failing: my resolver refuses to resolve names with non-LDH labels,
even if I put "options no-check-names" in /etc/resolv.conf.

> http://www.worldnames.net/pages/demo.cfm

Same story.  My resolver refuses to return the address.

Curiously, my three browsers apparently try to do three different
lookups.  All three display the Shift-JIS text correctly, but for the
first link, netscape 4.x does a lookup on ÅÐÏ¿½ê.nu, mozilla does a
lookup on %e7%99%bb%e9%8c%b2%e6%89%80.nu, and w3m does a lookup on
%93o%98^%8F%8A.nu.  I haven't checked, but I'm guessing that netscape
4.x converts the host name to euc-jp, mozilla converts it to UTF-8 and
then %-escapes it, and w3m leaves it as shift_jis but then %-escapes it.

AMC

---

From: "Bruce Thomson" <[email-masked]>
Date: Fri, 29 Mar 2002 17:38:22 +0900

These pages helped me get the resolver going. They might be applicable to
Debian as well.

http://www.i-dns.net/download/libc-freebsd/readme.txt
http://www.i-dns.net/download/libc-linux/readme.txt

Bruce