Subject: [I18nrp] Confusion among characters and strings

From: John C Klensin <[email-masked]>
Date: Tue, 12 Jun 2018 13:26:14 -0400

Hi.

I'm still trying to resist discussions of specific proposed
fixes, but, but "confusability" has now been mentioned by
several people as if it where the only issue, or even nearly the
only issue) and easily solved, let me push back on that as a
sort of third prong to suggesting that reviewing actually i18n
protocol work is different from reviewing protocols that have
non-ASCII elements or implications.

First, if one could wave a magic algorithm (or wand) at the
confusion issue and make it disappear, there would still be a
number of outstanding issues, with the question of what to do
about normalization anomalies at the top of today's version of
my personal list.  So "if we solve confusion, there are no other
important issues" is, at least IMO, fairly obviously false.

(A small note on what follows: where convenient, I've chosen
examples from scripts I think are likely to be readily
available, and to render in a reasonably consistent and
predictable way, for most people and computer setups who are
likely to be dealing with the ASCII in which most of this text
is written.  All non-ASCII characters have been identified by
the Unicode code points and I've sometimes omitted the graphic
forms because they were not easily inserted.  I've avoid scripts
that require special rendering rules, scripts that are
predominantly right-to-left, and scripts with very complex
characters for convenience, not because similar examples could
not be constructed with them.  I hope I've avoided this note
becoming an English/ASCII user's view of the world; if it seems
to drift in that direction, the reason is more likely the about
than that particular bias.)

The more important thing about confusion is that it spans a very
wide range of issues.  At one extreme are characters that
despite having graphemes that are in closely-related scripts,
historically copies of each other, are assigned to different
code points.  Consider the oft-cited "a" (U+0061) and "а"
(U+0430) or "o" (U+006F) and "ο" (U+03BF).  As what I think of
as another tier, consider cases that are extremely dependent on
choices of type styles, such as  "y" (U+0079), "у" (U+0443) and
"ч" (U+0447).  Some of these can also be traps for the unwary
(whether automated or human).  For example, if one is not aware
of the existence of a character that looks like "ч" separate
from "у", the two are a lot more likely to be mistaken for each
other.  Of course, typestyle distinctions can be used to reduce
or eliminate confusion, not just to cause it, with the classic
ASCII example being the use of a slashed 0 for zero,
unambiguously distinguishing from "0" and "0".  Proving that one
should be careful about what one wishes for, as one starts using
non-Basic Latin, that slashed zero is potentially confusable
with "ø" (U+00FB).   

As a final example in this general group for people to think
about, "3com" caused the original rule prohibiting domain name
labels from having leading digits. Is "Зcom" (first character
is U+0417) confusingly similar?  If yes, is it confusing to the
same or a greater degree in "label12З4"?

In another tier are coincidences in which similar graphemes
occur, with different "meanings" on mostly-unrelated scripts.
For example, are "o" (U+006F) and "o" (U+0665) confusable with
each other?  The answer probably depends on type style, type
size, how closely the user is looking and what she is expecting,
etc.  Or how about U+110B, U+17E0, or U+-25CB?.   For IDNs (but
they are not the only important case), it may or may not be
important that some of the code points in that list are
prohibited by IDNA2008: if IDNA2008 is adhered to by registries
or lookup applications follow the IDNA2008 rules for checking,
they will be rejected even without worrying about confusion.
However, we know of registries who ignore those rules and
register strings as labels that IDNA2008 prohibits (and there
are almost certainly more we don't know about if the whole DNS
is considered) and that UTR#46 recommends not making those
lookup-time checks, so it is hard to rely on that.

The sequence continues with a number of other cases, but the
above should make at least part of the problem clear: depending
on circumstances and expectations, people cannot (or will not)
distinguish among things that computer algorithms can
distinguish easier, especially if they are relying on Unicode
coed points and high-distinguishability fonts which the people
(especially in the presence of CSS on web pages and some
equivalents elsewhere).  Even if one uses artificial
intelligence-like techniques, the resulting algorithms and tests
are going to be no better than their training: where humans
cannot agree on what is confusable and there is no consensus
about "right", it is difficult to believe in the reliability and
accuracy of such methods.

Scripts that require special rendering and type families that
use kerning sufficiently aggressively to construct
near-ligatures create more "opportunities" in the form of cases
in which multiple code points must be considered as a sequence
even without combining characters.   As an ASCII example and
with some type styles, it can be hard to distinguish "iN" in the
middle of a string from "m" unless one has other context.

However, there is another point at the far end of that spectrum.
A person with sufficient expectations of seeing one thing, e.g.,
because they are expecting a particular script family and not
some others, will see what they expect to see.   For example, is
  Toys-Я-Us
insertion of a Cyrillic character (in which case it should not
match "Toys-R-Us" and might be considered confusingly similar)
or it is someone overdosing on cleverness (in which case maybe
it does compare equal -- a conclusions that I understand has
been reached by trademark authorities).  Similarly, with the
right choice of type styles, small type, or a hangover, 
    "n" and "ฑ" (U+0E11)

It is probably also worth remembering that, to a typical user,
to question of whether "color" and "colour" are equivalent is at
least as significant (and a potential source of confusion) as
the relationship between "color" and "co1or".

As an additional complication, there may be a difference between
the rules we would make if we are concerned about problems that
could arise from accidents or linguistic issues versus those are
are likely to occur only in the presence of malice (or an excess
of cleverness).   Coming back to the 3com example, "Зcom"
easily detected and quite likely to be malicious, but "Зсом"
(U+0417 U+0441 U+043E U+043C) is a great deal more problematic.
Similarly "g00gle" or "g00g1e" are almost certain to be
problematic (especially if one remembers those are all-ASCII
strings and hence that the first can be written "G00GLE", which
is less obvious), but we have many labels with digits in the
middle (e.g., to allow names for routing nodes to match ITU
standards) that are not issues at all.

Now, it is clear to me that, while some cases are easily
addressed, many are not, and some get well into the area of
subjective, context-dependent and user-experience-dependent,
judgments.  The question is what to do about it.  The two
answers I personally find unsatisfactory are, to exaggerate a
bit, "we can't solve all of the problem, so we should just give
up and tell users they are on their own" and "we should solve
the problems we can and then claim that all of the others are
edge cases that should not occur in practice".   I think I've
heard positions stated that soul a lot to me like one or the
other. so I might be in the minority.  At the same time, I think
that, no matter what we do, there are cases that depend on
either user vigilance or registrars taking much more
responsibility than we have often seen in recent years (or
both).  That observation isn't new: IDNA2008 imposes the latter
as a requirement.  If we are going to try to draw a boundary
between the cases we hope to address by rules and algorithms and
those cases we want or need to leave to others, I think we are
obligated to make that boundary as clear and easy to understand
as possible... and that means we both have to figure out how to
do the work and to convince ourselves and the IESG that we have
consensus about it and that the IETF should accept that
consensus.

best, 
  john

So it seems to me that another one of those decisions the IETF
needs to figure out how to address is where

---

From: Nico Williams <[email-masked]>
Date: Thu, 14 Jun 2018 12:00:03 -0500

> non-ASCII elements or implications.

Eh, I'm happy to discuss specifics too.

As to process ideas, I think those have been discussed extensively
already -- enough that a BoF could be had, though that is up to the IAB.
I don't think enhanced I18N processes will produce any of the specific
solutions you or I have in mind as to the issues listed below.

> important issues" is, at least IMO, fairly obviously false.

I'm surprised to read this.  I don't think anyone has said or implied
that "if we solve confusion, there are no other important issues", nor
has anyone waved away normalization issues.

However, other issues we seem to handle fairly well already, with few
controversies.

As to the controversies...

Others have commented on the I18N of email addresses.  I like Viktor's
comment in that regard that we had to choose a cardinal rule to break,
and given that, we probably broke the wrong one.

As to IDNA... it seems that UTR#46 is non-grata here, though I think
UTR#46 is correct, and IDNA2008 is wrong where it differs from it.

But enough provocations :^)

> [...]

Did you mean that 3com caused that rule to be abandoned?

There are plenty of domainnames with labels that start with an ASCII
digit, or even all-digits labels.  E.g.,

 - 7up.nl
 - 911.gov

Anyways, wasn't this rule there to protect against old, lame parsers
that assume a string represents an IP address if it starts with a digit?
Such parsers would only look for ASCII digits...  No need to extend such
a rule to non-ASCII digits.

(I'm told there's a very large number of such domains.  Everything seems
to work, so any such rule should be dropped, if we still had it.)

> etc.  [...]

I don't think we (the IETF) should look for any confusability beyond
grapheme look (including some range of representative fonts in the
analysis).  However, registries should where culturally or legally
relevant.

> [...] Or how about U+110B, U+17E0, or U+-25CB?.   For IDNs (but

I'm not keen on IDNA updates forbidding any of these.  All of these
should be allowed as far as IDNA goes.

Instead, each registries should a) define policies, b) enforce them.

One obvious and general policy is to disallow any registration that is
confusable with an existing registration by a different entity.  To
enforce this the registry/registrars would check a proposed new domain
against all domains in that TLD looking for confusable similarity (more
on this in a bit) and if existing domains owned by other entities match,
then reject the registration.  Note that the confusable similarity check
algorithm might change over time.

Some registries might want to allow only one script, or a mix of one
script and ASCII (for historical reasons, say), or whatever.

Some ccTLD registries might only allow digits from a script associated
with a language or languages associated with the corresponding country.

Another policy might be that malicious use of confusable domains not
rejected as confusable at registration time... can cause the
registration to be yanked.  Satiric use may or may not get treated the
same as malicious use.

I really like this approach of the registries setting TLD-specific
policies.  The registries are the entities most likely to know what is
culturally appropriate for each TLD!

> lookup-time checks, so it is hard to rely on that.

Yes, we don't have an IETF police.  I don't know how we can create a
credible IETF police, and IMO we shouldn't want such a thing.

Sometimes we need to update RFCs to reflect reality.

We should find out whether any such domains that break our rules are
accidents (e.g., due to poor software) or intentional (e.g., the
registry thinks our rules are lame).

> equivalents elsewhere).  [...]

Note that there are, e.g., browser plugins for detecting and
highlighting confusables.

Registries protecting against confusables is certainly desirable, I
think, but as you note, they will sometimes fail.

Having a user agent UI look for and highlight confusables is... tricky,
for as we know many users won't notice.  User agents can also
distinguish between user-whitelisted/blessed and other domains.

But I don't think those two problems add up to having to ban confusables
in IDNA.

>                  [...].  Even if one uses artificial

The set of sets of confusable graphemes will have to be a) open as long
as Unicode is open to new characters, b) open as long as there are such
sets to be identified still.  We can expect both, humans and automatic
confusable detection (e.g., AI) to be able to add, over time, to the set
of sets of confusable graphemes.

I don't think there's anything wrong with having this set be incomplete
at any time, especially early on.  That's just life.  As this set grows,
the protection of users will increase.

> middle of a string from "m" unless one has other context.

Again, we can discover these things over time and add them to an
evolving corpus of confusable detection metadata and algorithms to be
applied by registries and/or user agents.

> right choice of type styles, small type, or a hangover, 

We could also consider dyslexia...

At least as to lesser confusables which might fall outside a confusable
corpus used for registry policies, the fact that courts and trademark
authorities can enforce their own policies is useful: it allows us to
have a smaller confusable corpus, though we could also just add these to
it too.

> the relationship between "color" and "co1or".

Yes.  Where shall we stop?  :)  This is precisely why we need per-TLD
policies.  We're not going to be able to make an agreeable one-size-
fits-all policy, but we can have a one-size-fits-all meta-policy.

> standards) that are not issues at all.

Some of these might not be malicious but satiric.  Satire account names
that are confusable (to some degree) with those of their object of
satire are very common, and social media sites have to navigate this
with care, allowing some and rejecting others.  Some degree of
subjectivity will be needed.

Some registries may choose to not allow satiric domains.  Others may
allow some but not others, with decisions made by humans (with or
without AI help).

One might want to define something of a "confusable distance" metric for
deciding whether two names are "too confusable" for comfort.

> judgments.  The question is what to do about it.  The two

Yes, there is way too much subjectivity and way too much local cultural
variation.

Have you noticed that people who learn some language in their late teens
or later years... tend to have a hard time distinguishing some sounds
from that language?  I suspect the same is true for grapheme
recognition.  Thus, rules that make sense in TLDs with a prevalence of
some small set of languages will surely need to vary from those of other
TLDs where other languages prevail.  The more languages are associated
with some TLD (e.g., com), the more restrictive the rules will have to
be.

> edge cases that should not occur in practice".   I think I've

I want to punt to the registries as much as possible.  That does not
fall in either of those two catergories you list.

> consensus about it and that the IETF should accept that

The best we can do, IMO, is:

 - cajole registries to develop and enforce policies regarding
   confusables

 - coax ICANN to get the registries to do that

 - develop a few generic policies from which registries could choose to
   apply -- i.e., do their work for them to some extent

   maybe one required generic policy (described above)

 - work with the Unicode Consortium to develop some such genric policies

 - develop software tools for the registries and registrars

 - work with the Unicode Consortium to develop such tools

I don't think we can or should want to do anything else.

Note that all such work will be hard to fund at the IETF as it really
involves doing other people's work.

Most IETF participants are funded to work on Internet protocols, though
they tend to stray a little.  But I doubt most IETF participants could
get much time funded for participating in the above listed activities.

This makes me think that the best we can really do is:

 - cajole ICANN, the registries, the registrars, and the UC

We probably can't develop too much in the way of generic policies, or
tools without having people specifically funded to work on them.
However, perhaps ISOC can arrange or obtain funding for those.

Also, the UC is the best-placed organization for putting together
standard confusable sets -- not the IETF.

All that said, if specific proposals are made, we *can* expect
participants to review them, even if they are not funded to produce such
proposals -- reviewing other people's work is understood to be a cost of
doing business at the IETF.

Nico
--

---

From: John C Klensin <[email-masked]>
Date: Fri, 15 Jun 2018 00:56:42 -0400

Nico,

I can't spend any more time on this thread other than finishing
a response to Adam's from Wednesday afternoon.  I've been
working on that intermittently when I've been able to find time
and energy.  It is clearly up to Peter and Pete, who have drawn
the short straw, but I think this notes and the forthcoming one
may be suggestive of some ways the BOF should probably be shaped.

I find your comments about UTR#46 very problematic.  I suspect
they may explain some of the other communications issues we have
been having and the problem is not disagreements about the
specifications.

There are fundamental architectural differences between UTR#46
and IDNA2008 (and PRECIS), differences that became much more
obvious as UTR#46 evolved from what was supposed to be a
transition document from IDNA2003 to IDNA2008 (plus or minus a
few character exceptions that may be important but are not
architectural except as somewhat different philosophies playing
out) to today's document that allows (or urges) several
characters that have never been valid under IDNA2008 or IDAN2003
and that pose possibly-significant issues as identifiers.

Without debating the reasons of who is right and who is wrong
right now (I believe the current version of UTR#46 demonstrates
a fundamental lack of understanding about the DNS and identifier
usage the interacts it, but the authors obviously disagree), the
IETF made its choice among those fundamental architectural
decisions when it adapted IDNA2008 and later carried basic parts
of its architecture into PRECIS.   Nothing prevents you from
believing that UTR#46 is the right model and that the IETF
should get with it.  Nothing prevents you from generating one or
more I-Ds that explain that we made the wrong decision and how
we should transition from where we are now to a model more like
UTR#46 and that specifies that model (IMO, UTR#46 is not
actually complete as an IDN specification, but YMMD).  But your
suggesting ways to adjust, clarify, etc., specifications based
on the RFC4690-IDNA2008-PRECIS model when those suggestions are
rooted in rejection of that model seems to me to be very close
to a DoS attack, even if an unintentional one.

best,
    john

--On Thursday, June 14, 2018 12:00 -0500 Nico Williams
<[email-masked]> wrote:

>...
> As to IDNA... it seems that UTR#46 is non-grata here, though I
> think UTR#46 is correct, and IDNA2008 is wrong where it
> differs from it.
>...