Subject: RE: [Fwd: Re: Kerberos Revisions and UTF8]

From: Jeffrey Altman <[email-masked]>
Date: Thu, 9 Aug 2001 9:40:52 EDT

> There was never mention of ASCII either in the RFC-1510 - the definition
> is GeneralString.  Lets focus on the correct solution that gives a
> migration path. The Kerberos protocol must deal with non-Latin-1
> character sets - that is a reality.

John, I think you miss the point.  The definition in RFC-1510 is
"GeneralString" and what we all missed was the fact that
"GeneralString" is an ISO-2022 quantity not a "C" String.  The
restriction to using ASCII is that ASCII is the only data that can be
stored in a "GeneralString" without including ISO-2022 escapes to
specify which character-set is being used.

I completely agree that Kerberos must have support for non-ASCII
characters.  The problem at hand is two fold.  One, how do we describe
the current restrictions in such a way that it is compliant with the 
definition of "GeneralString" and so that people do not shoot
themselves in the foot anymore than we already have.  Two, how do we
change the representation of strings in such as way as to maintain
backwards compatibility with existing implementations which allowing
support for Unicode to go forward.

What we have here is a really nasty protocol problem because the
implementations of Kerberos and the authors of RFC-1510 did not
understand ASN.1 well enough to realize that "GeneralString" did not
describe the data type that was desired.

My suggestion for specifying a restriction of ASCII in Revisions is
simply to prevent any more damage from being done.  This problem is
not limited to Microsoft.  Every implementation of Kerberos I have
ever seen allows the use of arbitrary locally defined 8-bit character
sets to be used.  It just so happens that the locally defined 8-bit
character set for Windows is "UTF-8" instead of Latin1, Latin2, CP437,
CP850, ...  None of these implementations keep track of the character
set in use.  Nor do they use the proper ISO-2022 escapes to indicate
the character set in the GeneralString fields.  

I also am unaware of any Kerberos implementation that can handle the
receipt of ISO-2022 escapes to do the right thing.  Principals are not
stored with ISO-2022 escapes and their removal wouldn't make any sense
anyways.

The problem is we can't simply go and say to everyone, starting Sept 1
you must add ISO-2022 escapes to the principal database and upgrade
all of the clients and servers.  

Another data point to be used to argue for the ASCII restriction is
the informal poll that we performed after the Pittsburgh meeting.  In
that poll I asked everyone to go back home and search their KDC
databases for any principal names that included non-ASCII characters.
The answer that was returned was that none of the members of the
working group or of the MIT krbdev subscribers that responded used
anything other than ASCII.

I might also add that this issue (for better or worse) came to the
forefront at the Pittsburgh meeting because we were attempting to
figure out what Microsoft was doing to support Unicode in principal
names.  It was then that we discovered that the principal names
contained unescaped UTF8 strings.  It also conincided with Tom
studying ASN.1 specs as part of a review of the incompatibilities
between Revisions and RFC-1510.  So it was a freak accident that we
even came across this problem.

I'm not going to say that Microsoft caused this problem because the
reality is that none of us in the working group were aware of the
semantics of "GeneralString", but Microsoft does appear to have the one
and only implementation which is in violation of the spec.  If
by some chance the Microsoft engineers reading RFC-1510 understood the
definition of ASN.1 GeneralString and used the appropriate ISO-2002
escapes we would not be struggling with this.  We would simply be
going back and fixing all of the other implementations to either
enforce the ASCII restriction (since ASCII does not require escapes)
or add support for UTF-8 with escapes.

It is just an unfortunate situation that we are all in. 

- Jeff

 Jeffrey Altman * Sr.Software Designer      C-Kermit 8.0 Beta available
 The Kermit Project @ Columbia University   includes Secure Telnet and FTP
 http://www.kermit-project.org/             using Kerberos, SRP, and 
 [email-masked]          OpenSSL.  SSH soon to follow.

---

From: Tom Yu <[email-masked]>
Date: 09 Aug 2001 10:56:48 -0400

>>>>> "jaltman" == Jeffrey Altman <[email-masked]> writes:

>> There was never mention of ASCII either in the RFC-1510 - the
>> definition is GeneralString.  Lets focus on the correct solution
>> that gives a migration path. The Kerberos protocol must deal with
>> non-Latin-1 character sets - that is a reality.

jaltman> John, I think you miss the point.  The definition in RFC-1510
jaltman> is "GeneralString" and what we all missed was the fact that
jaltman> "GeneralString" is an ISO-2022 quantity not a "C" String.
jaltman> The restriction to using ASCII is that ASCII is the only data
jaltman> that can be stored in a "GeneralString" without including
jaltman> ISO-2022 escapes to specify which character-set is being
jaltman> used.

In addition, there are bugs in the ASN.1 spec itself with respect to
encoding of GeneralString, i.e. you can't encode a 96-character G set
at all in DER, which means being unable to send the right hand half of
Latin-1!  Of course, the point about ASCII being the only characters
transmissible without the use of escape sequences is quite valid.

jaltman> I completely agree that Kerberos must have support for
jaltman> non-ASCII characters.  The problem at hand is two fold.  One,
jaltman> how do we describe the current restrictions in such a way
jaltman> that it is compliant with the definition of "GeneralString"
jaltman> and so that people do not shoot themselves in the foot
jaltman> anymore than we already have.

This has been done by applying a constraint to GeneralString in the
definition of KerberosString, so that conformant implementations will
effectively only emit ASCII for old-style messages.

jaltman> Two, how do we change the representation of strings in such
jaltman> as way as to maintain backwards compatibility with existing
jaltman> implementations which allowing support for Unicode to go
jaltman> forward.

We are working on ways to do this.

jaltman> What we have here is a really nasty protocol problem because
jaltman> the implementations of Kerberos and the authors of RFC-1510
jaltman> did not understand ASN.1 well enough to realize that
jaltman> "GeneralString" did not describe the data type that was
jaltman> desired.

Unfortunately, this is quite true, it seems.  Someone or other thought
that GeneralString was just an octet string, with no particular other
semantics.

jaltman> My suggestion for specifying a restriction of ASCII in
jaltman> Revisions is simply to prevent any more damage from being
jaltman> done.

Of course, if we do as I suggested by constraining the use of
VisibleString, then it should be reasonably easy.

jaltman> None of these implementations keep track of the character set
jaltman> in use.  Nor do they use the proper ISO-2022 escapes to
jaltman> indicate the character set in the GeneralString fields.

Yeah.  The really big problem here is likely if some implementation
has a default character encoding that has something other than ASCII
designated/invoked into G0/GL by default, since then 7-bit characters
won't be ASCII...

jaltman> I also am unaware of any Kerberos implementation that can
jaltman> handle the receipt of ISO-2022 escapes to do the right thing.
jaltman> Principals are not stored with ISO-2022 escapes and their
jaltman> removal wouldn't make any sense anyways.

You're probably correct here.

jaltman> The problem is we can't simply go and say to everyone,
jaltman> starting Sept 1 you must add ISO-2022 escapes to the
jaltman> principal database and upgrade all of the clients and
jaltman> servers.

We are not proposing a flag day; that would very likely not pass IESG
muster. ;-)

jaltman> I might also add that this issue (for better or worse) came
jaltman> to the forefront at the Pittsburgh meeting because we were
jaltman> attempting to figure out what Microsoft was doing to support
jaltman> Unicode in principal names.  It was then that we discovered
jaltman> that the principal names contained unescaped UTF8 strings.
jaltman> It also conincided with Tom studying ASN.1 specs as part of a
jaltman> review of the incompatibilities between Revisions and
jaltman> RFC-1510.  So it was a freak accident that we even came
jaltman> across this problem.

It would have come up eventually anyway, perhaps at an even more
inconvenient time.

jaltman> I'm not going to say that Microsoft caused this problem
jaltman> because the reality is that none of us in the working group
jaltman> were aware of the semantics of "GeneralString", but Microsoft
jaltman> does appear to have the one and only implementation which is
jaltman> in violation of the spec.  If by some chance the Microsoft
jaltman> engineers reading RFC-1510 understood the definition of ASN.1
jaltman> GeneralString and used the appropriate ISO-2002 escapes we
jaltman> would not be struggling with this.  We would simply be going
jaltman> back and fixing all of the other implementations to either
jaltman> enforce the ASCII restriction (since ASCII does not require
jaltman> escapes) or add support for UTF-8 with escapes.

It is my impression that Microsoft's implementation sends unescaped
UTF-8 tagged as GeneralString.  Of course, even with escape sequences,
UTF-8 isn't valid in a GeneralString, since it's a separate encoding,
not a G set.

Keep in mind that the 7-bit piece of UTF-8 is ASCII, so even Microsoft
implementations are compliant if they don't send any chracters outside
of ASCII.

jaltman> It is just an unfortunate situation that we are all in. 

Agreed.

---Tom

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 9 Aug 2001 11:18:12 -0400

> implementations are compliant if they don't send any chracters outside

I just tried creating a principal, in an MIT KDCed realm, with a
Latin-1, eigth bit set, character in its name. Guess what: kadmind
allowed it and the KDC happily served tickets to it.

So the MIT and Microsoft implementations of Kerberos are both
non-compliant depending on their *use*, though both are also compliant
if their customers refrain from using anything but ASCII for their
principal's names. In fact, I bet that if an application linked with
MIT's libkrb5 treats C char strings as UTF-8 then MIT's implementation
could be said to be non-compliant in the same way as Microsoft's
because, at the heart of it, and but for instance separator quoting
issues, MIT's implementation treats GeneralString as an octet string and
only really does equality comparisons (yes, and name ACL matching) --
it's upto the apps that display princ names to humans and/or accept
human-typed princ names to impart meaning onto non-ASCII characters in
those princ names, and it just so happens that all such apps from MS use
UTF-8 and all such apps from MIT leave that to environmental factors.

> jaltman> It is just an unfortunate situation that we are all in. 

Nico
--
. 
-DISCLAIMER: an automatically appended disclaimer may follow. By posting-
-to a public e-mail mailing list I hereby grant permission to distribute-
-and copy this message.-

Visit our website at http://www.ubswarburg.com

This message contains confidential information and is intended only 
for the individual named.  If you are not the named addressee you 
should not disseminate, distribute or copy this e-mail.  Please 
notify the sender immediately by e-mail if you have received this 
e-mail by mistake and delete this e-mail from your system.

E-mail transmission cannot be guaranteed to be secure or error-free 
as information could be intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain viruses.  The sender therefore 
does not accept liability for any errors or omissions in the contents 
of this message which arise as a result of e-mail transmission.  If 
verification is required please request a hard-copy version.  This 
message is provided for informational purposes and should not be 
construed as a solicitation or offer to buy or sell any securities or 
related financial instruments.