Subject: [precis] Precis Java Implementation

From: Christian Schudt <[email-masked]>
Date: Mon, 21 Dec 2015 18:29:26 +0100

Dear all,

I’ve been working on an open source Java implementation for the Precis framework since a few weeks and I'd finally like to share it with you:

https://bitbucket.org/sco0ter/precis

It supports the core Precis framework (RFC 7564), as well as the username and opaque string profiles (RFC 7613) and the recently published Nickname profile (RFC 7700).

If anybody of you find some time and is familiar with coding/Java, I’d be happy to receive some review and feedback from you, since I am planning to publish a first version to Maven Central soon.

There’s still one outstanding issue, which is the correct order of rules (RFC 7564 and the profiles specify them differently, as a consequence an exception is either thrown or not). See my source code comment here:

https://bitbucket.org/sco0ter/precis/src/ecd82b75f3611dcb37ee0fd8890bfaf02b5caf9f/src/main/java/rocks/xmpp/precis/PrecisProfile.java?at=master&fileviewer=file-view-default#PrecisProfile.java-684

(I also asked about it here in this mailing list)

Thanks for feedback.

— Christian

---

From: Sam Whited <[email-masked]>
Date: Mon, 21 Dec 2015 14:39:42 -0600

Christian:

> I’ve been working on an open source Java implementation for the Precis framework since a few weeks and I'd finally like to share it with you:

Awesome; looks good!

I'm curious, what made you decide to calculate the derived properties
on prepare instead of precomputing them? Have you done any
benchmarking of this? I'd love to see the performance difference
between this way and pre-computing a big trie or something of derived
properties. I'm working on a Go implementation at the moment for the
standard library (feedback welcome on the code review:
https://go-review.googlesource.com/#/c/17933/, or on Bitbucket:
https://bitbucket.org/mellium/golang-x-text/src/HEAD/unicode/precis/)
and I did it this way, but maybe it's not worth the larger memory
requirements.

If you did want to pre-compute the properties, the code generator I
wrote for my Go implementation could probably be adapted to output
Java instead of Go fairly easily and might save you some time.

Best,
Sam

-- 
Sam Whited
pub 4096R/54083AE104EA7AD3
https://blog.samwhited.com

---

From: Tom Worster <[email-masked]>
Date: Mon, 21 Dec 2015 16:05:05 -0500

To what extent do you think we could combine our efforts on unit tests? Standard (or at least shared) test vectors would really help, given how hard it is for programers to decode the RFC texts.

I’m willing to redo my tests (https://github.com/tom--/precis <https://github.com/tom--/precis>) to use some TBD machine-readable test spec. If others are interested, I’d contribute to authoring the tests.

Tom

> On Dec 21, 2015, at 12:29 PM, Christian Schudt <[email-masked]> wrote:
> 
> Dear all,
> 
> _______________________________________________
> precis mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/precis

---

From: Christian Schudt <[email-masked]>
Date: Mon, 21 Dec 2015 23:08:17 +0100

> properties.

Thanks for your review and suggestion!
I am not sure what you mean by "pre-computing them“. Most of these derived properties (methods) are based on the java.lang.Character class and are similarly designed as e.g. Character.isDigit(int cp).
If you mean having a huge code point table, like in your tables.go file: I think Java already has such tables internally.
What could be improved here, is that Character.getType(cp) could only be invoked once. I haven’t done any benchmark for this, but I don’t expect a significant performance benefit.

— Christian

---

From: Christian Schudt <[email-masked]>
Date: Mon, 21 Dec 2015 23:17:39 +0100

> Am 21.12.2015 um 22:05 schrieb Tom Worster <[email-masked]>:
> 

Well… if there’s some XML (or other language-independent format) defining input and expected output for different Precis profiles, I think this would be nice. But still not too sure, if it’s worth the effort. Often I rely on Java internal methods (e.g. for NFC normalization) and I guess they are already well-tested.

Nonetheless, if you have some test vectors, let me know. It surely increases interoperability.

— Christian

---

From: Tom Worster <[email-masked]>
Date: Mon, 21 Dec 2015 17:27:16 -0500

> Well… if there’s some XML (or other language-independent format) defining input and expected output for different Precis profiles, I think this would be nice. But still not too sure, if it’s worth…

I had the idea noticing that your unit tests have hard-coded test ins, expects and asserts, just like mine do. Alternatively I can copy from your tests to get better coverage in mine.

Tom

---

From: Sam Whited <[email-masked]>
Date: Mon, 21 Dec 2015 16:32:10 -0600

On Mon, Dec 21, 2015 at 3:05 PM, Tom Worster <[email-masked]> wrote:
> To what extent do you think we could combine our efforts on unit tests?
> Standard (or at least shared) test vectors would really help, given how hard
> it is for programers to decode the RFC texts.

I'd love to help with this; I was considering writing and submitting a
suite of test vectors anyways and would love to have buy-in and
interoperability with other implementations.

Best,
Sam

---

From: Sam Whited <[email-masked]>
Date: Mon, 21 Dec 2015 18:50:48 -0600

Another quick note:

In your case folding
(https://bitbucket.org/sco0ter/precis/src/ecd82b75f3611dcb37ee0fd8890bfaf02b5caf9f/src/main/java/rocks/xmpp/precis/PrecisProfile.java?at=master&fileviewer=file-view-default#PrecisProfile.java-511),
shouldn't the locale be left undefined? Or does it specify that US
should be used as the default somewhere?

Best,
Sam

---

From: Christian Schudt <[email-masked]>
Date: Tue, 22 Dec 2015 08:46:25 +0100

> Am 22.12.2015 um 01:50 schrieb Sam Whited <[email-masked]>:
> 
> Another quick note:
> 
> In your case folding

I followed the recommendation here:
http://www.w3.org/International/wiki/Case_folding#Recommendations_for_Case_Folding

If left undefined, the system’s default locale is used, which might not be the locale used for the „common case folding“, i.e. the one marked with a „C“ in ftp://ftp.unicode.org/Public/UCD/latest/ucd/CaseFolding.txt. Most prominent example seems to be the special case for the Turkish I.

---

From: Sam Whited <[email-masked]>
Date: Tue, 22 Dec 2015 20:04:39 -0600

> What could be improved here, is that Character.getType(cp) could only be invoked once. I haven’t done any benchmark for this, but I don’t expect a significant performance benefit.

Out of curiosity, I answered my own question here. I'm using Go, which
also has lots of Unicode tables in the standard library, so I
benchmarked running the algorithm (I modified it slightly from the
version in my generator to remove the NFKC step, which is very slow,
this way it more closely resembles your algorithm), and looking up a
value in the large pre-generated trie. I have no idea where the
bottlenecks / optimizations in Java would be, so these results may be
meaningless to you, but, at least in Go, the single Trie lookup was
much faster:

$ go test -bench . -benchmem
PASS
BenchmarkAsciiLookup-4          300000000                3.85 ns/op
        0 B/op          0 allocs/op
BenchmarkFullwidthLookup-4      200000000                9.21 ns/op
        0 B/op          0 allocs/op
BenchmarkAsciiCalculate-4       100000000               17.4 ns/op
        0 B/op          0 allocs/op
BenchmarkFullwidthCalculate-4   20000000                71.4 ns/op
        0 B/op          0 allocs/op
ok      _/home/sam/Projects/golang-x-text/unicode/precis        7.632s

Each test here is looking up or calculating the derived properties for
a single character (the ASCII tests are looking up 'u' and the Unicode
tests are looking up 'ｕ' [full width] which was chosen very
scientifically, I assure you), the second column is the number of
tests that were run until the timings reached equilibrium.

For the worst case, there's a pretty good speed difference, whether
that difference is worth pre-generating the data is another matter, of
course ☺

Best,
Sam

---

From: Christian Schudt <[email-masked]>
Date: Wed, 23 Dec 2015 10:16:20 +0100

If you are interested, in Java getting the character type (property) is even a bit slower, but it’s of course machine dependent:

java.lang.Character.getType(codepoint) is constantly 39 ns/op on 300000000 iterations on my machine, no matter if it’s an ascii or a fullwidth character. (Oracle JDK 8 SE).
Java uses huge Unicode arrays internally to lookup the code point, I doubt that there’s much room for optimization and even if, probably nobody cares when saving a few nano seconds.

— Christian

> Am 23.12.2015 um 03:04 schrieb Sam Whited <[email-masked]>:
>

---

From: Sam Whited <[email-masked]>
Date: Wed, 23 Dec 2015 09:02:18 -0600

> Java uses huge Unicode arrays internally to lookup the code point, I doubt that there’s much room for optimization and even if, probably nobody cares when saving a few nano seconds.

Thanks! I actually am really interested; I used to do a lot of work on
Conversations (Android XMPP client) and was the one who originally
wrote the JID class / stringprep stuff for it. I'm planning on
updating it to use your library at some point, and my work roster is
rather large (several thousand people) which makes things incredibly
slow with the current stringprep implementation; I'm hoping this will
be a lot faster!

Thanks again for all your hard work on this.

Best,
Sam