Subject: [Tzdist-bis] Truncation of TZif file via TZDIST

From: Ken Murchison <[email-masked]>
Date: Fri, 3 Aug 2018 19:07:53 -0400

Below is the subsection that I came up with for TZDIST truncation of 
TZif files.  The text that requires that there be at least one leap 
second record for application/tzif-leap was probably misguided when I 
added it.  I'm thinking of replacing it with text that says something 
like:  "Only leap second records whose correction value applies to 
timestamps represented in the file should be included.  This may be zero 
records if the last timestamp represented by the file occurs prior to 
Jan 1, 1972."

I don't see any need to discuss the TZDIST "expand" action as it doesn't 
modify TZif data in any way.  The response is JSON representing 
transition times that occur within the expansion range.  If backed by 
TZif files, a TZDIST server would simply include transition times that 
fall in the expansion range, and expand a TZ string up to the end of the 
range, if needed.

5.1.  Truncating TZif Files

    As described in Section 3.9 of [RFC7808], TZDIST server MAY truncate
    time zone data.  Truncation MUST only be performed on the 64-bit
    header and data of a TZif file.

    When truncating the start of a TZif file, the server MUST:

    o  Set the first time type (time type 0) to indicate local time right
       before the start point of truncation range.

    o  Set the first transition time (specified as UT) and corresponding
       time type to indicate local time for the timestamp at the start
       point of the truncation range.

    o  For "application/tzif-leap" format files, include the leap second
       record whose occurence is closest to, but not later than, the
       start point of the truncation range (if one exists).

    When truncating the end of a TZif file, the server MUST NOT include a
    TZ string if all of the timestamps that it represents occur on or
    after the end of the truncation range.  Note that TZif has no
    mechanism to indicate the end point of a truncation range (the
    timestamp at which the local time is not represented by the TZif
    file).

    All transition times that fall inside the truncation range MUST be
    the same as those represented by a corresponding untruncated TZif
    file.

    TZDIST clients SHOULD NOT use a truncated TZif file to interpret
    timestamps outside the truncation time range.

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Paul Eggert <[email-masked]>
Date: Fri, 3 Aug 2018 18:04:50 -0700

Ken Murchison wrote:
> Below is the subsection that I came up with for TZDIST truncation of TZif 
> files.  The text that requires that there be at least one leap second record for 
> application/tzif-leap was probably misguided when I added it.

The patch I sent in 
<https://www.ietf.org/mail-archive/web/tzdist-bis/current/msg00248.html> 
attempted to address this by saying that that requirement applies only to 
untruncated files. But you're right, it might be better simply to remove that 
requirement.

> I'm thinking of 
> replacing it with text that says something like:  "Only leap second records 
> whose correction value applies to timestamps represented in the file should be 
> included.  This may be zero records if the last timestamp represented by the 
> file occurs prior to Jan 1, 1972."

You can tighten the bound by changing "Jan 1, 1972" to "1972-06-30 23:59:60 UTC".

> I don't see any need to discuss the TZDIST "expand" action as it doesn't modify 
> TZif data in any way.  The response is JSON representing transition times that 
> occur within the expansion range.

Yes, sorry, I thought "expand" applied to iCalendar data. I didn't know it was JSON.

> If backed by TZif files, a TZDIST server 
> would simply include transition times that fall in the expansion range, and 
> expand a TZ string up to the end of the range, if needed.

Thanks, this sounds like information that would be useful to add to the 
"Implementation guidance" section.

>     header and data of a TZif file.

Why prohibit truncation of 32-bit data? The motivation is to save space, and 
truncating 32-bit data would save space too.

>       before the start point of truncation range. 

This requirement is redundant, as the spec already says "Local time for 
timestamps before the first transition is specified by the first time type (time 
type 0)." If there's some reason to restate the requirement here, that reason 
should be documented; otherwise this is confusing.

>       point of the truncation range.

Another redundant requirement; same comments as before.

>        start point of the truncation range (if one exists).

occurence -> occurrence

Have you tested TZif files like that on real programs, such as glibc? I have a 
sneaking suspicion they might not work due to their unusual format.

>     timestamp at which the local time is not represented by the TZif

This paragraph is starting to hint at the motivation for the preceding 
restrictions, but this is all pretty hard to follow, at least for me. Also, in 
ordinary usage TZif files do have a mechanism to indicate the truncation end 
point, namely a transition record (which can be a no-op).

 >     the same as those represented by a corresponding untruncated TZif

First, this is too strong. If the original set of transitions is not optimized 
(e.g., contains no-op transitions), it should be OK to optimize the truncated 
set. Second, it's too weak, as the wording should cover not only transition 
times, but also time types and leap seconds.

How about the following wording instead?

-----

    As described in Section 3.9 of [RFC7808], a TZDIST service MAY
    truncate time zone data.  A truncated TZif file is valid from its
    first and up to but not including its last 64-bit transition times,
    if present.

    When truncating the start of a TZif file, the service MUST supply
    in the 64-bit data a first transition timestamp that is the start
    point of the truncation range.  As with untruncated TZif files,
    time type 0 indicates local time immediately before the start
    point, and the time type of the first transition indicates local
    time thereafter.

    When truncating the end of a TZif file, the service MUST supply in
    the 64-bit data a last transition timestamp that is the end point
    of the truncation range, and MUST supply an empty TZ string.  As
    with untruncated TZif files with empty TZ strings, a truncated TZif
    file does not indicate local time after the last transition.

    All represented information that falls inside the truncation range
    MUST be the same as that represented by a corresponding untruncated
    TZif file.

---

From: Ken Murchison <[email-masked]>
Date: Fri, 3 Aug 2018 21:33:04 -0400

On 08/03/2018 09:04 PM, Paul Eggert wrote:
>> added it.
> attempted to address this by saying that that requirement applies only 
> to untruncated files. But you're right, it might be better simply to 
> remove that requirement.

Yeah, I added the MUST NOT be zero to indicate that tzif-leap is 
supposed to include leap second records, but a TZ that was only in 
existence prior to 1972 (e.g. somewhere in Antarctica?) won't have any 
leap second records.

>
>> I'm thinking of replacing it with text that says something like:  
>> "Only leap second records whose correction value applies to 
>> timestamps represented in the file should be included.  This may be 
>> zero records if the last timestamp represented by the file occurs 
>> prior to Jan 1, 1972."
>
>> I don't see any need to discuss the TZDIST "expand" action as it 
>> doesn't modify TZif data in any way.  The response is JSON 
>> representing transition times that occur within the expansion range.
>
>> If backed by TZif files, a TZDIST server would simply include 
>> transition times that fall in the expansion range, and expand a TZ 
>> string up to the end of the range, if needed.
>
> Thanks, this sounds like information that would be useful to add to 
> the "Implementation guidance" section.
>
> Why prohibit truncation of 32-bit data? The motivation is to save 
> space, and truncating 32-bit data would save space too.

Looking back on the thread about truncation this morning, I thought 
someone suggested that we only deal with 64-bit data because 32-bit data 
doesn't have transitions prior to 1970.  Maybe I got myself confused.

> This requirement is redundant, as the spec already says "Local time 
> for timestamps before the first transition is specified by the first 
> time type (time type 0)." If there's some reason to restate the 
> requirement here, that reason should be documented; otherwise this is 
> confusing.

No specific reason.  I was trying to replicate similar text from RFC 
7808 where a "tombstone" VTIMZONE is constructed at the start of the 
truncation range.

> Another redundant requirement; same comments as before.

Yes, but if the start of the truncation range falls between two actual 
TZ transitions, the truncated file needs to add a no-op (or tombstone) 
transition so that the consumer of the data knows from what timestamp 
the data is valid.

> Have you tested TZif files like that on real programs, such as glibc? 
> I have a sneaking suspicion they might not work due to their unusual 
> format.

I have not, this is just a theoretical change.  I'd be happy to not 
bother truncating leap second records at all before the start of the 
range.  It doesn't add that much data.

> restrictions, but this is all pretty hard to follow, at least for me. 
> Also, in ordinary usage TZif files do have a mechanism to indicate the 
> truncation end point, namely a transition record (which can be a no-op).

Hmm.  Can you elaborate on your last point?  How would I indicate that a 
truncated TZif for America/Los_Angeles is only valid up to 2019-12-31?

> First, this is too strong. If the original set of transitions is not 
> optimized (e.g., contains no-op transitions), it should be OK to 
> optimize the truncated set. Second, it's too weak, as the wording 
> should cover not only transition times, but also time types and leap 
> seconds.

I thought I had encompassed wording in  your patch.  I guess not.

> How about the following wording instead?

In general, I like it.  Comments inline.

>    first and up to but not including its last 64-bit transition times,

Are we not worrying about 32-bit data, despite your query above?

>    file does not indicate local time after the last transition.

I thought you didn't want to go with an empty string because a TZ string 
could be used to cover a bunch of valid transitions compactly.  
Otherwise, I'm fine with this.

>    timestamps outside the truncation time range.

Do we not want to mention leap second records?  I'm fine including all 
of them, or just trimming those that occur after the end of the 
truncation range, which should be safe, correct?

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Ken Murchison <[email-masked]>
Date: Fri, 3 Aug 2018 21:53:04 -0400

> range.  It doesn't add that much data.

And not truncating before the range also avoid the conflict with your 
text that states that the correction of the first leap second record 
MUST be 1 or -1.

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Paul Eggert <[email-masked]>
Date: Fri, 3 Aug 2018 18:55:03 -0700

Ken Murchison wrote:

> Looking back on the thread about truncation this morning, I thought someone 
> suggested that we only deal with 64-bit data because 32-bit data doesn't have 
> transitions prior to 1970.

It's fine if the truncation spec places new constraints only on the 64-bit data, 
which I think is probably what was proposed. However, the truncation spec 
shouldn't say that the 32-bit data must be the same as the untruncated version; 
that would prohibit reasonable truncation. The truncation spec should be silent 
about what happens to the 32-bit data, to give the server freedom to truncate it 
or perhaps not (or even to eliminate it).

> if the start of the truncation range falls between two actual TZ 
> transitions, the truncated file needs to add a no-op (or tombstone) transition 
> so that the consumer of the data knows from what timestamp the data is valid.

Yes, that's fine, and follows from the wording I just proposed. Of course it 
could be worded more explicitly that this transition may not be in the 
untruncated version.

> I have not, this is just a theoretical change.  I'd be happy to not bother 
> truncating leap second records at all before the start of the range.  It doesn't 
> add that much data.

It might be better to not mention leap seconds, then. Just make it part of the 
general rule that all time info in the range must be preserved.

> truncated TZif for America/Los_Angeles is only valid up to 2019-12-31?

You append a trailing timestamp record equal to the end point.

> Are we not worrying about 32-bit data, despite your query above?

We are not specifically worrying about it here. 32-bit data in the truncated 
file must follow the same rules that 32-bit data must follow in any file: they 
have to be a subset of the 64-bit data, that's all.

> I thought you didn't want to go with an empty string because a TZ string could 
> be used to cover a bunch of valid transitions compactly.

Yes, that's what I wanted, but I'm giving that up. It seems to be more important 
to capture the end time. In practice, the compactness won't matter; nobody is 
going to be truncating to the year 9999999.

> Do we not want to mention leap second records?  I'm fine including all of them, 
> or just trimming those that occur after the end of the truncation range, which 
> should be safe, correct?

You should be able to omit all leap second records with transitions outside the 
truncation range. You may need to insert a leap second record to cover the first 
timestamps in the truncation range. It would be OK to suggest what was in your 
spec (clone the leap second record closest to, but not including the start of 
the truncation range), but this should be a suggestion, not a requirement. All 
that should be required is that we capture time info within the range accurately.

---

From: Paul Eggert <[email-masked]>
Date: Fri, 3 Aug 2018 18:59:32 -0700

Ken Murchison wrote:
> 
> And not truncating before the range also avoid the conflict with your text that 
> states that the correction of the first leap second record MUST be 1 or -1.

That shouldn't be a problem, as the conflict was resolved in the patch I 
proposed here:

https://www.ietf.org/mail-archive/web/tzdist-bis/current/msg00248.html

the first hunk of which states that it's OK to truncate leap second records 
before the start of the truncation period, so long as you represent timestamps 
within the period correctly.

---

From: Ken Murchison <[email-masked]>
Date: Fri, 3 Aug 2018 22:05:47 -0400

On 08/03/2018 09:59 PM, Paul Eggert wrote:
> That shouldn't be a problem, as the conflict was resolved in the patch 
> I proposed here:
>
> the first hunk of which states that it's OK to truncate leap second 
> records before the start of the truncation period, so long as you 
> represent timestamps within the period correctly.

But as you asked earlier, has this been tested in the field (a first 
leap second record with a correction of > 1)?  If not, I think I'm 
leaning towards just leaving leap second records alone.

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Paul Eggert <[email-masked]>
Date: Fri, 3 Aug 2018 23:51:32 -0700

Ken Murchison wrote:
> I think I'm leaning towards just leaving leap second records alone.

If we don't have implementation experience, it's a mistake for the spec to say 
anything, even for the spec to say that the leap second records should stay the 
same. The spec should just be silent, or should just note the problem.

---

From: Paul Eggert <[email-masked]>
Date: Wed, 8 Aug 2018 14:04:38 -0700

Ken Murchison wrote:
> 
> - Since a legacy time zone that ended prior to 1972 will not have any leap 
> second records, do we want to relax the MUST NOT be zero language for 
> application/tzif-leap ?

I would, yes. In the draft's discussion of the tzif-leap media type's leapcnt 
value, let's change "MUST be non-zero" to "MUST be non-zero unless the TZif file 
has been truncated with end time no greater than the first leap second".

> - Should we be using normative SHOULDs in the Appendix?

I would leave it alone. People don't normally look for normative text in 
appendices, and the Appendix is intended to be informative, not normative. 
(Perhaps the Appendix should say so explicitly? I'm not sure what the style is 
for RFCs.)

I found 3 occurrences of "should" in the appendix. The first two are subsumed by 
Section 4's "implementations SHOULD ignore the 32-bit header and data block" and 
so would not add anything if normative, and the last occurrence is in 
"Developers ... should keep this in mind" and so couldn't be normative anyway.

---

From: Eliot Lear <[email-masked]>
Date: Wed, 8 Aug 2018 23:08:06 +0200

On 08.08.18 23:04, Paul Eggert wrote:
> Ken Murchison wrote:
>>
>> - Since a legacy time zone that ended prior to 1972 will not have any
>> leap second records, do we want to relax the MUST NOT be zero
>> language for application/tzif-leap ?
>
> I would, yes. In the draft's discussion of the tzif-leap media type's
> leapcnt value, let's change "MUST be non-zero" to "MUST be non-zero
> unless the TZif file has been truncated with end time no greater than
> the first leap second".
>
> I would leave it alone. People don't normally look for normative text
> in appendices, and the Appendix is intended to be informative, not
> normative. (Perhaps the Appendix should say so explicitly? I'm not
> sure what the style is for RFCs.)
Bingo.

>
> I found 3 occurrences of "should" in the appendix. The first two are
> subsumed by Section 4's "implementations SHOULD ignore the 32-bit
> header and data block" and so would not add anything if normative, and
> the last occurrence is in "Developers ... should keep this in mind"
> and so couldn't be normative anyway.
>

Yeah, it's bad form to leave normative stuff in the Appendix.  HOWEVER,
if NOT doing so would cause an interoperability problem, then then we
should talk.

Eliot

---

From: Ken Murchison <[email-masked]>
Date: Wed, 8 Aug 2018 17:25:48 -0400

>> unless the TZif file has been truncated with end time no greater than

Will do.

>> sure what the style is for RFCs.)

Eliot, are you saying we should explicitly state that the Appendix is 
only informative?

> if NOT doing so would cause an interoperability problem, then then we

I believe that the Appendix is just informative, and Paul seems to agree.

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Ken Murchison <[email-masked]>
Date: Wed, 8 Aug 2018 17:33:37 -0400

> unless the TZif file has been truncated with end time no greater than 

Hmm.  But the TZif file doesn't have to be truncated for leapcnt to be 
zero.  But I can't come up with good wording that encompasses that.  
Using "last transition time" doesn't help because there may be a TZ 
string which extends the transitions to post-1972.

Any ideas?

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Eliot Lear <[email-masked]>
Date: Wed, 8 Aug 2018 23:37:57 +0200

Hi,

> Eliot, are you saying we should explicitly state that the Appendix is

No.  It is assumed to be so, but there is no formal prohibition of
adding normative language to the appendix.  It's just that people might
miss it.

---

From: Paul Eggert <[email-masked]>
Date: Wed, 8 Aug 2018 14:40:51 -0700

Ken Murchison wrote:
> But the TZif file doesn't have to be truncated for leapcnt to be zero.  But I 
> can't come up with good wording that encompasses that. Using "last transition 
> time" doesn't help because there may be a TZ string which extends the 
> transitions to post-1972.

If we're having any difficulty in this relatively unimportant area then I 
suggest just removing the "MUST be non-zero" requirement for tzif-leap. That's 
simpler and is good enough.

---

From: Ken Murchison <[email-masked]>
Date: Wed, 8 Aug 2018 17:47:25 -0400

On 08/08/2018 05:40 PM, Paul Eggert wrote:
> Ken Murchison wrote:
>> But the TZif file doesn't have to be truncated for leapcnt to be 
> If we're having any difficulty in this relatively unimportant area 
> then I suggest just removing the "MUST be non-zero" requirement for 
> tzif-leap. That's simpler and is good enough.

One last shot.  How does this work?

"MUST be non-zero provided that at least one leap second correction 
occurs prior to the last valid transition time"

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Paul Eggert <[email-masked]>
Date: Wed, 8 Aug 2018 15:08:36 -0700

Ken Murchison wrote:
> "MUST be non-zero provided that at least one leap second correction occurs prior 
> to the last valid transition time"

But that's a tautology. A leap second correction cannot occur without leapcnt 
being nonzero. So I don't see how this fixes anything.

Part of the issue here, which perhaps needs to be said explicitly, is that the 
TZif format makes no guarantees about truth. In particular, the leap second 
records in a TZif file need not agree with the official leap seconds published 
by the International Earth Rotation and Reference Systems Service. A tzif-leaps 
file can (falsely) state that no leap seconds have ever existed, and still 
conform to the format.

---

From: Ken Murchison <[email-masked]>
Date: Wed, 8 Aug 2018 18:10:22 -0400

On 08/08/2018 06:08 PM, Paul Eggert wrote:
> But that's a tautology. A leap second correction cannot occur without 
> leapcnt being nonzero. So I don't see how this fixes anything.
>
> Part of the issue here, which perhaps needs to be said explicitly, is 
> that the TZif format makes no guarantees about truth. In particular, 
> the leap second records in a TZif file need not agree with the 
> official leap seconds published by the International Earth Rotation 
> and Reference Systems Service. A tzif-leaps file can (falsely) state 
> that no leap seconds have ever existed, and still conform to the format.

Well, the same can be said about the transition times, correct?

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Paul Eggert <[email-masked]>
Date: Wed, 8 Aug 2018 15:45:42 -0700

Ken Murchison wrote:
>> by the International Earth Rotation and Reference Systems Service. A 
>> tzif-leaps file can (falsely) state that no leap seconds have ever existed, 
>> and still conform to the format.
> 

Of course. None of the data in a TZif file needs to be accurate.

---

From: Eliot Lear <[email-masked]>
Date: Thu, 9 Aug 2018 08:33:38 +0200

On 09.08.18 00:45, Paul Eggert wrote:
> Ken Murchison wrote:
>>> Part of the issue here, which perhaps needs to be said explicitly,
>>> is that the TZif format makes no guarantees about truth. In
>>> particular, the leap second records in a TZif file need not agree
>>> with the official leap seconds published by the International Earth
>>> Rotation and Reference Systems Service. A tzif-leaps file can
>>> (falsely) state that no leap seconds have ever existed, and still
>>> conform to the format.
>>

I don't think we need to mention that because it amounts to, “I read it
on the net.  Is it true?”  Maybe.  So what?  The protocol, just like
many IETF protocols is about conveyance of the information, not the
payload's accuracy.

Eliot

---

From: Ken Murchison <[email-masked]>
Date: Thu, 9 Aug 2018 06:02:05 -0400

> many IETF protocols is about conveyance of the information, not the

Right, I had no intention of mentioning this.  I think we're pretty 
close to done.  I gave it another full read last night and didn't find 
any issues.

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC