Subject: Re: [OPSAWG] adopting draft-gharris-opsawg-pcap and draft-tuexen-opsawg-pcapng

From: Guy Harris <[email-masked]>
Date: Sun, 4 Jul 2021 16:49:59 -0700

On Jun 29, 2021, at 5:29 PM, Michael Richardson <[email-masked]> wrote:

> There was many comments about how we should have used CBOR for PCAPNG, and I
> would agree that in hindsight, it would be good.  New section types for
> PCAPNG could well be encoded in CBOR.  I also advocated that for
> draft-ietf-quic-qlog-h3-events,draft-ietf-quic-qlog-main-schema,
> draft-ietf-quic-qlog-quic-events, but they didn't go that way. (At least, not
> yet?).  I haven't time to do QUIC stuff, but if there is interest, and I'll
> read the documents above, and see if I can come up with a proposal.
> 
> The proposal is to adopt draft-gharris-opsawg-pcap as an Informational
> document.  This documents the ~30 year old pcap format used by tcpdump,
> wireshark, etc.   Almost all of IPv6, DNSSEC, DNS extensions, etc. research
> done by many researchers, including for instance, the
> https://www.caida.org/projects/network_telescope/ have used pcap files as
> their capture format.
> We need to do this as *WG* and can not do this as ISE, because the pcap
> document establishes the critical LinkType registry.  One of the exchanges
> above is about how to "load" this rather large legacy into IANA.

Do the link-layer types belong in a pcap I-D/RFC, given that they're used in both pcap and pcapng, and given that there are a lot of them, so that the link-layer types information might swap the pcap format information if they're in a single spec?

We might want to have a pcap spec, a pcapng spec, and another document that gives detailed descriptions - as detailed as what's in

	https://www.tcpdump.org/linktypes.html

of the existing types.  The registry could point to the I-D/RFC for descriptions of the types; new types would get new RFCs.

> pcapng would be an IETF controlled document, the "pcap 2.0", but we can't
> really do as many changes as we might like.
> (I'd sure like to name it pcap2.0, as I hate the whole "Next Generation"
> moniker, but I don't know if that would fly at this late stage).
> 
> *MAYBE* PCAPNG should also be Informational, given that we can't really mess
> with it too much.
> 
> *MAYBE* PCAP2.0 should be just the structure as IETF Standards Track, with
> the section structure which is in PCAPNG (which is not changeable) should be
> an Informational document.  This involves more documents, but no additional
> text.

So, with that proposal for pcapng/pcap 2.0 would there be, as separate documents:

	the overall structure, currently covered by sections 1-4 and 6-7 of the spec, and not very changeable;

	the definitions of blocks and block-specific options, currently covered by sections 4 and 5, with the blocks currently specified not being changeable other than using reserved fields and adding new options, but with adding new block types being allowed;

and a registry for block and block-specific option types, with the registry pointing to the second document or to any new documents specifying new block types?

---

From: Michael Richardson <[email-masked]>
Date: Sun, 04 Jul 2021 21:18:59 -0400

> Do the link-layer types belong in a pcap I-D/RFC, given that they're
    > used in both pcap and pcapng, and given that there are a lot of them,
    > so that the link-layer types information might swap the pcap format
    > information if they're in a single spec?

Some document which would be the result of IETF Consensus needs to create the
link-layer repository, and it needs to initialize the database.

I think that you mean "swamp" where you write "swap"?
I don't think it's a problem that they are in the same document.
The WG could disagree, and tell us to split the document into two.

    > We might want to have a pcap spec, a pcapng spec, and another document
    > that gives detailed descriptions - as detailed as what's in

The IANA Considerations rules for the repository is "Specification Required",
but that doesn't mean that the specification has to be in the document we
submit.  It can be any stable document.  That includes an RFC, but isn't
restricted to that.
It is perfectly fine to point at linktypes.html from the IANA Registry.
There are presently more details in the table than I originally had intended
to include.

    > spec, and not very changeable;

Yes.  Call this "pcap2"

    > the definitions of blocks and block-specific options, currently covered
    > by sections 4 and 5, with the blocks currently specified not being
    > changeable other than using reserved fields and adding new options, but
    > with adding new block types being allowed;

I would include, in "pcap2" all the current blocks that correspond to capture
types present in pcap1.
Then, I'd put all the things which aren't about capture of IP packets
(e.g. systemd, USB, ...) into another (informational) document.
Or perhaps just leave them in a web page, having pointed at them in pcap2's
IANA Considerations.

    > and a registry for block and block-specific option types, with the
    > registry pointing to the second document or to any new documents
    > specifying new block types?

Well, we can't make forward pointers to documents that don't exist yet.
So, the forward pointers wind up in the IANA registry.

--
Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide

---

From: Guy Harris <[email-masked]>
Date: Sun, 4 Jul 2021 18:22:06 -0700

On Jun 30, 2021, at 12:43 AM, Joe Clarke (jclarke) <jclarke=[email-masked]> wrote:

> Given how systemd is prone to considerable change, I'd think it would be
> best to leave that out.

"systemd" is at least two different things.  It started out as a replacement for the various versions of the UN*X "init" process in Linux, launching daemons on demand similarly to Darwin's launchd.  It then grew into a bigger project providing a bunch of other system daemons, written to assume that the systemd process handled the launching of other daemons.

One of the other daemons is journald, which is a syslogd replacement:

	https://www.freedesktop.org/software/systemd/man/systemd-journald.service.html

The "systemd" block in pcapng is called the "systemd Journal Export Block"; it doesn't cover systemd as a whole, it just covers the format that journald uses to export entries from its log file.  The description of the export block points to the description of the export format:

	https://www.freedesktop.org/wiki/Software/systemd/export/

I can't state for certain that this format is intended to be extensible in a compatible fashion, so that it can be *extended* but won't be *incompatibly changed*, but that might end up being the case.

Given that the description of the systemd Journal Export Block largely punts the description of the format to the freedesktop.org page about the format, describing only a "wrapper" for a journal entry in the export format, it doesn't bother me *too* much to put it into the spec.

By the way, a few registries under

	https://www.iana.org/protocols

that I've looked out have some entries that point to RFCs and other entries that just have an email address for a contact.

Do any of those RFCs point to an external specification for most of the information about the item being discussed, similarly to what's the case for the systemd Journal Export Block?

Are there any registries that have "less than an RFC and more than a mailto: URL", i.e. that say more than just "ask this person, hopefully they're still around, still have that email address, and still remember what this entry in the registry was all about"?

> That said, the pull request language in -03 has
> me curious.  If someone were to raise a PR against this doc, that would
> be outside the IETF process.  Is the intent then that this would leave
> to a bis or other draft to update this document?  And if so, is that
> desirable, or would it be better to have this live in GitHub outside of
> the IETF process altogether (like the semver spec does)?

Is "the server spec" referring to the specifications that draft-claise-semver-00:

	https://tools.ietf.org/id/draft-claise-semver-00.html

is discussing?  That sounds as if it's handling the part of the process up to the submission of an I-D, and that the regular I-D -> RFC process takes effect once the spec is submitted as an I-D.  That document itself wasn't outside the IETF process, as it was an I-D (now expired), and the prices it described appeared to be for documents that eventually become handled by the IETF process.

---

From: Guy Harris <[email-masked]>
Date: Sun, 4 Jul 2021 18:39:27 -0700

> link-layer repository, and it needs to initialize the database.

That document doesn't have to be the same document that describes the structure of pcap files, just as there could be separate documents for the structure of pcapng/pcap 2.0 file and for the initial set of block types for pcapng/pcap 2.0.

> I think that you mean "swamp" where you write "swap"?

Yes.

> There are presently more details in the table than I originally had intended

Above, you say

> link-layer repository, and it needs to initialize the database.

Does "initialize the database" mean "provide the initial set of entries in the database"?  If so, does it have to provide anything more than "see the entry for LINKTYPE_XYZZY in www.tcpdump.org/linktypes.html for the XYZZY linktype"?  Or, with an extra layer of compression, just have a list of LINKTYPE_XYZZY values with a single "for each linktype in this list, see the entry for that Linotype in www.tcpdump.org/linktypes.html"?

> I would include, in "pcap2" all the current blocks that correspond to capture

("pcap1" presumably being the current pcap file format.)

What do you mean by "capture types"?

If you're referring to linktypes, different linktypes don't have different block types.  The only current (as opposed to deprecated) block types corresponding to what appears in pcap files are the Enhanced Packet Block (EPB) and the Simple Packet Block (SPB), and the latter is arguably different from a pcap record given that it doesn't have a time stamp - "editcap -T pcap pcap-file.pcap pcapng-file.pcapng" will write out one Section Header Block, one Interface Description Block, and a pile of EPBs.

> (e.g. systemd, USB, ...) into another (informational) document.

systemd journal entries and USB packets are different here - USB packets are stored in EPBs, just as they're stored in pcap files (i.e., USB packets have a LINKTYPE_ value usable in both pcap files and pcapng Interface Description Blocks - well, several of them, given that different OSes supply different pseudo-headers), but systemd journal entries have their own block type.

> Or perhaps just leave them in a web page, having pointed at them in pcap2's

So have the bulk of the information about linktypes, block types, and block-specific option types be on some web site not in in ietf.org, with the registry just giving linktype/block type/option names and numbers, along with a link to that site?

---

From: Randy Bush <[email-masked]>
Date: Tue, 06 Jul 2021 17:01:35 -0700

> Given how systemd is prone to considerable change, I'd think it would be

please.  and for many other reasons

randy

---
[email-masked]
`gpg --locate-external-keys --auto-key-locate wkd [email-masked]`
signatures are back, thanks to dmarc header butchery

---

From: Michael Richardson <[email-masked]>
Date: Wed, 14 Jul 2021 13:16:08 -0400

> The "systemd" block in pcapng is called the "systemd Journal Export
    > Block"; it doesn't cover systemd as a whole, it just covers the format
    > that journald uses to export entries from its log file.  The
    > description of the export block points to the description of the export
    > format:

So to be clear, the "systemd" block is about log file entries which are
partially binary.

I think that it deserves a document on it's own, regardless of what you might
think about systemd.

    > Do any of those RFCs point to an external specification for most of the
    > information about the item being discussed, similarly to what's the
    > case for the systemd Journal Export Block?

    > Are there any registries that have "less than an RFC and more than a
    > mailto: URL", i.e. that say more than just "ask this person, hopefully
    > they're still around, still have that email address, and still remember
    > what this entry in the registry was all about"?

Yes, this is called First Come, First Served.
It usually has an Designated Expert that IANA consults.

We haven't proposed any policy yet in section 11.1 of tuexen-opsawg-pcapng,
except for defining that 0x8000_0000 to 0xffff_ffff are for local use.
(IANA speak is: "PRIVATE USE")

We should probably reserve the first 64K for IETF Action (i.e. standards
track), 64K to 0x0fff_ffff for Specification Required (not necessarily an
RFC) and make the 0x1000_0000 to 0x7fff_ffff as FCFS.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        | network architect  [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: tom petch <[email-masked]>
Date: Thu, 15 Jul 2021 10:43:59 +0000

From: OPSAWG <[email-masked]> on behalf of Michael Richardson <[email-masked]>
Sent: 14 July 2021 18:16

<tp>

Not really - see RFC8126 s.4.

FCFS is no more than an e-mail.

Expert Review is the one that has a Designated Expert with guidelines for the expert provided by the RFC that created the registry.

Then there are the ones that require a specification; of any kind, of RFC required, of IETF Review and so on.  It is a wide spectrum but there is no IETF Action nor any standards track

More than an e-mail less than an RFC sounds like Specification Required or Expert Review, which can be combined if the RFC creating the registry so desires but the terminology needs to be precise and to be understood.  There are some WG that are an object lesson in what not to do:-(

Tom Petch

---

From: Michael Richardson <[email-masked]>
Date: Thu, 15 Jul 2021 14:02:50 -0400

>>>>> "tp" == OPSAWG  <[email-masked]> on behalf of Michael Richardson <[email-masked]> writes:
tom petch <[email-masked]> wrote:
    mcr> Yes, this is called First Come, First Served.
    mcr> It usually has an Designated Expert that IANA consults.

    tp> <tp>

    tp> Not really - see RFC8126 s.4.

    tp> FCFS is no more than an e-mail.

No, really.  IANA seems to punt to a designated expert to deal with FCFS.
I've done this recently for CBOR Tags.

The designated expert basically just filters for nonsense and duplicates.
If you think that's wrong, talk to the IAB/IESG.