Subject: Re: [ippm] WGLC for IOAM Direct Exports

From: Greg Mirsky <[email-masked]>
Date: Wed, 6 Oct 2021 16:30:15 -0700

Hi Tal,
thank you for your kind consideration of my comments and constructive
proposals. Please find my notes in-lined below under the GIM>> tag.

Regards,
Greg

On Thu, Sep 30, 2021 at 12:17 AM Tal Mizrahi <[email-masked]>
wrote:

> Hi Greg,
>
> Many thanks for the thorough review.
> Below you can find responses to your comments, and in some cases
> proposed text edits.
>
> If there are no further comments we will update the document accordingly.
>
> Thanks,
> The authors.
>
>
> On Wed, Sep 8, 2021 at 1:01 AM Greg Mirsky <[email-masked]> wrote:
> >
> > Dear Authors, WG Chairs, and All,
> > I've read the latest version of the draft and have several comments and
> questions to share with you. None of these are blocking, I support
> progressing this draft to publication. Please consider the notes below as
> an invitation to a discussion:
> >
> > It seems that the characterization of the Direct Export IOAM option in
> the following is not entirely accurate:
> >
> > The DEX option is used as a trigger to collect and/or export IOAM data.
> >
> > I propose re-wording:
> > The DEX option is used as a trigger to collect and export or only
> collect locally the IOAM data.
> >
> > Using "exporting and/or collecting" reflects the use of the Direct
> Export option type technically accurate but the order of actions is
> backward. What are your thoughts?
> >
>
> [TM] Right. The following text edit is proposed in the introduction.
>
> OLD:
>    This option is used as
>    a trigger for IOAM nodes to locally aggregate and process IOAM data,
>    and/or to export it to a receiving entity (or entities).
> NEW:
>    This option is used as
>    a trigger for IOAM nodes to locally aggregate and process IOAM data,
>    and/or to export it to a receiving entity (or entities). Throughout
>    the document this functionality is referred to as collection and/or
> exporting.
>
GIM>> Thank you, the new text is very helpful.

>
>
> > I find references to RFC 5475 and 7014 very appropriate and helpful. But
> I don't see these RFCs being referenced in draft-ietf-ippm-ioam-data. I
> think that excessive use of not only the Direct Export IAOM option but the
> IOAM method, in general, may adversely affect a network and the
> recommendations listed in RFC 5475 and 7014 are equally applicable.
> > Similarly, the good and helpful discussion of how to practically control
> the use of the Direct Export option is, in my opinion, equally applicable
> to other IOAM trace options and should be in the Security Considerations
> section of draft-ietf-ippm-ioam-data.
>
>
> [TM] The last two comments are about draft-ietf-ippm-ioam-data, which
> is currently in IESG review. I will take this feedback with the other
> authors to see how we can address this input at this point.
>
GIM>> Thank you.

>
>
> > The second paragraph in Section 3.1.2 is re-phrasing if not repeating
> what is already expressed in the last paragraph of the previous sub-section
> (though the limit is applied to exporting telemetry information). I think
> that it can be removed without any loss to the document.
>
> [TM] The second par of 3.1.2 refers to *exporting*, while the last par
> of 3.1.1 refers to *encapsulating*, so these are two complementary
> requirements.
>
GIM>> Thank you for clarifying that to me.

>
>
> > I think that it will be helpful to provide some recommendations, i.e.,
> SHOULD do this, how an implementation handles the excess of packets to
> export. Drop silently, drop and log notification, or try to shape that flow?
>
> [TM] Assuming an amplification scenario, a large number of packets may
> be dropped, so logging each drop may defeat the purpose. A drop
> counter may be a good idea here, but this document does not really
> define counter requirements in general, so it seems like more of an
> implementation-specific aspect, or possibly can be considered as part
> of the YANG model.
>
GIM>> I think that logging a warning with the throttling control is a good
option. What is your opinion? A counter of throttled DEX packets is
helpful. Perhaps both mechanisms can be recommended.

>
> > I was really puzzled by this requirement in Section 3.1.2
> >
> >    Exported packets SHOULD NOT be exported over a path or a tunnel that
> >
> >    is subject to IOAM direct exporting.
> >
> > I hope you can clarify it for me.
>
>
> [TM] As explained later in that paragraph: "This requirement is
> intended to prevent nested exporting and/or exporting loops."
> This was thoroughly discussed in the working group, and came out of
> the security-related review.
>
GIM>> As I understand IOAM DEX, packets with telemetry information are
out-of-band relative to the monitored data flow and encapsulated to avoid
being taken for an IOAM packet. I imagine that exported telemetry data can
be collected via, for example, Kafka or gRPC. Am I missing something?

>
> >
> > This requirement
> >
> >    Furthermore, IOAM encapsulating
> >    nodes that can identify a packet as an IOAM exported packet MUST NOT
> >    push a DEX option into such a packet.
> > brings up questions:
> >
> > How can a node identify a packet as the IOAM exported packet?
>
> [TM] The requirement means that if an encapsulating node's ACL can
> detect exported packets, then DEX should not be applied.
>
GIM>> Please consider my previous note. I understand that that gets very
close to an implementation choice but it seems that the document should be
clear in its normative statements.

>
> > What should be done if the IOAM exported packet contains the IOAM header
> with the DEX option?
>
> [TM] The current document does not require an IOAM transit node to
> parse beyond the (external) IOAM encapsulation if there are nested
> IOAM encapsulations.
>
GIM>> I have to admit that I'm lost with the reference to an "external IOAM
encapsulation". What is the purpose of using IOAM in the exported packet?

>
> >
> > What is special about the decapsulating node that is not DEX-capable?
> >
> >    A decapsulating node that does not support the DEX option
> >    MUST remove it, along with any other IOAM options carried in the
> >    packet if such exist.
> > Shouldn't that be a general behavior of a decapsulating IOAM node?
>
> [TM] Right. The following text edit is proposed:
>
> OLD:
>    A transit IOAM node that does not support the DEX option SHOULD
>    ignore it.  A decapsulating node that does not support the DEX option
>    MUST remove it, along with any other IOAM options carried in the
>    packet if such exist.
> NEW:
>    A transit or decapsulating IOAM node that does not support the DEX
> option SHOULD
>    ignore it.  Note that as per <xref target="I-D.ietf-ippm-ioam-data"/> a
>    decapsulating node removes the IOAM encapsulation and all its IOAM
> options,
>    and specifically in the case where one of these options is a
> (possibly unknown)
>    DEX option.
>
GIM>> What could be an alternative to ignoring IOAM DEX option? I feel that
MUST is what is required here. Would you agree?

>
>
> >
> > The format of the first four-octet word in Figure 2 is different from
> the format defined in draft-ietf-ippm-ioam-data:
> >
> >     0                   1                   2                   3
> >     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
> >    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> >    |        Namespace-ID           |NodeLen  | Flags | RemainingLen|
> >    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> >    |               IOAM-Trace-Type                 |  Reserved     |
> >    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> > Is this intentional? If it is, how does an implementation know which
> format to use?
>
> [TM] Each IOAM Option-Type has a different header format. The current
> document defines a new Option-Type, which was not defined in
> draft-ietf-ippm-ioam-data.
> The IOAM-Type field in the lower layer header determines which
> Option-Type is used and how its header should be parsed.
>
GIM>> How does an IOAM node differentiate between different Option-Types?
Is that expected to be communicated to an IOAM node over the management or
control plane and not part of the IOAM encapsulation? If that is the case,
then, I imagine, changing between IOAM tracing modes will require
reconfiguration of all IOAM nodes. Doesn't seem operationally friendly.

>
>
> >
> > It is stated in the draft that:
> >
> > The length of the DEX option is at least 8 octets.
> >
> > But the figure 2 in the draft does not include any field that reflects
> the length of the IOAM option. Is that because the figure is out of sync?
> If not, how can an implementation verify the length? Also, what is the
> behavior if the length is less than 8 octets?
> >
>
> [TM] Quoting the draft:
>
>    The existence of the optional fields is
>    indicated by the corresponding flags in the Extension-Flags field.
>    ...
>    Thus, the Extension-Flags field explicitly indicates
>    the length of the DEX option.
>
>
> > RE: PerformanceConsiderations
> >
> > I agree with you that a DEX-encapsulating node and DEX-transit nodes
> must have rate-limiting controls (as noted earlier, the behavior of the
> transit node needs more detailed specification). But I think that the
> document should also note that exported packets could be sent over the
> network out-of-band, relative to the monitored flow. As a result, the
> impact of packets carrying telemetry information might be not direct and
> harmful to the data flow. I see that as one of the important benefits of
> the Direct Export method transporting on-path telemetry information.
> >
>
> [TM] Right. The following new text is proposed:
>
> It should be noted that in some networks DEX data may be exported over
> an out-of-band network, in which a large volume of exported traffic
> does not compromise user traffic. In this case an operator may choose
> to disable the exporting rate limiting.
>
GIM>> Thank you for the proposed update. I thought that the main advantage
of the IOAM DEX mode is the ability to export the telemetry information
out-of-band. And, as I've noted above, the mechanism to collect and
transport the telemetry is outside the scope of this specification. I think
that would significantly simplify the document without any adverse impact
on the value it provides.

>
> >
> > Regards,
> > Greg
> >
> > On Mon, Aug 30, 2021 at 1:58 PM Tommy Pauly <tpauly=
> [email-masked]> wrote:
> >>
> >> Hello IPPM,
> >>
> >> This email starts a Working Group Last Call on two related IOAM
> documents, draft-ietf-ippm-ioam-flags and
> draft-ietf-ippm-ioam-direct-export.
> >>
> >> In-situ OAM Loopback and Active Flags
> >> https://datatracker.ietf.org/doc/draft-ietf-ippm-ioam-flags/
> >> https://datatracker.ietf.org/doc/html/draft-ietf-ippm-ioam-flags-06
> >>
> >> In-situ OAM Direct Exporting
> >> https://datatracker.ietf.org/doc/draft-ietf-ippm-ioam-direct-export/
> >>
> https://datatracker.ietf.org/doc/html/draft-ietf-ippm-ioam-direct-export-06
> >>
> >> Please provide feedback by replying to this email with your reviews and
> if you think these documents are ready to progress, by Wednesday, September
> 15.
> >>
> >> Best,
> >> Tommy & Ian
> >>
> >> _______________________________________________
> >> ippm mailing list
> >> [email-masked]
> >> https://www.ietf.org/mailman/listinfo/ippm
> >
> > _______________________________________________
> > ippm mailing list
> > [email-masked]
> > https://www.ietf.org/mailman/listinfo/ippm
>

---

From: Tal Mizrahi <[email-masked]>
Date: Thu, 7 Oct 2021 08:22:17 +0300

Hi Greg,

Thanks for the responses.
Please see additional responses below marked with [[TM]].
If there are no further comments we will post an updated version.

Cheers,
Tal.

On Thu, Oct 7, 2021 at 2:30 AM Greg Mirsky <[email-masked]> wrote:
>
> Hi Tal,
> thank you for your kind consideration of my comments and constructive proposals. Please find my notes in-lined below under the GIM>> tag.
>
> Regards,
> Greg
>
> On Thu, Sep 30, 2021 at 12:17 AM Tal Mizrahi <[email-masked]> wrote:
>>
>> Hi Greg,
>>
>> > I've read the latest version of the draft and have several comments and questions to share with you. None of these are blocking, I support progressing this draft to publication. Please consider the notes below as an invitation to a discussion:
>> >
>> > It seems that the characterization of the Direct Export IOAM option in the following is not entirely accurate:
>> >
>> > The DEX option is used as a trigger to collect and export or only collect locally the IOAM data.
>> >
>> > Using "exporting and/or collecting" reflects the use of the Direct Export option type technically accurate but the order of actions is backward. What are your thoughts?
>> >
>>
>> > I find references to RFC 5475 and 7014 very appropriate and helpful. But I don't see these RFCs being referenced in draft-ietf-ippm-ioam-data. I think that excessive use of not only the Direct Export IAOM option but the IOAM method, in general, may adversely affect a network and the recommendations listed in RFC 5475 and 7014 are equally applicable.
>> > Similarly, the good and helpful discussion of how to practically control the use of the Direct Export option is, in my opinion, equally applicable to other IOAM trace options and should be in the Security Considerations section of draft-ietf-ippm-ioam-data.
>>
>>
>> > The second paragraph in Section 3.1.2 is re-phrasing if not repeating what is already expressed in the last paragraph of the previous sub-section (though the limit is applied to exporting telemetry information). I think that it can be removed without any loss to the document.
>>
>> > I think that it will be helpful to provide some recommendations, i.e., SHOULD do this, how an implementation handles the excess of packets to export. Drop silently, drop and log notification, or try to shape that flow?
>>
> GIM>> I think that logging a warning with the throttling control is a good option. What is your opinion? A counter of throttled DEX packets is helpful. Perhaps both mechanisms can be recommended.

[[TM]] The following new sentence is proposed to be added to Section 3.1.2:
NEW:
An IOAM node MAY maintain a counter or a set of counters that count
the events in which the IOAM node receives a packet with the DEX
option type and does not collect and/or export data due to the rate
limits.

> GIM>> As I understand IOAM DEX, packets with telemetry information are out-of-band relative to the monitored data flow and encapsulated to avoid being taken for an IOAM packet. I imagine that exported telemetry data can be collected via, for example, Kafka or gRPC. Am I missing something?

[[TM]] Right, the export encapsulation may be any of the methods you
mentioned. However, if the export path runs through another IOAM
domain it creates a nested IOAM scenario. This scenario was raised by
Martin Duke in his security-related review, and we added this text in
order to address this scenario. This also relates to the next comment
(about pushing the DEX option onto exported packets).

> GIM>> Please consider my previous note. I understand that that gets very close to an implementation choice but it seems that the document should be clear in its normative statements.
>>
>>
>> > What should be done if the IOAM exported packet contains the IOAM header with the DEX option?
>>
> GIM>> I have to admit that I'm lost with the reference to an "external IOAM encapsulation". What is the purpose of using IOAM in the exported packet?

[[TM]] My point was that in the scenario in which an exported packet
is forwarded through an IOAM domain we do not expect IOAM transit
nodes to detect this. We expect IOAM transit nodes to parse the outer
encapsulation, but not beyond that.

>>    decapsulating node removes the IOAM encapsulation and all its IOAM options,
> GIM>> What could be an alternative to ignoring IOAM DEX option? I feel that MUST is what is required here. Would you agree?

[[TM]] Interesting point, because a node that does not support the DEX
option does not comply to this document, and so the sentence we
suggested would be pointless.
I suggest the following text instead:
NEW:
A transit or decapsulating IOAM node that receives an unknown IOAM option
type ignores it (as defined in <xref target="I-D.ietf-ippm-ioam-data"/>), and
specifically nodes that do not support the DEX option ignore it.
Note that as per <xref target="I-D.ietf-ippm-ioam-data"/> a
decapsulating node removes the IOAM encapsulation and all its IOAM options,
and specifically in the case where one of these options is a
(possibly unknown) DEX option.

>>
>>
>>
>> >
>> > The format of the first four-octet word in Figure 2 is different from the format defined in draft-ietf-ippm-ioam-data:
>> >
>> >     0                   1                   2                   3
>> > Is this intentional? If it is, how does an implementation know which format to use?
>>
> GIM>> How does an IOAM node differentiate between different Option-Types? Is that expected to be communicated to an IOAM node over the management or control plane and not part of the IOAM encapsulation? If that is the case, then, I imagine, changing between IOAM tracing modes will require reconfiguration of all IOAM nodes. Doesn't seem operationally friendly.

[[TM]] This question is not strictly related to the current draft, but
anyway an IOAM node differentiates between different option types
according to the IOAM Type field which resides in the lower layer
header. For example, in the IPv6 option
(draft-ietf-ippm-ioam-ipv6-options-06), the IPv6 Hop-by-Hop Option
header includes the IOAM-Type field:

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Option Type  |  Opt Data Len |   Reserved    |   IOAM Type   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

>> > But the figure 2 in the draft does not include any field that reflects the length of the IOAM option. Is that because the figure is out of sync? If not, how can an implementation verify the length? Also, what is the behavior if the length is less than 8 octets?
>> >
>>
>> [TM] Quoting the draft:
>>
>> > I agree with you that a DEX-encapsulating node and DEX-transit nodes must have rate-limiting controls (as noted earlier, the behavior of the transit node needs more detailed specification). But I think that the document should also note that exported packets could be sent over the network out-of-band, relative to the monitored flow. As a result, the impact of packets carrying telemetry information might be not direct and harmful to the data flow. I see that as one of the important benefits of the Direct Export method transporting on-path telemetry information.
>> >
>>
> GIM>> Thank you for the proposed update. I thought that the main advantage of the IOAM DEX mode is the ability to export the telemetry information out-of-band. And, as I've noted above, the mechanism to collect and transport the telemetry is outside the scope of this specification. I think that would significantly simplify the document without any adverse impact on the value it provides.
>>
>>
>> >
>> > Regards,
>> > Greg
>> >
>> > On Mon, Aug 30, 2021 at 1:58 PM Tommy Pauly <tpauly=[email-masked]> wrote:
>> >>
>> >> Hello IPPM,
>> >>
>> >> This email starts a Working Group Last Call on two related IOAM documents, draft-ietf-ippm-ioam-flags and draft-ietf-ippm-ioam-direct-export.
>> >>
>> >> Please provide feedback by replying to this email with your reviews and if you think these documents are ready to progress, by Wednesday, September 15.
>> >>
>> >> Best,
>> >> Tommy & Ian
>> >>

---

From: Greg Mirsky <[email-masked]>
Date: Fri, 8 Oct 2021 09:03:57 -0700

Hi Tal,
thank you for your responses to my comments and questions. Please find my
follow-up notes in-lined below under the GIM>> tag.

Regards,
Greg

On Thu, Sep 30, 2021 at 12:42 AM Tal Mizrahi <[email-masked]>
wrote:

> Hi Greg,
>
> Many thanks for the detailed comments.
> Please see the responses below, as well as suggested text edits in some
> cases.
>
> Cheers,
> The authors.
>
> On Wed, Sep 8, 2021 at 1:00 AM Greg Mirsky <[email-masked]> wrote:
> >
> questions to share with you. I would much appreciate the opportunity to
> discuss these before I can state my support for progressing this draft to
> publication.
> >
> > The Abstract states that:
> >
> >    In-situ Operations, Administration, and Maintenance (IOAM) records
> >    operational and telemetry information in packets while they traverse
> >    a path between two points in the network.
> >
> > I think that is not a complete description of IOAM options as it
> excludes the Direct Export option. It might be easier to improve by
> s/records/collects/. Also, what, in your opinion, is the distinction
> between operational and telemetry information? I think that operational
> information collected from remote nodes is one example of telemetry
> information. What do you think?
>
> [TM] Agreed. Your proposal makes sense: s/records/collects/
> Regarding the difference between operational and telemetry
> information, the way I see it "telemetry" refers to measurements that
> are being exported to an external/remote entity, whereas operational
> information is not measurement-based, such as the Node ID or Interface
> ID.
>
GIM>> I think that there's no standardized definition of "telemetry" and
"telemetry information". I share the idea expressed in the following:

Telemetry is automated communication that processes from multiple data
sources. Telemetry data improves customer experiences, monitors security,
application health, quality, and performance.

For different cases, the set of information referred to as telemetry data
variates. For example, CPU and buffer utilization are valuable for
virtualized environments. In other words, I understand telemetry data as
information extracted from a remote system that characterizes the system's
state. In any case, providing a definition would be helpful to a reader of
the document.

>
>
> >
> > nit s/the the/the/
>
> [TM] Agreed.
>
> > I think that Abstract and Introduction can be improved by explaining the
> purpose of defining the Loopback and Active flags. What each of them is
> intended to achieve and improve?
>
> [TM] The point is well-taken, and the following text will be added to
> the introduction. In order to avoid text repetition, I believe it
> would be best not to add it to the abstract as well.
>
> OLD:
> -
> NEW:
> The Loopback flag is used to request that each transit device along
> the path loops back a copy of the data packet to the sender. The
> Active flag indicates that a packet is used for active measurement.
>
GIM>> Thank you for the proposed text. I have several notes to the new text:

   - As I understand the behavior required from a node processing the
   Loopback flag, the returned packet is not the complete but truncated data
   packet.
   - Is "active measurement" used in the context of RFC 7799? If "yes",
   does that mean that the packet is a specifically generated test packet?

>
> > It is stated in the Introduction that:
> >
> >    This document defines
> >    two new flags in the Pre-allocated and Incremental Trace options: the
> >    Loopback and Active flags.
> > Does that mean that neither of the two flags is applicable in the Direct
> Export option? Should draft-ietf-ippm-ioam-direct-export have an explicit
> statement regarding the applicability of the Loopback and Active flags? And
> if any is applicable, describe the behavior of an IOAM node receiving it.
>
> [TM] Right.
> The direct exporting draft explicitly states the behavior of each
> field in the DEX option header including the flags, so there seems to
> be no need to specify flags that are supported in other options.
>
GIM>> I'm a bit confused. Can you help me with the text from the DEX draft
that specifies the use of the Loopback and Active flags in combination with
the DEX trace option?

>
> >
> > In Section 4 the purpose of using the Loopback flag is explained as:
> >
> >    Loopback allows an IOAM encapsulating node to trace the path to a
> >    given destination, and to receive per-hop data about both the forward
> >    and the return path.
> > In my previous comments, I've pointed out that the Loopback mode is just
> a special case of the Direct Export option. In addition to that, I would
> appreciate your explanation of how the IOAM encapsulating node receives
> per-hop data about the return path. How does that work, for example, in an
> MPLS or SFC NSH environment?
>
> [TM] It is slightly similar to direct exporting, but is not a special
> case of direct exporting. In direct exporting telemetry/operational
> data is not pushed into the data packet, but the data packet with the
> DEX option triggers the data collection and optionally also exporting,
> but exporting is not mandatory and may be performed later, for example
> by the management plane. On the other hand, a trace option with the
> loopback flag causes all IOAM transit nodes to push IOAM data onto the
> trace option, and also loop a copy of the packet back to the sender.
>
GIM>> Thank you for providing that comparison between the result of
processing DEX and the Loopback. As I understand it, the result of DEX,
i.e., immediate export or local processing with exporting a batch of
telemetry data, is determined by the local policy. If that is the case, the
local policy can produce the same result as for the Loopback flag, which
could be the default behavior for the DEX.

>
>
> >
> > Further it Section 4, it is stated that:
> >
> >    The two IOAM looped back copies are
> >    terminated by the encapsulating node.
> > But I don't see any technical explanation that could sustain that
> assumption. As I understand the diagram in figure 1, a transit node doesn't
> know the identity of the IOAM encapsulating node. Correct? If that is the
> case, where does the transit node direct the looped packet?
>
> [TM] As described in the draft, the loopback flag is not necessarily
> applicable to all possible encapsulations. The way the return path is
> defined is encapsulation-specific. The draft currently states:
>
>    Loopback can be used only if a return path from transit nodes and
>    destination nodes towards the source (encapsulating node) exists.
>    Specifically, loopback is only applicable in encapsulations in which
>    the identity of the encapsulating node is available in the
>    encapsulation header.  If an encapsulating node receives a looped
>    back packet that was not originated from the current encapsulating
>    node, the packet is dropped.
>
GIM>> Please help me understand, the looped back packet is transmitted
using the same encapsulation as the data packet with the Loopback flag?
What is the rationale? If we use MPLS LSP Ping as an example, the response
can be sent in several ways, but the default is in IP/UDP encapsulation as
the commonly available transport. Why not use that and send the looped
packet over IP/UDP? I believe that the applicability of this mechanism will
be much broader.

>
>
> > It seems that the text in the last paragraph attempts to walk around my
> questions. But I don't find that paragraph sufficient. It lacks the
> normative language to guide a developer to produce resilient
> implementation. For example, how to interpret:
> >    Loopback can be used only if a return path from transit nodes and
> >    destination nodes towards the source (encapsulating node) exists.
> > That reads like a recommendation, not a requirement. As a result, an
> implementation may transmit an IOAM packet with the Loopback fag set
> regardless of whether there's a return path or not. Then, how must a
> transit node process such an IOAM packet?
> >
>
> [TM] Right, the last response and quoted text is relevant here as
> well. Please also note the following text from the draft:
>
> it is assumed that the
>    source address is available in the encapsulation header.  Thus, the
>    source address of the original packet is used as the destination
>    address in the copied packet.
>
> I propose to add the following sentence:
> The loopback flag MUST NOT be set if it is not guaranteed that there
> is a return path from each of the IOAM transit and IOAM decapsulating
> nodes, or if the encapsulating node's address is not available in the
> encapsulation header.
>
GIM>> Thank you for the proposed text. I'd note that referring to the
"encapsulating node's address" and "source address" might be interpreted as
applying only to IP. In some networks, for example, BIER, the identity of
the IOAM encapsulating node, BFIR, is known from its identifier that, in
turn, can be mapped to the IP address.

>
>
> > A very concerning statement I've found in Section 4.1
> >
> >    The encapsulating node either generates synthetic packets with an
> >    IOAM trace option that has the Loopback flag set ...
> > IOAM has been defined as a hybrid measurement protocol. This statement
> implies that it is to be used as an active OAM protocol (all per RFC 7799
> definitions). The requirements for active OAM protocols in terms of their
> applicability and, most of, security are very different from those set for
> a hybrid measurement protocol. I believe that extending IOAM into an active
> mode of measurement requires more discussion and the audience from other
> IETF WGs, e.g., 6man, SFC, BIER, MPLS, BFD, RTGWG.
>
> [TM] This draft does not define an active measurement protocol. The
> point in the draft is that the loopback flag can be used in active
> measurement packets.
>
GIM>> I have to disagree. Per RFC 7799, the use of specially
constructed test packets is the primary characteristic of an active
measurement method. By introducing the Active flag, this drat makes the
IOAM into an active measurement protocol with all the requirements to an
active measurement protocol to be applied.

> The following text edit is proposed:
>
> OLD:
>    The encapsulating node either generates synthetic packets with an
>    IOAM trace option that has the Loopback flag set, or sets the loopack
>    flag in a subset of the in-transit data packets.  Loopback is used
>    either proactively or on-demand, i.e., when a failure is detected.
> NEW:
>    The loopback flag can be set in the IOAM trace option either in a
>    subset of the in-transit data packets, or in synthetic packets that
>    are generated by the encapsulating node. Loopback can be used
>    either proactively or on-demand, i.e., when a failure is detected.
>
> >
> > The next statement explains the use case for the Loopback flag:
> >
> >    Loopback is used
> >    either proactively or on-demand, i.e., when a failure is detected.
> > That is the same use of ICMP and LSP Ping. What is the benefit of using
> the Loopback flag instead of readily available ICMP? Does it work as L2
> Linktrace? I don't see that as a sufficient benefit considering the
> complexity, exclusions, and risks associated and identified in the draft.
> It is clear that there are tools like ICMP and LSP Ping that adequately
> address scenarios listed in the draft. Don't see any technical reason for
> the introduction of another mechanism that does not have any significant
> benefit compared to the existing OAM tools. Especially, considering the
> limited information that is allowed to be collected using the Loopback IOAM
> flag ("every transit node that processes this trace option only adds a
> single data field, which is the Hop_Lim and node_id data field"). With the
> abundance of information and additional verification achieved using ICMP
> and LSP Ping, there's even less rationale in the introduction of this
> over-complex and unnatural mechanism.
>
> [TM] Right, as the draft says, the loopback flag is an "...alternative
> to Traceroute, that allows the encapsulating node to receive responses
> from multiple transit nodes along the path". Indeed, it slightly
> resembles ETH-LT (Ethernet Linktrace) in the sense that a single
> packet from the encapsulating node allows to quickly trace the entire
> path, but as with IOAM in general, using the loopback flag does not
> mandate a specific encapsulation.
>
GIM>> Also, I must note that "pro-active" use of the Loopback is possible
only as part of the active measurement protocol.

>
> >
> > The last paragraph states the requirement:
> >
> >    IOAM encapsulating nodes MUST NOT push an IOAM encapsulation with the
> >    Loopback flag onto data packets that already include an IOAM
> >    encapsulation.
> > I couldn't find any guidance on how an IOAM node that receives such
> invalid combination processes the packet.
> >
>
> [TM] The current draft only specifies the flags. Specifically in the
> scenario described above, the encapsulating node does not set the
> loopback flag. This is intended to reduce the impact of potential
> amplification attacks.
>
GIM>> I am still not clear about "MUST NOT push an IOAM encapsulation with
the Loopback flag" in the draft and your explanation "the encapsulating
node does not set the Loopback flag". As I understand it, in the given
network for the monitored data flow, only one IOAM encapsulating node can
exist. If that is correct, where does the existing IOAM encapsulation
come from?

>
> >
> > It is assumed that the identity of the IOAM encapsulating node is known
> to the node that is expected to sent a copy of the received packet back.
> But what if that is not the case? How must that node act?
>
> [TM] Agreed. The following new text is proposed to be added to Section 4.1:
> If the address of the encapsulating node is not available in the
> encapsulation header, then the transit/decapsulating node does not
> loop back a copy of the original packet.
>
GIM>> I think that points to the significant limitation of the proposed
mechanism. Comparing with MPLS LSP Ping, we see that it allows tracing the
LSP even though MPLS encapsulation does not identify the ingress LER while
the proposed Loopback would not work at all.

>
> > I believe that there's inconsistency between Figure 1 and the following
> text in Section 4.2
> >
> >    Thus, the
> >    source address of the original packet is used as the destination
> >    address in the copied packet.
> > It seems that the text assumes that the IOAM encapsulating node adds
> also a network layer encapsulation. But how would it work, for example, in
> SFC NSH data plane? Or MPLS?
>
> [TM] Right, the paragraph beginning with "Loopback can be used
> only..." (discussed in response to one of the previous comments) makes
> that restriction.
>
> >
> > Section 4.4 states that
> >
> >    If the Node ID matches the current
> >    IOAM node, it indicates that this is a looped back packet that was
> >    initiated by the current IOAM node, and processed accordingly.
> > But according to the definition of node_id in Section 5.4.2.1 of
> draft-ietf-ippm-ioam-data, "Node identifier field to uniquely identify a
> node within the IOAM-Namespace and associated IOAM-Domain." It appears that
> simply comparing Node IDs is not sufficient to verify that the node
> received its looped back packet. What is your opinion?
>
> [TM] Agreed. The following update is proposed:
> OLD:
> If the Node ID matches
> NEW:
> If the Node ID and IOAM-Namespace match
>
>
> >
> > If, as stated in Section 5,
> >
> >    It [Active flag] is not intended as a replacement for existing
> >    active OAM protocols, which may run in higher layers and make use of
> >    the Active flag.
> > Then the question, What is the purpose of the Active flag? Existing
> active protocols, Fault Management and Performance Monitoring, are
> identified by, for example, use of a well-known destination port number or,
> in case of MPLS, Generic Associated Channel type. There's no apparent need
> of using IOAM option header to identify the payload as a control message of
> an active OAM protocol. On the contrary, that would likely cause more
> confusion and challenges to interoperation among different implementations.
> >
> > Three cases analyzed in Section 5 require special consideration, as
> these likely present the motivation for the introduction of the Active flag:
> >
> > the first use case is not really using the Active flag at all and thus
> the text doesn't seem adding any value to the draft;
> > it is not clear what is viewed as the benefit of using the Active flag
> compared with using any of the existing active OAM protocols in the second
> use case. I would appreciate it if you can clarify that for me. Since an
> SFC NSH network is used as an example, is it required that a Service
> Function (SF) or an SF Proxy process the IOAM header with Active flag set?
> Or would a Service Function Forwarder (SFF) avoid sending a packet with the
> IOAM header that has an Active flag set? And can you compare that with the
> procedures defined in the Active OAM for SFC draft?
> > My questions for the third use case described in Section 5 are the same
> as above because I don't see the benefit of using a replicated data packet
> (less benefits if it is truncated) compared to a special test packet.
> >
> > In my conclusion on the Active flag, I don't see any clear benefit of
> introducing it and using whether in place or in addition to the existing
> active OAM protocols.
>
> [TM] General response to all the Active flag comments: this was
> extensively discussed on the mailing list and in previous IETF
> meetings, and your comments about the previous versions of the
> document have contributed to greatly improving Section 5, which
> describes the main use cases of the Active flag.
>
GIM>> I have to disagree with you here. I don't find the explanation in the
document sufficient to justify the introduction of another mechanism to
achieve what has already been satisfactorily addressed in IETF. Without a
more detailed specification of the problem solved by the Active flag and
how that is achieved, this mechanism appears like the answer is looking for
a question, like a loaded gun without safety.

>
>
> >
> > I am looking forward to your feedback and the continued discussion.
> >
> > Regards,
> > Greg
> >

---

From: Greg Mirsky <[email-masked]>
Date: Sun, 10 Oct 2021 10:23:22 -0700

Hi Tal,
thank you for carefully considering my comments and suggestions. Please
find my follow-up notes in-lined below under the GIM>> tag.

Regards,
Greg

>    the document this functionality is referred to as collection and/or
GIM>> As I understand it, it is the local policy that controls the behavior
of the node receiving the IOAM DEX indicator. Similar to IPFIX, the
information can be aggregated and exported as a calculated metric, e.g.,
percentile or mean value, or exported as soon as possible as recorded. The
new text is certainly better but it seems it does not express the role of
the local policy in defining the processing behavior, the transport to be
used, and the identity of the Collector. One of possible ways could be to
state that the definition of all these controls is outside the scope of the
document.

> authors to see how we can address this input at this point.
GIM>> Thanks.

> of 3.1.1 refers to *encapsulating*, so these are two complementary
GIM>> Is it the intention of this draft to define the encapsulation of
exported information? What if the local policy mandates the node
calculating p95 and p99 of packet delay and inter-packet delay variation?
Can you help me by describing how these metrics are encapsulated?

> implementation-specific aspect, or possibly can be considered as part
GIM>> I am still struggling to understand how IOAM DEX triggers
amplification. Can you add more details, steps to help me understand it?

> the security-related review.
GIM>> If the definition of exporting, including the encapsulation, choice
of transport, and the Collector is outside the scope, then this statement,
in my opinion, should be removed. If all these issues are in the scope,
then  there is more, much more information that is missing.

> detect exported packets, then DEX should not be applied.
GIM>> I think that whether this belongs in the document depends on the
answer to my question above - what is in the scope of this document.

> parse beyond the (external) IOAM encapsulation if there are nested
GIM>> Same, as above - need to clarify the scope of the document first.

>    and specifically in the case where one of these options is a
GIM>> Why is it SHOULD? In what situation a node MAY act on the DEX?

> Option-Type is used and how its header should be parsed.
GIM>> Thank you for the clarification. Is that the intention to start new
drafts to define the DEX IOAM Option-Type separately?

> to disable the exporting rate limiting.
GIM>> I think that the text can be stronger, recommending that
information is exported out-of-band and explaining the interpretation of
out-of-band in the case of IOAM.

---

From: Tal Mizrahi <[email-masked]>
Date: Wed, 13 Oct 2021 15:11:49 +0300

Dear Chairs,

We want to thank Greg for his thorough review and useful comments.
Although these comments were defined by Greg as non-blocking we made
an effort to address them either by updating the draft or by
responding to Greg's comments.
Some further responses can be found below, marked [[TM]].

We have posted an updated version (07) that seems to address these
comments, as well as comments received from Bernard Aboba (TSV-ART
reviewer).
https://datatracker.ietf.org/doc/draft-ietf-ippm-ioam-direct-export/

We believe the draft is ready to be submitted to the IESG.

Thanks,
Authors.

On Sun, Oct 10, 2021 at 8:23 PM Greg Mirsky <[email-masked]> wrote:
>
> Hi Tal,
> thank you for carefully considering my comments and suggestions. Please find my follow-up notes in-lined below under the GIM>> tag.
>
> Regards,
> Greg
>
> GIM>> As I understand it, it is the local policy that controls the behavior of the node receiving the IOAM DEX indicator. Similar to IPFIX, the information can be aggregated and exported as a calculated metric, e.g., percentile or mean value, or exported as soon as possible as recorded. The new text is certainly better but it seems it does not express the role of the local policy in defining the processing behavior, the transport to be used, and the identity of the Collector. One of possible ways could be to state that the definition of all these controls is outside the scope of the document.

[[TM]] Right. The following text edit is proposed.
OLD:
Note that even though the IOAM Option-Type is called "Direct Export",
      it depends on the deployment whether the receipt of a packet with DEX
      option type leads to the creation of another packet. Some deployments
      might simply use the packet with the DEX option type to trigger local
      processing of OAM data.
NEW:
Note that even though the IOAM Option-Type is called "Direct Export",
      it depends on the deployment whether the receipt of a packet with DEX
      option type leads to the creation of another packet. Some deployments
      might simply use the packet with the DEX option type to trigger local
      processing of OAM data. The functionality of this local processing is not
  within the scope of this document.

> GIM>> Is it the intention of this draft to define the encapsulation of exported information? What if the local policy mandates the node calculating p95 and p99 of packet delay and inter-packet delay variation? Can you help me by describing how these metrics are encapsulated?

[[TM]] As emphasized in the document: "The exporting method and format
are outside the scope of this document."
Section 3.1.2 actually refers (as its name suggests) to "Responding to
the DEX Trigger".

> GIM>> I am still struggling to understand how IOAM DEX triggers amplification. Can you add more details, steps to help me understand it?

[[TM]] This is explained in the document:

   The amplification effect of DEX may be worse in wide area networks in
   which there are multiple IOAM domains.  For example, if DEX is used
   in IOAM domain 1 for exporting IOAM data to a receiving entity, then
   the exported packets of domain 1 can be forwarded through IOAM domain
   2, in which they are subject to DEX.  The exported packets of domain
   2 may in turn be forwarded through another IOAM domain (or through
   domain 1), and theoretically this recursive amplification may
   continue infinitely.

> GIM>> If the definition of exporting, including the encapsulation, choice of transport, and the Collector is outside the scope, then this statement, in my opinion, should be removed. If all these issues are in the scope, then  there is more, much more information that is missing.

[[TM]] As explained in the abstract, "The exporting method and format
are outside the scope of this document."
However, the current document has a paragraph that defines security
requirements for the export method. These requirements came out of the
very detailed security discussions we had with Martin Duke and Mirja
Kühlewind.

> GIM>> I think that whether this belongs in the document depends on the answer to my question above - what is in the scope of this document.

[[TM]] See the response above.

> GIM>> Why is it SHOULD? In what situation a node MAY act on the DEX?

Right.
Proposed text:
          A transit or decapsulating IOAM node that receives an
unknown IOAM option
          type ignores it (as defined in <xref
target="I-D.ietf-ippm-ioam-data"/>), and
          specifically nodes that do not support the DEX option ignore it.
          Note that as per <xref target="I-D.ietf-ippm-ioam-data"/> a
          decapsulating node removes the IOAM encapsulation and all
its IOAM options,
          and specifically in the case where one of these options is a
          (possibly unknown) DEX option.

> GIM>> Thank you for the clarification. Is that the intention to start new drafts to define the DEX IOAM Option-Type separately?

[[TM]] Right. IOAM-Type values are defined in per encapsulation
drafts, such as draft-ietf-ippm-ioam-ipv6-options.

> GIM>> I think that the text can be stronger, recommending that information is exported out-of-band and explaining the interpretation of out-of-band in the case of IOAM.
>>
>>
>> >
>> > Regards,
>> > Greg
>> >

---

From: Tal Mizrahi <[email-masked]>
Date: Wed, 13 Oct 2021 15:18:50 +0300

Dear Chairs,

Thanks again to Greg for a very thorough review that helped us improve
the quality and clarity of the draft. We believe we addressed most of
the comments, either by updating the text, or by responding to the
comments.
Some further responses can be found below, marked [[[TM]]].

Note (1): it looks like some of the comments are basically about the
purpose of the document. Addressing these documents would probably
require a complete rewriting of the document with a different purpose.
At this point we do not think it would make sense to align to these
comments. These comments can be easily found by searching for "(1)"
through this email.

We have posted an updated version of this draft (07):
https://datatracker.ietf.org/doc/draft-ietf-ippm-ioam-direct-export/

Thanks,
Authors.

On Fri, Oct 8, 2021 at 7:04 PM Greg Mirsky <[email-masked]> wrote:
>
> Hi Tal,
> thank you for your responses to my comments and questions. Please find my follow-up notes in-lined below under the GIM>> tag.
>
> Regards,
> Greg
>
> On Thu, Sep 30, 2021 at 12:42 AM Tal Mizrahi <[email-masked]> wrote:
>>
>> Hi Greg,
>>
>> Please see the responses below, as well as suggested text edits in some cases.
>>
>> Cheers,
>> The authors.
>>
>> > I've read the latest version of the draft and have several comments and questions to share with you. I would much appreciate the opportunity to discuss these before I can state my support for progressing this draft to publication.
>> >
>> > I think that is not a complete description of IOAM options as it excludes the Direct Export option. It might be easier to improve by s/records/collects/. Also, what, in your opinion, is the distinction between operational and telemetry information? I think that operational information collected from remote nodes is one example of telemetry information. What do you think?
>>
> GIM>> I think that there's no standardized definition of "telemetry" and "telemetry information". I share the idea expressed in the following:
>
> Telemetry is automated communication that processes from multiple data sources. Telemetry data improves customer experiences, monitors security, application health, quality, and performance.
>
> For different cases, the set of information referred to as telemetry data variates. For example, CPU and buffer utilization are valuable for virtualized environments. In other words, I understand telemetry data as information extracted from a remote system that characterizes the system's state. In any case, providing a definition would be helpful to a reader of the document.

[[[TM]]] This word is mentioned only once in the document (in the
abstract), and not sure adding a definition for this particular word
would change the readability of the document.

>>
>>
>>
>> >
>> > nit s/the the/the/
>>
>> [TM] Agreed.
>>
>> > I think that Abstract and Introduction can be improved by explaining the purpose of defining the Loopback and Active flags. What each of them is intended to achieve and improve?
>>
> As I understand the behavior required from a node processing the Loopback flag, the returned packet is not the complete but truncated data packet.
> Is "active measurement" used in the context of RFC 7799? If "yes", does that mean that the packet is a specifically generated test packet?

[[[TM]]] These are two good points. Based on these comments the
following text is proposed instead.
NEW:
The Loopback flag is used to request that each transit device along
the path loops back a truncated copy of the data packet to the sender. The
Active flag indicates that a packet is used for active measurement.
The term active measurement in the context of this document is as defined
in [RFC7799].

>> > Does that mean that neither of the two flags is applicable in the Direct Export option? Should draft-ietf-ippm-ioam-direct-export have an explicit statement regarding the applicability of the Loopback and Active flags? And if any is applicable, describe the behavior of an IOAM node receiving it.
>>
>> [TM] Right.
> GIM>> I'm a bit confused. Can you help me with the text from the DEX draft that specifies the use of the Loopback and Active flags in combination with the DEX trace option?

[[[TM]]] The DEX option does not support the loopback and active
flags. That is why the DEX document does not discuss these flags. The
DEX document defines the DEX option header including all the flags
that are relevant to the DEX option (loopback and active are not
included).

>> > In my previous comments, I've pointed out that the Loopback mode is just a special case of the Direct Export option. In addition to that, I would appreciate your explanation of how the IOAM encapsulating node receives per-hop data about the return path. How does that work, for example, in an MPLS or SFC NSH environment?
>>
> GIM>> Thank you for providing that comparison between the result of processing DEX and the Loopback. As I understand it, the result of DEX, i.e., immediate export or local processing with exporting a batch of telemetry data, is determined by the local policy. If that is the case, the local policy can produce the same result as for the Loopback flag, which could be the default behavior for the DEX.

[[[TM]]] Not exactly, because when using DEX in node X, the policy in
node X enables immediate exporting of the IOAM data of node X, but
does not enable exporting of the IOAM data of all the transit nodes
between the encapsulating node and node X. The former is possible with
DEX, but the latter is only possible with Loopback. This is just one
example of why Loopback is not a special case of DEX.

>> > But I don't see any technical explanation that could sustain that assumption. As I understand the diagram in figure 1, a transit node doesn't know the identity of the IOAM encapsulating node. Correct? If that is the case, where does the transit node direct the looped packet?
>>
> GIM>> Please help me understand, the looped back packet is transmitted using the same encapsulation as the data packet with the Loopback flag? What is the rationale? If we use MPLS LSP Ping as an example, the response can be sent in several ways, but the default is in IP/UDP encapsulation as the commonly available transport. Why not use that and send the looped packet over IP/UDP? I believe that the applicability of this mechanism will be much broader.

[[[TM]]] The rationale is to define very simple Loopback
functionality. Simplicity is an advantage as it enables hardware
implementations.

>>
>>
>>
>> > It seems that the text in the last paragraph attempts to walk around my questions. But I don't find that paragraph sufficient. It lacks the normative language to guide a developer to produce resilient implementation. For example, how to interpret:
>> > That reads like a recommendation, not a requirement. As a result, an implementation may transmit an IOAM packet with the Loopback fag set regardless of whether there's a return path or not. Then, how must a transit node process such an IOAM packet?
>> >
>>
> GIM>> Thank you for the proposed text. I'd note that referring to the "encapsulating node's address" and "source address" might be interpreted as applying only to IP. In some networks, for example, BIER, the identity of the IOAM encapsulating node, BFIR, is known from its identifier that, in turn, can be mapped to the IP address.

[[[TM]]] Right. Proposed text:
The loopback flag MUST NOT be set if it is not guaranteed that there
is a return path from each of the IOAM transit and IOAM decapsulating
nodes, or if the encapsulating node's identity is not available in the
encapsulation header.

>> > IOAM has been defined as a hybrid measurement protocol. This statement implies that it is to be used as an active OAM protocol (all per RFC 7799 definitions). The requirements for active OAM protocols in terms of their applicability and, most of, security are very different from those set for a hybrid measurement protocol. I believe that extending IOAM into an active mode of measurement requires more discussion and the audience from other IETF WGs, e.g., 6man, SFC, BIER, MPLS, BFD, RTGWG.
>>
> GIM>> I have to disagree. Per RFC 7799, the use of specially constructed test packets is the primary characteristic of an active measurement method. By introducing the Active flag, this drat makes the IOAM into an active measurement protocol with all the requirements to an active measurement protocol to be applied.

[[[TM]]] See note (1).

>> > That is the same use of ICMP and LSP Ping. What is the benefit of using the Loopback flag instead of readily available ICMP? Does it work as L2 Linktrace? I don't see that as a sufficient benefit considering the complexity, exclusions, and risks associated and identified in the draft. It is clear that there are tools like ICMP and LSP Ping that adequately address scenarios listed in the draft. Don't see any technical reason for the introduction of another mechanism that does not have any significant benefit compared to the existing OAM tools. Especially, considering the limited information that is allowed to be collected using the Loopback IOAM flag ("every transit node that processes this trace option only adds a single data field, which is the Hop_Lim and node_id data field"). With the abundance of information and additional verification achieved using ICMP and LSP Ping, there's even less rationale in the introduction of this over-complex and unnatural mechanism.
>>
> GIM>> Also, I must note that "pro-active" use of the Loopback is possible only as part of the active measurement protocol.

[[[TM]]] See note (1).

>> > I couldn't find any guidance on how an IOAM node that receives such invalid combination processes the packet.
>> >
>>
> GIM>> I am still not clear about "MUST NOT push an IOAM encapsulation with the Loopback flag" in the draft and your explanation "the encapsulating node does not set the Loopback flag". As I understand it, in the given network for the monitored data flow, only one IOAM encapsulating node can exist. If that is correct, where does the existing IOAM encapsulation come from?

[[[TM]]] The scenario that was raised by Martin Duke in his security
review is a nested IOAM scenario, where traffic in an IOAM domain is
forwarded through another domain/tunnel, which is also IOAM-enabled.
This is explained in the draft:

   This requirement is intended to prevent IOAM Loopback
   nesting, where looped back packets may be subject to loopback in a
   nested IOAM domain.

>>
>>
>> >
>> > It is assumed that the identity of the IOAM encapsulating node is known to the node that is expected to sent a copy of the received packet back. But what if that is not the case? How must that node act?
>>
> GIM>> I think that points to the significant limitation of the proposed mechanism. Comparing with MPLS LSP Ping, we see that it allows tracing the LSP even though MPLS encapsulation does not identify the ingress LER while the proposed Loopback would not work at all.

[[[TM]]] See note (1).

>>
>>
>> > I believe that there's inconsistency between Figure 1 and the following text in Section 4.2
>> >
>> >    Thus, the
>> > It seems that the text assumes that the IOAM encapsulating node adds also a network layer encapsulation. But how would it work, for example, in SFC NSH data plane? Or MPLS?
>>
>> > But according to the definition of node_id in Section 5.4.2.1 of draft-ietf-ippm-ioam-data, "Node identifier field to uniquely identify a node within the IOAM-Namespace and associated IOAM-Domain." It appears that simply comparing Node IDs is not sufficient to verify that the node received its looped back packet. What is your opinion?
>>
>> > Then the question, What is the purpose of the Active flag? Existing active protocols, Fault Management and Performance Monitoring, are identified by, for example, use of a well-known destination port number or, in case of MPLS, Generic Associated Channel type. There's no apparent need of using IOAM option header to identify the payload as a control message of an active OAM protocol. On the contrary, that would likely cause more confusion and challenges to interoperation among different implementations.
>> >
>> > Three cases analyzed in Section 5 require special consideration, as these likely present the motivation for the introduction of the Active flag:
>> >
>> > the first use case is not really using the Active flag at all and thus the text doesn't seem adding any value to the draft;
>> > it is not clear what is viewed as the benefit of using the Active flag compared with using any of the existing active OAM protocols in the second use case. I would appreciate it if you can clarify that for me. Since an SFC NSH network is used as an example, is it required that a Service Function (SF) or an SF Proxy process the IOAM header with Active flag set? Or would a Service Function Forwarder (SFF) avoid sending a packet with the IOAM header that has an Active flag set? And can you compare that with the procedures defined in the Active OAM for SFC draft?
>> > My questions for the third use case described in Section 5 are the same as above because I don't see the benefit of using a replicated data packet (less benefits if it is truncated) compared to a special test packet.
>> >
>> > In my conclusion on the Active flag, I don't see any clear benefit of introducing it and using whether in place or in addition to the existing active OAM protocols.
>>
> GIM>> I have to disagree with you here. I don't find the explanation in the document sufficient to justify the introduction of another mechanism to achieve what has already been satisfactorily addressed in IETF. Without a more detailed specification of the problem solved by the Active flag and how that is achieved, this mechanism appears like the answer is looking for a question, like a loaded gun without safety.

[[[TM]]] See note (1).

---

From: Tal Mizrahi <[email-masked]>
Date: Wed, 13 Oct 2021 15:24:19 +0300

Minor correction - here is the correct link to the updated version of the draft:
https://datatracker.ietf.org/doc/draft-ietf-ippm-ioam-flags/
:)
Cheers,
Tal.

---

From: Greg Mirsky <[email-masked]>
Date: Sat, 16 Oct 2021 15:44:44 -0700

Hi Tal,
thank you for uploading the new version of the draft. I appreciate your
consideration of my comments and believe that we're converging, though I
have some questions that concern me:

   - The added text sets new restrictions on the applicability of the
   Loopback flag:

   The loopback flag MUST NOT be set if it is not guaranteed that there
   is a return path from each of the IOAM transit and IOAM decapsulating
   nodes, or if the encapsulating node's identity is not available in
   the encapsulation header.

Can you clarify how an encapsulating node can guarantee "that there is a
return path from each of the IOAM transit and IOAM decapsulating nodes"
when setting the Loopback flag for the first time? I would imagine that for
the first packet marked with the Loopback flag, the encapsulating node may
only know the identity of the decapsulating node, not of every node that
the marked packet will traverse. Also, what is understood as "a return
path"? Is that a path with the same encapsulation as the marked with the
Loopback flag packet or any type?

And I also have a concern about the scope of the applicability of the
Loopback flag by the "MUST NOT ...  if the encapsulating node's identity is
not available in the encapsulation header". I believe that among
encapsulation types defined at IETF, that requirement automatically
excludes MPLS and SFC NSH. Depending on the interpretation of "a return
path", BIER MPLS might be excluded too. As a result, it appears that the
Loopback flag maybe only applicable in an IPv6 network. If that is the
case, I cannot see a reason to use a constrained resource (space in the
Flags field) for a very limited application.

   - Another update clarifies that:

   If the address of the encapsulating
   node is not available in the encapsulation header, then the transit/
   decapsulating node does not loop back a copy of the original packet.

That seems directly related to the requirement I've pointed out above. I
have a question about the next text:

   - How does a node "knows" if the address of the encapsulating node is
   available? I can imagine a situation, for example, SFC NSH over an IPv6
   network. In that case, the encapsulation includes the source address, but
   how the node processing Loopback flag knows that that is the address of the
   encapsulation node?

   - And the next explanation of the operation brought a new question:

   The address of the node performing the copy operation is used as the
   source address.
It is not clear whether it is a requirement or a recommendation. Also, it
appears that the draft assumes that a looped back packet is always
encapsulated to convey the sender's identity. But, as I've noted earlier,
that is the case for a very few types of encapsulation defined at IETF.

   - For our discussion of the use of the Active flag, it seems as you've
   added only the following:

           The Active flag indicates that a packet is used for active
     measurement.  The term active measurement in the context of this
     document is as defined in [RFC7799].

I firmly believe that the document does not provide sufficient information
about types of active measurement in which a packet with the Active flag is
recommended or required to be used. In that situation, because the use of
the Active flag is clearly underdefined, I suggest its definition and all
references be removed from the draft. Once there is a better understanding
of a type of active measurement that benefits from using this method, the
definition of the Active flag would be appropriate.

Regards,
Greg

On Wed, Oct 13, 2021 at 5:24 AM Tal Mizrahi <[email-masked]>
wrote:

> > > thank you for your responses to my comments and questions. Please find
> my follow-up notes in-lined below under the GIM>> tag.
> > >
> > > Regards,
> > > Greg
> > >
> > >> > I've read the latest version of the draft and have several comments
> and questions to share with you. I would much appreciate the opportunity to
> > > GIM>> I think that there's no standardized definition of "telemetry"
> and "telemetry information". I share the idea expressed in the following:
> > >
> > > For different cases, the set of information referred to as telemetry
> data variates. For example, CPU and buffer utilization are valuable for
> > >> > I think that Abstract and Introduction can be improved by
> explaining the purpose of defining the Loopback and Active flags. What each
> of them is intended to achieve and improve?
> > >>
> > > As I understand the behavior required from a node processing the
> > > Is "active measurement" used in the context of RFC 7799? If "yes",
> > >> > Does that mean that neither of the two flags is applicable in the
> Direct Export option? Should draft-ietf-ippm-ioam-direct-export have an
> explicit statement regarding the applicability of the Loopback and Active
> flags? And if any is applicable, describe the behavior of an IOAM node
> receiving it.
> > >>
> > >> [TM] Right.
> > > GIM>> I'm a bit confused. Can you help me with the text from the DEX
> draft that specifies the use of the Loopback and Active flags in
> combination with the DEX trace option?
> >
> > >> > In my previous comments, I've pointed out that the Loopback mode is
> just a special case of the Direct Export option. In addition to that, I
> would appreciate your explanation of how the IOAM encapsulating node
> receives per-hop data about the return path. How does that work, for
> example, in an MPLS or SFC NSH environment?
> > >>
> > >> > It seems that the text in the last paragraph attempts to walk
> around my questions. But I don't find that paragraph sufficient. It lacks
> the normative language to guide a developer to produce resilient
> > >> > That reads like a recommendation, not a requirement. As a result,
> an implementation may transmit an IOAM packet with the Loopback fag set
> > >> > IOAM has been defined as a hybrid measurement protocol. This
> statement implies that it is to be used as an active OAM protocol (all per
> RFC 7799 definitions). The requirements for active OAM protocols in terms
> of their applicability and, most of, security are very different from those
> set for a hybrid measurement protocol. I believe that extending IOAM into
> an active mode of measurement requires more discussion and the audience
> from other IETF WGs, e.g., 6man, SFC, BIER, MPLS, BFD, RTGWG.
> > >>
> > >> > That is the same use of ICMP and LSP Ping. What is the benefit of
> using the Loopback flag instead of readily available ICMP? Does it work as
> L2 Linktrace? I don't see that as a sufficient benefit considering the
> > > GIM>> Also, I must note that "pro-active" use of the Loopback is
> possible only as part of the active measurement protocol.
> >
> > [[[TM]]] See note (1).
> >
> > >>
> > >>
> > >> >
> > > GIM>> I am still not clear about "MUST NOT push an IOAM encapsulation
> with the Loopback flag" in the draft and your explanation "the
> encapsulating node does not set the Loopback flag". As I understand it, in
> the given network for the monitored data flow, only one IOAM encapsulating
> node can exist. If that is correct, where does the existing IOAM
> encapsulation come from?
> >
> > >> > It is assumed that the identity of the IOAM encapsulating node is
> known to the node that is expected to sent a copy of the received packet
> back. But what if that is not the case? How must that node act?
> > >>
> > > GIM>> I think that points to the significant limitation of the
> proposed mechanism. Comparing with MPLS LSP Ping, we see that it allows
> tracing the LSP even though MPLS encapsulation does not identify the
> ingress LER while the proposed Loopback would not work at all.
> >
> > [[[TM]]] See note (1).
> >
> > >>
> > >>
> > >> > It seems that the text assumes that the IOAM encapsulating node
> adds also a network layer encapsulation. But how would it work, for
> example, in SFC NSH data plane? Or MPLS?
> > >>
> > >> > the first use case is not really using the Active flag at all and
> thus the text doesn't seem adding any value to the draft;
> > >> > it is not clear what is viewed as the benefit of using the Active
> flag compared with using any of the existing active OAM protocols in the
> second use case. I would appreciate it if you can clarify that for me.
> Since an SFC NSH network is used as an example, is it required that a
> Service Function (SF) or an SF Proxy process the IOAM header with Active
> flag set? Or would a Service Function Forwarder (SFF) avoid sending a
> packet with the IOAM header that has an Active flag set? And can you
> compare that with the procedures defined in the Active OAM for SFC draft?
> > >> > My questions for the third use case described in Section 5 are the
> same as above because I don't see the benefit of using a replicated data
> packet (less benefits if it is truncated) compared to a special test packet.
> > >> >
> > >> > In my conclusion on the Active flag, I don't see any clear benefit
> of introducing it and using whether in place or in addition to the existing
> active OAM protocols.
> > >>
> > > GIM>> I have to disagree with you here. I don't find the explanation
> in the document sufficient to justify the introduction of another mechanism
> to achieve what has already been satisfactorily addressed in IETF. Without
> a more detailed specification of the problem solved by the Active flag and
> > >> >> Please provide feedback by replying to this email with your
> reviews and if you think these documents are ready to progress, by
> Wednesday, September 15.
> > >> >>
> > >> >> Best,
> > >> >> Tommy & Ian
> > >> >>

---

From: Greg Mirsky <[email-masked]>
Date: Sat, 16 Oct 2021 16:09:31 -0700

Hi Tal,
thank you for posting a new version with updates resulting from our
discussion. I greatly appreciate the consideration you extended to my
comments. As I've mentioned, my comments are non-blocking further progress
of this document. I've read the new version and have some suggestions you
can consider for the next update of the draft:

   - I think that the added text can be a recommendation as the out-of-band
   transport has a lesser adverse impact on the customer traffic:

           It should be noted that in some networks DEX data may be exported
     over an out-of-band network, in which a large volume of exported
     traffic does not compromise user traffic.

   - I propose a minor editorial update as follows:

OLD TEXT:
           In this case an operator
     may choose to disable the exporting rate limiting.
NEW TEXT:
           In this case an operator
     may choose to relax the exporting rate limiting.

   - A new requirement added to the Security Consideration is very useful
   but it seems to suggest a particular implementation:

           Furthermore, an IOAM node MUST
     gain explicit consent to export data to a receiving entity before
     starting to send exported data.

I can imagine that an orchestrator, not an exporting telemetry information
node, obtains the required consent. The sentence might be re-worded as
follows:

           Furthermore, explicit consent to export data to a receiving
entity
           MUST be obtained before activating the IOAM Direct Export trace
           option.

Regards,
Greg

On Wed, Oct 13, 2021 at 5:12 AM Tal Mizrahi <[email-masked]>
wrote:

> and questions to share with you. None of these are blocking, I support
> >> > It seems that the characterization of the Direct Export IOAM option
> in the following is not entirely accurate:
> >> >
> > GIM>> As I understand it, it is the local policy that controls the
> behavior of the node receiving the IOAM DEX indicator. Similar to IPFIX,
> the information can be aggregated and exported as a calculated metric,
> e.g., percentile or mean value, or exported as soon as possible as
> recorded. The new text is certainly better but it seems it does not express
> the role of the local policy in defining the processing behavior, the
> transport to be used, and the identity of the Collector. One of possible
> ways could be to state that the definition of all these controls is outside
> the scope of the document.
>
> >> > I find references to RFC 5475 and 7014 very appropriate and helpful.
> But I don't see these RFCs being referenced in draft-ietf-ippm-ioam-data. I
> >> > Similarly, the good and helpful discussion of how to practically
> control the use of the Direct Export option is, in my opinion, equally
> applicable to other IOAM trace options and should be in the Security
> Considerations section of draft-ietf-ippm-ioam-data.
> >>
> >>
> >> > I think that it will be helpful to provide some recommendations,
> i.e., SHOULD do this, how an implementation handles the excess of packets
> to export. Drop silently, drop and log notification, or try to shape that
> flow?
> >>
> > GIM>> If the definition of exporting, including the encapsulation,
> choice of transport, and the Collector is outside the scope, then this
> statement, in my opinion, should be removed. If all these issues are in the
> scope, then  there is more, much more information that is missing.
>
> > GIM>> Thank you for the clarification. Is that the intention to start
> new drafts to define the DEX IOAM Option-Type separately?
>
> >> > But the figure 2 in the draft does not include any field that
> reflects the length of the IOAM option. Is that because the figure is out
> of sync? If not, how can an implementation verify the length? Also, what is
> the behavior if the length is less than 8 octets?
> >> >
> >>
> >> [TM] Quoting the draft:
> >>
> >> >> Please provide feedback by replying to this email with your reviews
> and if you think these documents are ready to progress, by Wednesday,
> September 15.
> >> >>
> >> >> Best,
> >> >> Tommy & Ian
> >> >>