Subject: Re: [alto] AD review of draft-ietf-alto-performance-metrics-15

From: "Y. Richard Yang" <[email-masked]>
Date: Fri, 2 Apr 2021 21:18:21 -0400

Hi Martin,

Thank you so much for the wonderful review. Please see below.

On Mon, Mar 29, 2021 at 2:35 PM Martin Duke <[email-masked]> wrote:

> Hello authors,
>
> Thank you very much for writing this draft. It is clearly a useful
> extension to ALTO and is quite clearly written, even to someone who is not
> a practitioner. I have numerous comments/questions and a few nits.
>
> These points are all invitations to discussion, rather than commands about
> what to change, as I've missed much of the WG deliberations that led to
> this text.
>
> COMMENTS:
> - There are six authors. Having more than 5 editors/authors listed on the
> front page requires strong justification and chair/AD approval. See the RFC
> Editor statement [1]. You are encouraged to designate a few editors for the
> front page and list as many authors as desired at the end.
>
>
I saw in the next email that this could be OK. Thanks for the follow-up
note.

> - Sec 2.1. The cost-source model is conceptually sound, but the
> justification for it seems underexplained. What exactly is a client going
> to do with this information? What different behaviors would a client
> execute if the context was e.g. "sla" instead of "nominal?" To the extent
> the parameters are not machine readable, like links to webpages, are we
> really expecting this information to be presented to the humans behind ALTO
> clients?
>

Good comment. This is a point which the WG indeed discussed multiple times.
The decision was that providing the source will give the server the ability
to indicate the source, and the client knows more about the context. Some
operational aspects are discussed in Sec. 5.1. There were some discussions
on potential normative specifications on the client-side and/or the
server-side, but the final decision is that ALTO provides non-normative
information. For example, the basic cost values are non-normative, and only
best-effort information. There can be out-of-ALTO control loops, for
example, business contracts, and the cost-source supports indication of
such information channel.

The comment on clarifying whether the info will be for humans behind is a
great one. One use case setting discussed is that the link can provide
machine-readable specifications such as a machine-readable language
specifying the measurement parameters, but this is considered out of scope.
Will you recommend that we include more some more elaboration in the
operational considerations section (i.e., extending Sec. 5.1)?

>
> - Sec 2.1 I am confused about the meaning of the "sla" cost-source. Does
> this refer to an SLA the ALTO client has with the network? Between the
> target IP and the network? Or something else? If the first, does this link
> to client authentication in some way? If the second, what are the privacy
> implications of exposing these SLAs?
>

It is target IP and the network. Here is some text in the current version
on the authentication and privacy aspects (Sec. 6):
"Indeed, TE
   performance is a highly sensitive ISP information; therefore, sharing
   TE metric values in numerical mode requires full mutual confidence
   between the entities managing the ALTO server and the ALTO client.
   ALTO servers will most likely distribute numerical TE performance to
   ALTO clients under strict and formal mutual trust agreements.  On the
   other hand, ALTO clients must be cognizant on the risks attached to
   such information that they would have acquired outside formal
   conditions of mutual trust."

Will this be OK?

- Sec 2.1. Related to the above, the text suggests that any cost-source
> expressed as "import" could also be expressed as "estimation". Why would
> the server do this? The text should say, or perhaps it would be
> conceptually cleaner if "estimation" and "import" were mutually exclusive
> sources by definition.
>

In the early WG discussion, they were considered separate, and then the
agreement was that import is a special case of estimation, with more
specific dependency tracking. Consider data provenance of how the ALTO data
are computed. Estimation means that the server does not want to indicate
the specific details, and the important gives a precise indication of the
exact protocols.

> - Sec 3. I would prefer it if the parameters field in each of these
> definitions was a bit more strict. This relates to my confusion about
> machine-readable vs. human readable data; if this is meant to be
> machine-readable, then e.g. Sec 3.4.4 should be more specific in spelling
> out that the IGP protocols should be in a format with the RFC number, for
> instance. If it's to be human readable for a purpose I don't understand,
> then these looser definitions are probably OK.
>
>
This is a very good comment as well! One possibility discussed was
following (the format such as those specified in
https://tools.ietf.org/html/draft-ietf-ippm-metric-registry-24#section-7.1.2)
using a format such as RFCXXXXsecY, but such very specific specification
will cause moving dependency. Hence, the decision was to make the link
opaque to be agreed between the client and the server.

> - Sec 3.4 Unlike the other metrics, I have no idea what a client is to do
> with the hop count metric, since clients don't care about hop count. Hops
> only affect users through delay and loss rate, which is present in other
> metrics. Is the intent here to provide a proxy for delay when direct delay
> information is not available? If so, we should say so.
>
>
Yes. There are some settings such as a data center where such a proxy
appears to be reasonable.

> - Sec 5.3. I suggest a reword.
>
> OLD:
> To address this issue, the only defined "routingcost" metric can be
>    only "estimation".
>
> NEW:
> To address this issue, if the "routingcost" metric contains a cost-context
> field, it MUST be "estimation."
>
>
Sounds good.

> What should clients do if it's not "estimation?" Can they use it or reject
> the metric
> as malformed?
>
>
The basic design guideline of ALTO information is that if the client does
not understand or trust the information, the client should just ignore it.
The base protocol (RFC7285) discussed false or modified (RFC 7285 Sec.
15.1.1) and undesirable (RFC7285, Sec. 15.2.1). The basic guideline
(RFC7285, Sec. 8.3.7, "ALTO implementations MUST ignore unknown fields when
processing ALTO messages."). So I will say reject if not a valid source.

> - Sec 5.4.1: "...the ALTO server may provide the client with the validity
> period of the exposed metric values."
>
> Shouldn't there be a standard format for this? Or are you implying the use
> of cost-calendar?
>

Good catch. The decision of the WG at the time was to use HTTP whenever
possible. For example, the freshness is indicated by HTTP timestamp (see
Sec. 5.2); by consistency, then, we should use HTTP Expires. We can add
this. Agree?

>
> - Sec 5.4.2: I don't understand what this section is saying. Can the
> server provide new metrics not in the spec? Or is it saying that the server
> can take singletons about link one-way delays and compose path one-way and
> two-way delays, for example?
>
>
Your second understanding (from link to path composition) is correct.
Reading this paragraph again, I agree that it can be revised a bit to
clarify. The basic issue is that the "source" information can often be link
level. For example, routing protocols often measure and report only per
link loss, not end-to-end loss; similarly, routing protocols report link
level available bandwidth, not end-to-end available bandwidth. We can take
a quick pass to make sure it has only the second understanding.

> NITS:
> - Sec 1. An initial sentence introducing ALTO at the beginning would be
> helpful, e.g.
>
> "ALTO [RFC 7285] provides a means for client to identify the most
> efficient information source when multiple copies of such information
> exist, by quering path information from an HTTP server."
>
>
Sounds good. We will revise.

> - Sec 2. The second paragraph is a little hard to read. Suggestion:
>
> OLD:
>
> On the other hand, to be able to use coarse-grained information such
>    as routing system information (e.g., [RFC8571 <https://datatracker.ietf.org/doc/html/rfc8571>]), which may not
>    provide fine-grained information such as (iii) Method of Measurement
>    or Calculation and (vi) Measurement Timing, this document provides
>    context information to indicate the source of information and hence
>    available metric details.
>
> NEW:
>
>   This document specifies context information to indicate the metric
> source, which can allow clients to obtain fine-grained information like
> (ii) Method of Measurement or Calculation and (vi) Measurement Timing."
>
>
OK. Thanks!

> - Sec 2.1 In Fig. 1, please expand "NBI" on first use.
>
>
Jan mentioned this as well, but it was fixed in a version which we did not
upload it. Will do in the next upload.

> - Sec 3.1.3 Expand "PID" on first use.
>
>
Will do.

> - Sec 3.1.4 s/recommended/RECOMMENDED
>
>
OK.

> - Sec 3.4 s/metric hopcount/hopcount metric
>
>
OK.

> - Sec 4.1.3 would this be correct: s/give the throughput/give the maximum
> throughput
>
>
The intention is the TCP/Reno-friendly throughput. References include
RFC3649 and Sec. 5.1 of RFC 8312.

- Sec 6. s/is a highly sensitive/is highly sensitive
Sounds good.

Thank you so much!

Richard

> Thanks
> Martin
>
> [1] https://www.rfc-editor.org/pipermail/rfc-interest/2015-May/008869.html
> _______________________________________________
> alto mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/alto
>

---

From: "Y. Richard Yang" <[email-masked]>
Date: Fri, 2 Apr 2021 21:20:20 -0400

Hi Martin,

On Tue, Mar 30, 2021 at 2:24 PM Martin Duke <[email-masked]> wrote:

> Sorry for the fragmented review, but there are a couple of more issues:
>
> - The authors should do a review of all lower-case occurrences of must,
> should, may, and recommended. At least a few of them should be capitalized
> to indicate normative requirements.
>

Yes. We are taking a pass and are almost done.

>
> - IMO, from a quick review,  I-D.ietf-ippm-initial-registry as written is
> normative and should be listed as such. However, I think it would be better
> to simply refer to the actual registry (
> https://www.iana.org/assignments/performance-metrics/performance-metrics.xhtml)
> rather than tie it to the initial entries.
>
> Agree that referring to the registry is a better design and we will do the
normative reference in the update.

Thanks!
Richard

> On Mon, Mar 29, 2021 at 5:30 PM Martin Duke <[email-masked]>
> wrote:
>
>> One small correction: I'm jumping the gun on the author policy; 6 is
>> probably OK for now.
>>
>> On Mon, Mar 29, 2021 at 11:33 AM Martin Duke <[email-masked]>
>> wrote:
>>
>>> Hello authors,
>>>
>>> These points are all invitations to discussion, rather than commands
>>> about what to change, as I've missed much of the WG deliberations that led
>>> to this text.
>>>
>>> COMMENTS:
>>> - There are six authors. Having more than 5 editors/authors listed on
>>> the front page requires strong justification and chair/AD approval. See the
>>> RFC Editor statement [1]. You are encouraged to designate a few editors for
>>> the front page and list as many authors as desired at the end.
>>>
>>> - Sec 3.4 Unlike the other metrics, I have no idea what a client is to
>>> do with the hop count metric, since clients don't care about hop count.
>>> Hops only affect users through delay and loss rate, which is present in
>>> other metrics. Is the intent here to provide a proxy for delay when direct
>>> delay information is not available? If so, we should say so.
>>>
>>> To address this issue, if the "routingcost" metric contains a
>>> cost-context field, it MUST be "estimation."
>>>
>>> What should clients do if it's not "estimation?" Can they use it or
>>> reject the metric
>>> as malformed?
>>>
>>> - Sec 5.4.1: "...the ALTO server may provide the client with the
>>> validity period of the exposed metric values."
>>>
>>> Shouldn't there be a standard format for this? Or are you implying the
>>> use of cost-calendar?
>>>
>>> - Sec 4.1.3 would this be correct: s/give the throughput/give the
>>> maximum throughput
>>>
>>> [1]
>>> https://www.rfc-editor.org/pipermail/rfc-interest/2015-May/008869.html
>>>

---

From: Martin Duke <[email-masked]>
Date: Mon, 5 Apr 2021 08:03:07 -0700

Hi Richard,

Where needed, responses inline.

On Fri, Apr 2, 2021 at 6:18 PM Y. Richard Yang <[email-masked]> wrote:

> Good comment. This is a point which the WG indeed discussed multiple
> times. The decision was that providing the source will give the server the
> ability to indicate the source, and the client knows more about the
> context. Some operational aspects are discussed in Sec. 5.1. There were
> some discussions on potential normative specifications on the client-side
> and/or the server-side, but the final decision is that ALTO provides
> non-normative information. For example, the basic cost values are
> non-normative, and only best-effort information. There can be out-of-ALTO
> control loops, for example, business contracts, and the cost-source
> supports indication of such information channel.
>

If it were me, I'd put some of it in the intro and some motivation in 2.1,
and maybe go through some miscellaneous considerations in 5. But that's
less important that it be there somewhere.

If the intent is that it be machine-readable, then there are several places
where this standard is going to need more standardization (i.e. precise
definition of text fields).

But zooming out: I understand that the point is that "the client knows more
about the context," which is pretty much what 5.1 says. But I don't
understand if the "client" is the user or a user agent, and what either one
would actually do with the information. Would the application execute a
policy based on the source? Why would it use a latency that came from an
sla, but not from measurement? etc.

>    conditions of mutual trust."

That privacy information is alright, but exposing the details of
third-party SLAs deserves special attention. But to follow up your answer:
if the client has a better SLA than the target, this won't show up in the
metrics at all?

> the specific details, and the important gives a precise indication of the

OK, I now understand that "import" implies a specific set of parameters. I
can't understand what value this distinction has, but that just circles
around to me not understanding the cost-source information at all.

Sure.

Martin

---

From: "Y. Richard Yang" <[email-masked]>
Date: Thu, 8 Apr 2021 16:04:16 -0400

Hi Martin,

Thanks for the quick feedback. Please see below.

On Mon, Apr 5, 2021 at 11:03 AM Martin Duke <[email-masked]> wrote:

> Thanks for the note. We are adding a few sentences in 5.

> If the intent is that it be machine-readable, then there are several
> places where this standard is going to need more standardization (i.e.
> precise definition of text fields).
>

Some of the authors discussed the issue and feel that going down the path
of making the content machine-readable, in a systematic way, adds
substantial complexity.
.

> But zooming out: I understand that the point is that "the client knows
> more about the context," which is pretty much what 5.1 says. But I don't

This is a good comment. Here is an example (
https://ipnetwork.bgtmo.ip.att.net/pws/averages.html) that motivated some
early discussions. In this example, AT&T post both target (aka sla. should
we change the name from sla to service-level-target?) and actual
measurements. In this sense, ALTO can be considered as a standard way of
providing and update the information. Both target and actual values can be
useful. Make sense?

> third-party SLAs deserves special attention.
>

Yes.

> But to follow up your answer: if the client has a better SLA than the
> target, this won't show up in the metrics at all?
>

Now I see that I need to clarify. The metric is end-to-end, from src IP to
dst (target) IP. Does this clarify? I can add a sentence to clarify this.

> around to me not understanding the cost-source information at all.

I see. Let me try to clarify slightly differently. import means that the
ALTO server can provide a precise source of information using specific
parameters, and estimate is that it comes from a black-box (the server does
not reveal). Thinking about it a bit more, we can go down the path of
specifying a precise format (rfc/section just as ippm) when specifying
import. Will this be a direction that you want to go?

> Sec. 5.2); by consistency, then, we should use HTTP Expires. We can add
Thanks a ton!

Richard