Subject: Re: [Gen-art] Gen-ART review of draft-ietf-aqm-eval-guidelines-11

From: Kuhn Nicolas <[email-masked]>
Date: Tue, 17 May 2016 08:31:08 +0000

Thanks a lot for this review. We have proposed answer to the different points and hope that this is more clear. When we believe that it was needed, we have suggested text changes. If you think that this should result in the submission of a new ID, please let us know. 

Kind regards, 

The authors

-----Message d'origine-----
De : Ralph Droms (rdroms) [mailto:[email-masked]] 
Envoyé : jeudi 28 avril 2016 21:23
À : [email-masked]
Cc : [email-masked]
Objet : Gen-ART review of draft-ietf-aqm-eval-guidelines-11

I am the assigned Gen-ART reviewer for this draft. The General Area Review Team (Gen-ART) reviews all IETF documents being processed by the IESG for the IETF Chair.  Please treat these comments just like any other last call comments.

For more information, please see the FAQ at

<http://wiki.tools.ietf.org/area/gen/trac/wiki/GenArtfaq>.

Document: draft-ietf-aqm-eval-guidelines-11
Reviewer: Ralph Droms
Review Date: 2016-04-28
IETF LC End Date: 2016-05-04
IESG Telechat date: 2016-05-19

Summary: This draft is on the right track but has open issues, described in the review.

In general, I think the document could be read, implemented and used to generate useful characterizations of AQM schemes.  However, the motivations for some of the measurements and scenarios seems weak to me, which might compromise the weight given to the conclusions drawn from the guidelines.

Major issues:

None.  However, the list of minor issues and nits, taken together, could be considered a major issue to be resolved before publication.

Minor issues:

#####

I often react to the use of RFC 2119 language in an Informational document by asking is that language really necessary?  I'll ask the question here: in the context of this Informational document, which appears to be entirely advisory in providing guidelines, what does the use of RFC 2119 "requirements language" add to the meaning of the document.

>> Authors: Indeed, the use of RFC 2119 language is not mandatory for such information document. However, using it enables us to introduce weight in the different parameterizations of the tests. Even though, it is not mandatory, we believe that it eases the reading of the document, for someone familiar with the IETF wording.

#####

Figure 1 is not clear to me.  Where are the physical links and interfaces?  Are there multiple physical senders and receivers or are "senders A" instantiated on a single host (does it make a difference)?  Are there static-sized buffers for each interface or do all the buffers share one memory space?

>> Authors: We acknowledge that Figure 1 is not very clear. We have voluntarily omitted precisions on the amount of senders, receivers and traffic classes since the instantiation on a specific testbed would remove the generality of the figure and the described architecture. We believe that the text helps in reading the figure. Also, the rationale of this figure is to explain the notation more than going deeper in the topology that is anyway very generic.    

#####

In section 3.1, is there a need to say something about the relative capacities of the various links and the rates at which the various flows generate traffic?

>> Authors: These capacities are described in a later section when needed, and to remain high level and not focus on any applicability context (wi-fi, rural satellite access, fibber access, etc.) they are not specified for the whole document. The rates at which the flows generate traffic is specified for each further described scenario.

#####

I would have trouble following the guidelines set out in section 4.3.1.  I can understand the need for consideration of the tunable control parameters when comparing different AQM schemes.  However, I don't know what "comparable" means for control parameters that are likely quite different between AQM schemes.  I also think one would want to compare optimal control settings for the different schemes, to compare best-case performance.  Or, for AQM schemes whose performance is highly dependent on operational conditions, one might want to compare settings that are sub-optimal for any particular test condition but that give better performance over a wide range of conditions.

>> Authors: The intent of the first recommendation is to make testers be aware of which control points control which behavior and be conscious to make apples to apples comparison. 
To further precise this, we could change the text is section 4.3.1 as follows :
"1. Similar control parameters and implications: Testers should be aware of the control parameters of the different schemes that control similar behavior. Testers should also be aware of the input value ranges and corresponding implications. For example, consider two different schemes - (A) queue-length based AQM scheme, and (B) queueing-delay based scheme. A and B are likely to have different kinds of control inputs to control the target delay - target queue length in A vs. target queuing delay in B, for example. Setting parameter values such as 100MB for A vs. 10ms for B will have different implications depending on evaluation context.  Such context-dependent implications must be considered before drawing conclusions on performance comparisons. Also, it would be preferable if an AQM proposal listed such parameters and discussed how each relates to network characteristics such as capacity, average RTT etc."

#####

Section 4.4 seems to give advice to the AQM designer rather than describe guidelines for characterization.  Section 4.4 should either be rewritten to give guidelines for structuring measurements to account for varying packet sizes or the section should be elided.

>> Authors: We could to modify text for 2nd paragraph of 4.4, if you think that this clarifies the issue.
" An AQM scheme SHOULD adhere to the recommendations outlined in [RFC7141], and SHOULD NOT provide undue advantage to flows with smaller packets [RFC7567]. In order to evaluate if an AQM scheme is biased towards flows with smaller size packets, sender A in Figure 1 can be instantiated with two long standing TCP flows with different packet sizes - 500 bytes vs. 1500 bytes, respectively and metrics such as goodput, loss rate can be compared for these two flows. "

#####

In section 4.5, what is the motivation for giving the advice about ECN to AQM designers?  I can understand that ECN will have affect the impact of AQM, but for this document I think the section should focus on measurement guidlines that account for that impact.

>> Authors: The scope of introducing this part is mostly related to remain the tester that if their AQM supports ECN, this must be presented, since it could be seen as an important move for the deployment of ECN

#####

The specific topology in section 10 does not seem well-motivated to me.  Why is router R with no AQM included in the topology?  The choice of measurements is similarly not well-motivated.  Why would it not be of interest to run all the tests described earlier in the document?

>> Authors: It is worth pointing out that this specific topology is just a suggestion. The router without AQM has been included to remind the tester that the positioning of multiple AQMs have to be looked at with routers that do not have AQMs in mind. Since the placement of AQMs is mostly expected to be on "the last mile", we believe that evaluating all the tests described earlier in the document may not be needed. However, in case of multiple AQMs, their interactions should be considered.

#####

Nits/editorial comments:

There are several instances of the word "advice" which should be replaced with "advise"; e.g., in section 2.3.

>> Authors: Thanks.

#####

Last sentence of the abstract: I don't get the meaning of "precautionary characterizations of AQM schemes".  I recommend that the phrase be reworded

>> Authors: After another read of the abstract, we believe that the "precautionary" may not be needed. What about : « characterizations of AQM schemes".  

#####

Section 1, first paragraph: The last sentence doesn't follow the rest of the paragraph and I recommend that it be elided.

>> Authors: IMHO, we may even remove that sentence.

#####

Section 1, third paragraph: This text is redundant with the text in the Glossary section:

   When speaking of a specific queue in this
   document, "buffer occupancy" refers to the amount of data (measured
   in bytes or packets) that are in the queue, and the "maximum buffer
   size" refers to the maximum buffer occupancy.

>> Authors: It may be clearer to remove that sentence.

#####

Section 1, third paragraph:

OLD:

   In real
   implementations of switches, a global memory is often shared between
   the available devices, and thus, the maximum buffer size may vary
   over the time.

NEW:

   In switches and routers, a global memory space is often shared
   between the available interfaces, and thus, the maximum buffer size
   for any given interface may vary over the time.

>> Authors: Thanks.

#####

Section 1, fifth paragraph, last sentence: Is this document just concerned with "deployability" or more generally with "applicability, performance and deployability"?

>> Authors: We believe that the « deployability » involves the performance and the applicability, but we can further specify if you think it is important.

#####

Section 1.1, first paragraph: Would it be helpful to qualify "goodput" as "goodput in individual flows", to contrast with "goodput at a router"?  If "goodput" is well-known in this community to be "flow goodput", no change is needed.

>> Authors: IMHO, it is well-known to be flow goodput.

#####

Section 1.1, second paragraph: What is "BDP", as in "BDP-sized buffer"?

>> Authors: Indeed, the BDP has not been defined. We could add it in the glossary ?

#####

Section 1.1, third paragraph, first sentence: "environment" is unclear; perhaps "deployment scenario" or "use case"?  I'm not sure what was meant.

>> Authors: Indeed, deployment scenario may be clearer.

#####

Section 1.3: for completeness, a definition of "latency" would be useful.

>> Authors: We can add a definition of latency in the glossary.

#####

Section 2, first paragraph: I'm going to sound pedantic here, but it seems to me this section is not just intended "to better quantify (1) the reduction of latency, (2) maximization of goodput and (3) the trade-off between these two" but also to define other performance metrics that get to the goal from the abstract of describing "various criteria for performing precautionary characterizations of AQM schemes"

>> Authors: We believe that the "criterias for performaning characterizations" is covered by the last sentence of the paragraphe :
"This section provides normative requirements for metrics that can be
   used to assess the performance of an AQM scheme.
"  

#####

Section 2, last paragraph: What is "application-limited traffic"?  Later, what is "non application-limited" traffic?

>> Authors: We may propose a definition in the application limited traffic. A maybe more accurate name could be "Rate-Limited Traffic" ?

#####

Section 2.2: s/of/and/

>> Authors: Thanks.

#####

Section 2.5, first paragraph: s/induces/requires/  ???

>> Authors: Agree.

#####

Section 2.6, second paragraph is just not clear to me.  In particular, what is the antecedent of "they":

   The introduction of an AQM scheme would impact these
   metrics (end-to-end latency and jitter) and therefore they should be
   considered in the end-to-end evaluation of performance."

>> Authors: « They » refers the « these metrics ».

#####

Section 5.1: is this new text correct?

OLD:

   The transmission of the non application-limited flow must start
   before the transmission of the application-limited flow and only
   after the steady state has been reached by non application-limited
   flow.

NEW:

   The transmission of the non application-limited flow must start
   first and the transmission of the application-limited flow starts
   after the non application-limited flow has reached steady state.

>> Authors: Yes. It seems better to me.

#####

Throughout section 5, the text like "the graph described in Section 2.7 could be generated" seems redundant.

>> Authors: Indeed. We do not want readers to have to look at all the documents, but rather can directly jump on the test of their interest.

#####

Section 6.1, second paragraph: I don't understand the last sentence.  What is the "type of fairness" that the AQM scheme might affect?

>> Authors: « Type of fairness «  refers to the rtt fairness.

#####

Section 6.2, second paragraph: The words up to the first period don't constitute a sentence.

>> Authors: I do not understand the comment.

#####

In section 7.2, what is "IW10"?  Why are the particular traffic flow chosen in figure 2?  s/could/should/ in the last paragraph?

>> Authors: This can be added in the glossary. IW10 means Initial congestion Window set to 10. They have been chosen because they will generate bursts of data.

---

From: Kuhn Nicolas <[email-masked]>
Date: Tue, 17 May 2016 10:53:26 +0000

Hi again, 

FYI, attached to this email is what could be a new ID for the document. 

Kind regards,

Nicolas

-----Message d'origine-----
De : Kuhn Nicolas 
Envoyé : mardi 17 mai 2016 10:31
À : Ralph Droms (rdroms); [email-masked]
Cc : [email-masked]; Mirja Kuehlewind (IETF)
Objet : RE: Gen-ART review of draft-ietf-aqm-eval-guidelines-11

Kind regards, 

The authors

-----Message d'origine-----
De : Ralph Droms (rdroms) [mailto:[email-masked]] Envoyé : jeudi 28 avril 2016 21:23 À : [email-masked] Cc : [email-masked]
Objet : Gen-ART review of draft-ietf-aqm-eval-guidelines-11

Major issues:

Minor issues:

#####

#####

#####

#####

#####

#####

>> Authors: The scope of introducing this part is mostly related to 
>> remain the tester that if their AQM supports ECN, this must be 
>> presented, since it could be seen as an important move for the 
>> deployment of ECN

#####

#####

Nits/editorial comments:

#####

#####

#####

#####

OLD:

   In real

NEW:

#####

#####

#####

#####

#####

#####

#####

#####

Section 2.2: s/of/and/

#####

#####

#####

OLD:

NEW:

#####

#####

#####

#####

---

From: Kuhn Nicolas <[email-masked]>
Date: Wed, 18 May 2016 15:55:35 +0000

Dear Ralph, 

For your information, we have been working on a -12 version, which diffs with the v11 are attached to this email.
We can push a new ID after the telechat tomorrow if you think it is necessary.
On top of many small things, we have adapted the normative throughout the whole document (but , kept it for the conclusion section, since someone who want to characterize an AQM may want to be sure on what scenarios are more important. The normative language induces some weighs on the different scenarios. I hope you re fine with it.)

Kind regards,

Nicolas

-----Message d'origine-----

Major issues:

Minor issues:

Nits/editorial comments:

OLD:

   In real

NEW:

Section 2.2: s/of/and/

OLD:

NEW:

---

From: Jari Arkko <[email-masked]>
Date: Wed, 18 May 2016 21:29:09 -0400

Thanks for your review, Ralph!

>>> Authors: Indeed, the use of RFC 2119 language is not mandatory for such information document. However, using it enables us to introduce weight in the different parameterizations of the tests. Eve…

I think that’s right.

>>> Authors: We acknowledge that Figure 1 is not very clear. We have voluntarily omitted precisions on the amount of senders, receivers and traffic classes since the instantiation on a specific testb…

My opinion is that Figure 1 was very hard to read, even with reading the text. I’d like to see some improvement in either the text or the figure.

>>> Authors: These capacities are described in a later section when needed, and to remain high level and not focus on any applicability context (wi-fi, rural satellite access, fibber access, etc.) th…

OK

> "1. Similar control parameters and implications: Testers should be aware of the control parameters of the different schemes that control similar behavior. Testers should also be aware of the input value ranges and corresponding implications. For example, consider two different schemes - (A) queue-length based AQM scheme, and (B) queueing-delay based scheme. A and B are likely to have different kinds of control inputs to control the target delay - target queue length in A vs. target queuing delay in B, for example. Setting parameter values such as 100MB for A vs. 10ms for B will have different implications depending on evaluation context.  Such context-dependent implications must be considered before drawing conclusions on performance comparisons. Also, it would be preferable if an AQM proposal listed such parameters and discussed how each relates to network characteristics such as capacity, average RTT etc.”

OK for me

> " An AQM scheme SHOULD adhere to the recommendations outlined in [RFC7141], and SHOULD NOT provide undue advantage to flows with smaller packets [RFC7567]. In order to evaluate if an AQM scheme is biased towards flows with smaller size packets, sender A in Figure 1 can be instantiated with two long standing TCP flows with different packet sizes - 500 bytes vs. 1500 bytes, respectively and metrics such as goodput, loss rate can be compared for these two flows.
> “

OK

>>> Authors: The scope of introducing this part is mostly related to remain the tester that if their AQM supports ECN, this must be presented, since it could be seen as an important move for the depl…

I tend to agree with Ralph on this.

I’m not sure why we need this for assessment or characterisation.

>>> Authors: It is worth pointing out that this specific topology is just a suggestion. The router without AQM has been included to remind the tester that the positioning of multiple AQMs have to be…

The text wasn’t clear in my mind that this is just a suggestion and/or an example.

Jari

---

From: "Ralph Droms (rdroms)" <[email-masked]>
Date: Sat, 21 May 2016 10:34:16 +0000

> On May 18, 2016, at 9:29 PM 5/18/16, Jari Arkko <[email-masked]> wrote:
> 

You're welcome.  I'm glad to hear you found the review valuable.

Responses in line...

>>>> Authors: Indeed, the use of RFC 2119 language is not mandatory for such information document. However, using it enables us to introduce weight in the different parameterizations of the tests. Ev…

OK.

> My opinion is that Figure 1 was very hard to read, even with reading the text. I’d like to see some improvement in either the text or the figure.

I can understand that the figure might be an illustrative abstraction, but I still think it would be helpful to have more detail in the text and some restructuring of the figure.

OK

>> "1. Similar control parameters and implications: Testers should be aware of the control parameters of the different schemes that control similar behavior. Testers should also be aware of the input…

I think the suggested text is OK, too.

>> " An AQM scheme SHOULD adhere to the recommendations outlined in [RFC7141], and SHOULD NOT provide undue advantage to flows with smaller packets [RFC7567]. In order to evaluate if an AQM scheme is…

OK with me, too.

> I’m not sure why we need this for assessment or characterization.

The purpose of the section isn't at all clear to me.

> The text wasn’t clear in my mind that this is just a suggestion and/or an example.

Some clarification here would be helpful.

- Ralph