Subject: Re: [ippm] Magnus Westerlund's Discuss on draft-ietf-ippm-capacity-metric-method-06: (with DISCUSS)

From: Magnus Westerlund <[email-masked]>
Date: Tue, 2 Mar 2021 10:26:52 +0000

Hi Al,

Would it be possible for you and your co-authors to call in tomorrow Wednesday at 16:30 CET to the TSV-Office hours to discuss this document?  

Meeting link:
    https://ietf.webex.com/ietf/j.php?MTID=m689aa12ae4b319f4e371988b8330a863 
Meeting number:
    185 768 2828
Password:
    PNvNTuQm823

Please see inline. 

> > > >
> > > > ------------------------------------------------------------------
> > > > ----
> > > > DISCUSS:
> > > > ------------------------------------------------------------------
> > > > ----
> > > >
> > > > A) Section 8. Method of Measurement
> > > >
> > > > I think the metrics are fine, what makes me quite worried here is
> > > > the measurement method. My concerns with it are the following.
> > > >
> > > > 1. The application of this measurement method is not clearly scoped.
> <snip>
> [acm] we agreed on text adding "access" applicability to the scope section.
> 
> > > > However in
> > > > that context I think the definition and protection against severe
> > > > congestion has significant short comings. The main reason is that
> > > > the during a configurable time period (default 1 s) the sender
> > > > will attempt to send at a specified rate by a table independently
> > > > on what happens during that second.
> > >
> > > [acm]
> > > Not quite, 1 second is the default measurement interval for
> > > Capacity, but sender rate adjustments occur much faster (and we add
> > > a default at 50ms). This is a an important point (and one that Ben
> > > also noticed, regarding variable F in section 8.1). So, I have added FT as a
> parameter in section 4:
> > >
> > > o FT, the feedback time interval between status feedback messages
> > > communicating measurement results, sent from the receiver to control
> > > the sender. The results are evaluated to determine how to adjust the
> > > current offered load rate at the sender (default 50ms)
> > >
> > > -=-=-=-=-=-=-=-
> > > Note that variable F in section 8.1 is redundant with parameter F in
> > Section
> > > 4,
> > > the number of flows (in-06). So we changed the section 8.1 variable
> > > F to
> > FT
> > > in the working text.
> >
> > Okay, that makes things clearer. With all the equal intervals in the
> > metrics I had missinterpreted that also the transmission would be
> > uniform during the measurement intervals.
> >
> > However, when rereading Section 8.1 I do have to wonder if the
> > non-cumaltive feedback actually creates two issues. First, it appears
> > to loose information for reordering that crosses the time when the FT timer
> fires, due to reset.
> [acm]
> I don't understand how the sequence error counting "loses information"
> when reordered packets cross a measurement feedback boundary. I'm not
> sure what aspect of measurement you are "resetting", but I assume it is
>     "The accumulated statistics are then
>      reset by the receiver for the next feedback interval."
> 
> Suppose I have two measurement intervals and I receive:
> 
> ||  1  2  3  5  6 || 4  7  8  9 ...||
> 
> where || is the measurement feedback boundary.
> 
> Packet 4 arrives late enough from its original position to span the boundary.
> The 3->5 sequence is one sequence error, and the 4->7 sequence is another
> error.
> This example produces two sequence errors in different feedback intervals,
> but that's a typical measurement boundary problem. We can't get rid of
> measurement boundaries, and they affect many measurements.
> 
> Note that a reordered packet contributes to IP-Layer Capacity, by definition.
> 
> Perhaps you had some other scenario in mind?
> 

It was the above type of cases I was considering. I was worried that you don't detect some cases of issues if you reset and have no memory of last received packets. I think one obvious case that doesn't report as out of order for really short ones are this case when zero or single packets are delivered. Then a reordering could look like this.

|| 1   || 3 || 2 || and without a memory of the last received this would not be an out of order event. Nor would a packet loss like this

|| 1 ||     ||  3 || 

So when FB is on the same magnitude as the sent packet interval this measurement method appear to have issues. I have not analyzed if a single last seen packet number memory would solve most or all issues. 

At least it would also solve the issue of this

|| 1 2  3 4 || 6 7 8 || that I am also uncertain if that is detected. 

So, I think more details are needed to what is required on the feedback mechanism to detect these type of issues. 

> 
> > In addition if the feedback is not reliable it looses the information
> > for that interval.
> [acm]
> That's right, and:
> 1. the sending rate does not increase or decrease without feedback 2. the
> feedback is traveling the reverse path that the test do not congest with
>    test traffic
> 3. the running code has watchdog time-outs that *terminate the
> connection* if
>    either the sender or receiver go quiet
> 
> In essence, the test method is not reliable byte stream transfer like TCP.
> We can shut-down the test traffic very quickly if something is wrong and
> useful measurement is in question.
> 
> 

Al, I hope you understand that my concern here is that the documents description of the control algorithm is incomplete and are missing important protection aspects. As well as its behavior to different parameterizations in various conditions. 

> > And making feedback reliabel could cause worse HOL issues for
> > reacting to later feedback that are recived prior to the lost one.
> >
> [acm]
> So the alternative to un-reliable feedback can be worse?
> Good thing it's not planned.

So lets assume that receiver measures two feedback intervals in a row that there where reordering events etc. So in case the first report is lost, then from the senders perspective would there be an advantage to know that both intervals indicated issues at this rate? It at least indicates that the measurement that one is over capacity is consistent. 

Also, will the sender know that feedback is missing. This is all depending on the protocol used for the measurements. 

So I think there are two exiting protocols that I could reasonably easily implement this capacity measurement into. These two are QUIC and RTP/UDP. By taking of the shelf implementations I would have frameworks that would contain a number of features that appear needed to be able to do this measurement and receive feedback needed.

> 
> >
> > >
> > >
> > > >
> > > > 2. The algorithm for adjusting rate is table driven but give no guidance
> on
> > > > how
> > > > to construct the table and limitations on value changes in the table. In
> > > > addition the algorithm discusses larger steps in the table without any
> > > > reflection of what theses steps sides may represent in offered load.
> > >
> > > [acm]
> > > We can add (Len suggested the following text addition):
> > > OLD
> > > 8.1. Load Rate Adjustment Algorithm
> > >
> > > A table SHALL be pre-built defining all the offered load rates that
> > > will be supported (R1 through Rn, in ascending order, corresponding
> > > to indexed rows in the table). Each rate is defined as datagrams of...
> > >
> > > NEW
> > > 8.1. Load Rate Adjustment Algorithm
> > >
> > > A table SHALL be pre-built defining all the offered load rates that
> > > will be supported (R1 through Rn, in ascending order, corresponding
> > > to indexed rows in the table). It is RECOMMENDED that rates begin with
> > > 0.5 Mbps at index zero, use 1 Mbps at index one, and then continue in
> > > 1 Mbps increments to 1 Gbps. Above 1 Gbps, and up to 10 Gbps, it is
> > > RECOMMENDED that 100 Mbps increments be used. Above 10 Gbps,
> > > increments of 1 Gbps are RECOMMENDED. Each rate is defined as...
> >
> > Is this what you actually used in your test implementation?
> [acm]
> Yes, except that the current table stops at 10Gbps. We haven't had the
> opportunity to test >10Gbps.
> 
> > At my first glance
> > this recommendation looks to suffer from rather severe step effects and
> also
> > make the respond to losses behave strange around the transitions.
> Wouldn't some
> > type of logarithmic progression be more appropriate here for initial
> > probing?
> [acm]
> Len and I considered various algorithms for the search.
> 
> Logarithmic increase typically means more rate overshoot than Linear
> increases.
> Unfortunately, a large rate overshoot means that the queues will fill and
> need
> a longer time to bleed-off, meaning that rate reductions will take you far
> from
> the "right neighborhood" again.
> 
> Our experience is that we avoid large overshoot with fast or slow linear
> increases.
> It means we've taken some care to keep the network running. We haven't
> broken
> any test path yet, and we bail-out quickly and completely if something goes
> wrong.

I understand that there are several considerations here that are contradicting.

1. Quickly finding roughly the existing capacity
2. Avoid significant overshoot that build up to much queue if that is possible
3. Have sufficient fine grained control around the capacity so that adjustment step finds the capacity.

For example the last I am worried when you define the step size as 1 GBPS above 10 GBPS. My fear is that one 1 GBPS more traffic will increase the buffer occupancy so quickly that one alternates between not filled, and over filled between stepping up and receiving the feedback in the control cycle. We have to remember that with a FB of 50 ms the delay in the control cycle will be up to RTT + 50 ms.  

So I am speculating and you haven't tested. So if you put in table construction recommendations, then I do think you should be very clear that these are untested. 

> 
> >
> > If I have 1 GBPS line rate, there is a 1000 steps in the table to this value.
> > Even if I increase with the suggested 10 steps until first congestion seen, it
> > will take 100 steps, and with 50 ms feedback interval that is 5 seconds
> before
> > it is in the right ball park.
> [acm]
> Your math is correct.
> 
> Remember that the only assumption we made when building the table of
> sending rates
> is that the maximum *somewhere between 500kbps and 10Gbps*. Our lab
> tests used
> unknown rates between 50Mbps and 10Gbps, as the "ground truth" that we
> asked UDP
> and TCP-based methods to measure correctly. Measurements on production
> networks
> encountered many different technologies. Some subscriber rates were 5 to
> 10 Mbps
> on upstream.
> 
> > And I get one random loss at 10 mbps, then its 990
> > steps, In such a situation the whole measurement period (10 s) would be
> over
> > before one has reached actual capacity.
> [acm]
> I'm sorry, that's not quite correct, assuming delay range meets the criteria
> below, which would be consistent with "one random loss".
> 

Yes, my mistake, but let me replace one random loss with one congestion event due to a single intermittent transaction. So harder to trigger if the capacity is large, easier if it is low and then it matters less. So this reduced the issue significantly.

> The text says:
>   If the feedback indicates that sequence number anomalies were detected
> OR
>   the delay range was above the upper threshold, the offered load rate is
>   decreased.  (by one step)
> 
> But when the next feedback message arrives with no loss, and the
> "congestion"
> state has not been declared, the relevant text is:
> 
>   If the feedback indicates that no sequence number anomalies were
> detected AND
>   the delay range was below the lower threshold, the offered load rate is
> increased.
>   If congestion has not been confirmed up to this point, the offered load rate
> is
>   increased by more than one rate (e.g., Rx+10).
> 
> and we return to the high speed increases, because:
> 
>   Lastly, the method for inferring congestion is that there were sequence
>   number anomalies AND/OR the delay range was above the upper threshold
> for
>   *two* consecutive feedback intervals.
> 
> So, there is a single step back for the single random loss, but then
> immediately
> back to Rx+10 increases.
> 
> 

Ok. Good that this is less than a issue then I thought. 

> >
> > To me it appears that the probing (slow start) equivalent do need
> logarithmic
> > increase to reach likely capacity quickly. Then how big the adjustment is
> > actually dependent on what extra delay one consider the target for the
> test.
> [acm]
> Our delay variation values are low, but need to accommodate the relatively
> high
> delay variation of some access technologies. We learned this during our
> testing
> on production networks.
> 
> > Having a step size of 1 GBPS if probing a 2.5 GBPS path would likely make it
> > very hard to keep the delay in the intended interval when it would
> fluxtuate
> > between 500 mbps to much traffic and then 500 mbps to little. Sure with
> > sufficiently short FT it will likely work in this algorithm. However, I wonder
> > about regulation stability here for differnet RTT, FTs and buffer depth
> > fluxtuations.
> [acm]
> I'm sorry, that's not quite correct, the text we proposed to add says:
> 
>    It is RECOMMENDED that rates begin with 0.5 Mbps at index zero,
>    use 1 Mbps at index one, and then continue in 1 Mbps increments to 1
> Gbps.
>    Above 1 Gbps, and up to 10 Gbps, it is RECOMMENDED that 100 Mbps
> increments be used.
> and                                                        ^^^^^^^^
>    Above 10 Gbps, increments of 1 Gbps are RECOMMENDED.
> 
> Your example falls in the 1Gbps to 10Gbps range, where table increments are
> 100Mbps.
> 
> >
> > From my perspective I think this is a indication that the load rate
> adjustment
> > algorithm is not ready to be a standards track specification.
> [acm]
> Given several corrections above, the authors ask that you reconsider your
> position.
> Please read on.
> 
> >
> > I would recommend that you actually take out the control algorithm and
> write a
> > high level functional description of what needs to happen when measuring
> this
> > capacity.
> [acm]
> We worked for several weeks in December to make the current high-level
> description
> an accurate one. The IESG review has resulted added some useful details
> that
> I have shared along the way.
> 
> >
> > If I understand this correctly the requirements on the measurement are
> the
> > following.
> >
> > - Need to seek the available capacity so that several measurement period
> are
> > likely to be done at capacity
> > - Must not create persistent congestion as the capacity measurement
> should be
> > based on traffic capacity that doesn't cause more standing queue that X,
> where X
> > is some additiona delay in ms compared to minimal one way delay. And X is
> > actually something that is configurable for a measurement campagin as
> capacity
> > for a given one way delay and delay variation can be highly relevant to
> > know.
> [acm]
> These are not identical to our requirements.  For example:
>  - Both the metric and method consider a number of measurement intervals,
> and the Maximum IP-Layer Capacity is determined from one (or more) of the
> intervals.

> 
> 
> >
> > What else is needed?
> >
> > Are synchronized clocks needed or just relative delay changes necessary?
> [acm]
> Just delay variation, and the safest is RT delay variation.

So sender side transmission timestamping and the receiver measure delay variations based on these transmission times would suffice. 

Good. 

> 
> >
> > >
> > > -=-=-=-=-=-=-=-
> > >
> > > >
> > > > 3. Third the algorithms reaction to any sequence number gaps is
> dependent on
> > > > delay and how it is related to unspecified delay thresholds. Also no text
> > > > discussion how these thresholds should be configured for safe
> operation.
> > >
> > > [acm]
> > > We can add some details in the paragraph below:
> > > OLD
> > > If the feedback indicates that sequence number anomalies were
> detected OR
> > > the delay range was above the upper threshold, the offered load rate is
> > > decreased.
> > > Also, if congestion is now ...
> > > NEW
> > > If the feedback indicates that sequence number anomalies were
> detected OR
> > > the delay range was above the upper threshold, the offered load rate is
> decreased.
> > > The RECOMMENDED values are 0 for sequence number gaps and 30-90
> ms for lower
> > > and upper delay thresholds. Also, if congestion is now ...
> >
> > Ok, but the delay values as I noted before highly dependent of what my
> goal with
> > the capacity metric is. If I want to figure out the capacity for like say XR or
> > cloud gaming applications that maybe have much lower OWD variances and
> absolute
> > values so maybe my values are 10-25 ms.
> [acm]
> We intend to measure the limit of the access technology, with a set of
> parameters
> that work well for all technologies we have tested so far.
> 
> Notice that I didn't type the word "application" above. Or "user experience".
> 
> Sure, there is sensitivity to the parameters chosen, and we supplied our
> well-tested defaults to maximize results comparability and technology
> coverage
> (with no twiddling).
> 
> 
> >
> > How much explaration have you done of the control stability over a range
> > of parameters? Do you have any material about that?
> [acm]
> Yes. There are several parameter ranges we examined.
> 
> If we set the delay thresholds high enough, we see the RTT grow as the
> queues
> fill to max and tail-drop finally restricts the rate. We can measure the
> extent of buffer bloat this way (if it is present). It's Not our goal.
> 
> We have used lower thresholds of delay variation, which work fine on the
> PON 1Gbps access services.
> 
> In the collaborative testing of the Open Broadband Open Source project,
> one participant contributed tests with a 5G system that exhibited systematic
> low-level loss and reordering in his lab. For this unusual case, Len added
> the features to set a loss threshold above zero, and to tolerate reordered
> and duplicate packets with no penalty in rate adjustment.
> 
> We have tried a range of test durations (I=20, 30 for example).
> 
> We have tried different steep-ness of ramp-up slope. Rate += 10 steps
> works well, even when measuring rates separated by 3 orders of magnitude.
> 
> 
> But for the co-authors, it was more important that the load adjustment
> search
> produce the correct Maximum IP-Layer Capacity for each of the lab
> conditions
> we created (including challenging conditions with competing traffic, long
> delay
> etc.), and the many access technologies we tested in production use
> (where again we encountered similar challenging conditions).
> 
> 

Thanks for the clarifications. 

> >
> >
> > >
> > > -=-=-=-=-=-=-=-
> > >
> > > Please also note many requirements for safe operation in Section 10,
> > > Security Considerations.
> > >
> > > >
> > > > B) Section 8. Method of Measurement
> > > >
> > > > There are no specification of the measurement protocol here that
> provides
> > > > sequence numbers, and the feedback channel as well as the control
> channel.
> > >
> > > [acm]
> > > That is correct. The Scope does not include protocol development.
> > >
> > > > Is this intended to use TWAMP?
> > >
> > > [acm]
> > > Maybe, but a lot of extensions would be involved.
> >
> >
> >
> > >
> > > >
> > > > From my perspective this document defines the metrics on standards
> track
> > > > level. However, the method for actually running the measurements are
> not
> > > > specified on a standards track level.
> > >
> > > [acm]
> > > In IPPM work, the methods of measurement are described more broadly
> than
> > > the metrics, as actions and operations the Src and Dst hosts perform to
> > > send and receive, and calculate the results.
> > >
> > > IPPM Methods of Measurement have not included protocol
> requirements in
> > > the past, in any of our Standards Track Metrics RFCs.  In fact, we
> developed
> > > a measurement-specific criteria for moving our RFCs along the standards
> track
> > > that has nothing to do with protocols or interoperability.
> > > See BCP 176 aka RFC 6576:
> https://protect2.fireeye.com/v1/url?k=65acc415-3a37fcc7-65ac848e-
> 866132fe445e-b4323cb4b6e4206f&q=1&e=5fe1792c-f641-4903-9a32-
> 317350790872&u=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%
> 2F%2Ftools.ietf.org%2Fhtml%2Frfc6576__%3B%21%21BhdT%213
> RMQMZOAFloQ1FvtC_LIrUsDqBGxQeJZ1F4GFt8TpbQ60w841wT54sLLoKS9-
> Yc$
> > > IP Performance Metrics (IPPM) Standard Advancement Testing
> > >
> > > > No one can build implementation.
> > >
> > > [acm]
> > > I'm sorry, but that is not correct.  Please see Section 8.4.
> >
> > Sorry, that was poorly formulated. I mean that you can't give this
> specification
> > to a guy on a island without external communication and have them
> > implemented it and it will work with someone else implementation.
> [acm]
> But then you are asking for protocol-level interoperability, Magnus.
> That is not our scope, or the scope of any IPPM Metric and Method RFCs.
> The procedures of BCP 176 tell us when independent implementations
> produce
> equivalent results, which is IETF's definition of "works with" for metrics
> and methods.
> 
> 
> > You have clearly
> > implemented a solutiont that works for some set of parameters. And I am
> asking how
> > much of the reasonable parameter space you have tested.
> [acm]
> Right. I answered this question qualitatively above, but the co-authors
> claim that an equally important question is the breadth of access
> technologies
> we have tested.
> 
> The tests conducted over 2+ years used the following production access
> types:
> 
> 1. Fixed: DOCSIS 3.0 cable modem with "triple-play" capability and embedded
> WiFi and
> Wired GigE switch (two manufacturers).
> 2. Mobile: LTE cellular phone with a Cat 12 modem (600 Mbps Downlink, 50
> Mbps uplink).
> 3. Fixed: passive optical network (PON) "F", 1 Gbps service.
> 4. Fixed: PON "T", 1000 Mbps Service.
> 5. Fixed: VDSL, service, at various rates <100 Mbps.
> 6. Fixed: ADSL, 1.5 Mbps.
> 7. Mobile: LTE enabled router with ETH LAN to client host
> 8. Fixed: DOCSIS 3.1 cable modem with "triple-play" capability and embedded
> WiFi and
> Wired GigE switch (two other manufacturers).
> 
> 
> >
> > Based on this discussion I don't think I can build an implementation that
> > fulfills the measurement goals, becasue I have questions about them. And I
> > suspect it would take substantial amount of experimentation to get it to
> > work correctly over a broader range of input parameters.
> [acm]
> Now we refer you to the references in the memo, particularly Appendix X
> of Y.1540:
> 
>    [Y.1540]   Y.1540, I. R., "Internet protocol data communication
>               service - IP packet transfer and availability performance
>               parameters", December 2019,
>               <https://www.itu.int/rec/T-REC-Y.1540-201912-I/en>.
> 
>    [Y.Sup60]  Morton, A., Rapporteur, "Recommendation Y.Sup60,
> Interpreting
>               ITU-T Y.1540 maximum IP-layer capacity measurements", June
>               2020, <https://www.itu.int/rec/T-REC-Y.Sup60/en>.
> 
> and Liaisons, were many of the experimental results are summarized:
> 
>    [LS-SG12-A]
>               12, I. S., "LS - Harmonization of IP Capacity and Latency
>               Parameters: Revision of Draft Rec. Y.1540 on IP packet
>               transfer performance parameters and New Annex A with Lab
>               Evaluation Plan", May 2019,
>               <https://datatracker.ietf.org/liaison/1632/>.
> 
>    [LS-SG12-B]
>               12, I. S., "LS on harmonization of IP Capacity and Latency
>               Parameters: Consent of Draft Rec. Y.1540 on IP packet
>               transfer performance parameters and New Annex A with Lab &
>               Field Evaluation Plans", March 2019,
>               <https://datatracker.ietf.org/liaison/1645/>.
> 
> Also, see our slides from the Hackathons at IETF 105 and 106, and the
> IPPM WG sessions slides beginning with IETF-105, July 2019.
> You might also look into the discussions on the mailing list.
> Some other results are available to those with ITU-TIES accounts.
> 
> The load adjustment algorithm itself was improved after experimentation,
> adding the fast ramp-up with rate += 10 when feedback indicates no
> impairments. The original/current algorithms appear in Y.1540 Annexes A
> and B, respectively.
> 
> 
> >
> >
> >
> > >
> > > > And if the section is
> > > > intended to provide requirements on a protocol that performs these
> > > > measurements
> > > > I think several aspects are missing. There appear several ways forward
> here
> > > > to
> > > > resolve this; one is to split out the method of measurement and define
> it
> > > > separately to standard tracks level using a particular protocol, another
> > > > is to write it purely as requirements on a measurement protocols.
> > >
> > > [acm]
> > > As stated above, connecting a method with a single protocol is not IPPM's
> way.
> >
> > That is fine. However, I find the attempt to specify a specific load regulator
> > in the method of measurement to take this specification beyond a general
> method
> > of measurment. The high level requirement appear to be that to correctly
> find
> > the capacity, and that requires that one load to the point where buffers are
> > filled sufficiently to introduce extra delay or where AQM starts dropping or
> > marking some of the load. Thus, I am questioning if the described algorithm
> will
> > adqueately solve that issue over a wider range of parameters.
> >
> > So if you have more information to show at least which range it has been
> proven
> > to do its work and with what input parameters?
> [acm]
> Yes. See ~10 references  and replies above.
> 
> > I hope you understand that I
> > expect this load control algorithm to get simularly scrutinized to congestion
> > control algorithms that we standardize in IETF.
> [acm]
> Yes, although it is a surprising at this point, we certainly understand your
> current position.
> 
> However, Rüdiger made a relevant point in our discussions (why our
> algorithm's
> role is different from Transport Area congestion control algorithms,
> and need not be subjected to the same scrutiny):
> 
>     This is a measurement method designed for infrequent and sensible
> maximum
>     capacity assessment, instantiated only in an OAM or diagnostic tool.
> 
>     It is not a blueprint for a congestion control algorithm (CCA) in a bulk
>     transfer protocol that runs by default and is globally deployed by
>     commodity stacks.
> 
> We don't want to re-create any TCP CCA: they weren't designed for accurate
> measurement of maximum rate (as the referenced measurements show).
> 
> It appears that the most recent (2018) standardized and widely used
> CCA is Cubic (RFC 8312 https://tools.ietf.org/html/rfc8312 ).
> 
> The great TCP CCA Census (2019)
> https://datatracker.ietf.org/meeting/109/materials/slides-109-iccrg-the-
> great-internet-tcp-congestion-control-census-00
> 
> finds that BBR versions account for greater popularity on Alexa-250 sites
> (25.2%) than CUBIC, and more than 40% of downstream traffic on the
> Internet
> (slide 14). I found some references to BBR in ICCRG drafts, but no RFC.
> I would guess that BBR has already provided CCA for more traffic than the
> test traffic complying to this memo ever will.
> 
> Our overall method works similar to BBR: Received rate per RTT is the
> feedback to the sender.
> 
> We added Applicability to the access portion, not the global Internet
> where standardized transport protocol CCAs must operate.
> 
> We are not specifying a transport CCA that must support many applications.
> Measurement is the *only* application (for an IP-layer metric).
> 
> Early impressions have been formed on several erroneous assumptions
> regarding algorithm stability (operation above 1Gbps) and
> suitability for purpose (one random loss case).

I agree to misunderstandings and misinterpretation of what was defined. And I do have to say that some of these assumptions or guesses are due to the lack of clarity of the specification. Which from my perspective strengthens my position that this is not yet standards track quality around the load algorithm. 

> 
> Ad-hoc methods resulting in TCP-based under-estimates of Internet Speed
> are the problem we attack here! Implementation of harmonized industry
> standards are the solution.
> 
> We believe in rough consensus and running code.
> 

So BBR is frequently discussed in ICCRG, and the BBR v1 was found to have some truly horrible stable state. Like one where it was running on at 30% packet loss without reacting as it only contained delay. BBRv2 has addressed this so it will react to higher degrees of packet loss. 

So I do believe in running code to and as long as you are monitoring your impact I think experimentation is the right way here. However, if we put the label IETF proposed standard on the load algorithm people will assume that it is safe to use. That is my issue here and I don't think we can give that recommendation even with a limited applicability statement in the current form. 

> 
> We also ask that you understand our position, that tests with many different
> access technologies in production, and careful comparison of ad-hoc
> methods
> claiming to make the similar measurements in the lab and the field are
> equally,
> if not more important than even more parameter investigations at this point.
> I have personally been running lab tests since September 2018 with various
> tools.
> Len released his first version of the code in Feb 2019,
> and we immediately focused on tests with his utility instead of UDP packet
> blasters like iPerf, and Trex with my own Binary Search with Loss verification
> algorithm that we use in device benchmarking (cross-over with BMWG).
> 
> 
> >
> > I would very much prefer to take out the load algorithm and place it in a
> > seperate document where it can have a tighter scope description and more
> > discussion about about that it does its job.
> >
> > I hope this clarifies what my concerns are with this document in its
> > current form.
> [acm]
> Yes, and we have rather exhaustively argued to go ahead here, especially
> since a much less-frequently-used testing-only algorithm is a different
> situation
> than specifying a CCA for global TCP deployment: transport area's usual role.
> 
> We hope you can now appreciate the years of study, experimentation and
> running code that you apparently first encountered last Thursday,
> and will look into some more of the supporting background material.
> 

I do appreciate your work. However, I still don't think the load control algorithm description is sufficiently well specified and parameterized to be published in a standards track document. Maybe it can be made into that. However, I think this would make an excellent experimental specification with some additional clarifications on parameters ranges and defined responses when feedback is not received in a timely fashion. Also I think you can clarify the protocol requirements for the measurement methodology. Then a crystal clear applicability statement for the load algorithm. 

Here I think we can see another point in actually splitting out the load algorithm and that is related to the applicability statement. Does the metric and the load algorithm have the same applicability? As a metric it is not truly restricted, it is just a question of the impact of measuring the metric across the internet. 

Also, I think we should consider somewhat the update and evolution path that hopefully will occur here. I think the load algorithm is likely to benefit from further clarification in its specification in the future. Thus, having it in a separate specification makes it easier to update and maintain. 

Cheers

Magnus Westerlund

---

From: <[email-masked]>
Date: Wed, 3 Mar 2021 10:37:29 +0000

Hi Magnus, hi Al

I'm available in the afternoon today. 

I agree that the topic presence and interpretation of sequence numbers may benefit from more text. I didn't implement or design the code. If sequence numbers are continuous across all intervals st and feedback reflects the number of packets received since the preceding feedback message and the sequence number of the last packet, this should be transparent (I'm not talking about re-ordering here, but continuous sequence numbers and reporting of them to some extent helps). I'm not sure whether the protocol is or needs to be able to detect whether a first feedback message (or any other) was dropped. My take is, if the feedback channel is lossy (i.e., worse than infrequent random drop), any test designed to detect maximum capacities at any place should stop as soon as possible. I prefer that to figuring out when a feedback should have been present, whether it was lost and could be re-transmitted and so on. 
The test preamble may benefit from more text, maybe (after the preamble, an upper RTT limit should be known and the sender may behave quite conservative during the preamble phase).

[MW]: "For example the last I am worried when you define the step size as 1 GBPS above 10 GBPS. My fear is that one 1 GBPS more traffic will increase the buffer occupancy so quickly"
The same step width percentage is applied from 1 Mbps to 1 Gbps, steps of 1 Mbps. In the Gbps range, the application is even more sensible (1 Gbps steps start since 10 Gbps). An access of more than 10 Gbps is physically 20 Gbps at least (a bundle) and the likely shaper rate is often n*1 Gbps. st is set to 50 ms. From what I can tell, that buffer size often is available at that bandwidth. One could try to work with a smaller value of st at that bandwidth. The number of packets transmitted needs to ensure, that a shaper or policer kicks in, i.e., above the burst tolerance of any traffic conditioner. Than that burst tolerance roughly needs to be below st, if expressed in ms@shaper rate (simply fixing the number of packets during st might be a bad choice).

Integration into an existing protocol - I leave this to implementers. I'd prefer the metric and method to be on standards track, and making it part of an existing protocol shouldn't be a requirement. Al and Len have compared the implementation with TCP based speed tests and the implementation proved to measure access bandwidth with better accuracy and precision in many cases (under differing circumstances). I regard this as an important feature of a standard implementation. IETF so far didn't standardize a  method to measure access speeds with high precision - I'm not happy with TCP based speed tests as an informal standard and a method showing better precision being classified as "experimental" by IETF (noting that your motivation derives from another issue here, but let's keep an eye on the whole thing. Please don't take this as "the purpose justifies the means" - implementations need to work reliable of course).

Al, there's a nit in Section 6.5:

OLD
   A single constant interval dt SHOULD be
   chosen so that is an integer multiple of increasing values k times
                          ^^^^
   serialisation delay of a path MTU at the physical interface speed
   where traffic conditioning is expected.

NEW
   A single constant interval dt SHOULD be
   chosen so that it is an integer multiple of increasing values k times
                          ^^^^
   serialisation delay of a path MTU at the physical interface speed
   where traffic conditioning is expected.

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: Magnus Westerlund <[email-masked]> 
Gesendet: Dienstag, 2. März 2021 11:27
An: [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Betreff: RE: Magnus Westerlund's Discuss on draft-ietf-ippm-capacity-metric-method-06: (with DISCUSS)

Hi Al,

Meeting link:

Please see inline. 

|| 1 ||     ||  3 || 

Good. 

Cheers

Magnus Westerlund

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Wed, 3 Mar 2021 13:39:28 +0000

Hi Magnus,

Len and I can meet tomorrow, but I haven't heard from Rudiger.
Also, I somehow failed to press *send* on this message last night ???

We have extracted your comments in the remaining areas for discussion, below.

Al 

<Reordering example: >

|| 1 ||     ||  3 || 

@@@@Answer: the text says:
	The accumulated statistics are then reset by the receiver for the next feedback interval.

and nothing about clearing the entire memory of the measurement code.

Also, assuming you are looking for recommended ranges for parameters like FB (=FT in the text), we can certainly provide them. FT is never = single packet send interval and << dt duration of the sub-interval.

-=-=-=-=-=-=-=-=-=-=-=-=-

@@@@Answer: We can add more details, but we need to agree on the list before we add the parameters, etc., and then we are done. To be clear, we have already added many default settings for parameter values. That's far-beyond what IPPM methods have required, and it recognizes both the number of parameters involved in this method/algorithm and the extensive testing we have completed (where we haven't broken anything in the 'net). 

@@@@Also, it makes no sense to test for equivalent results with different parameter sets.  BCP 176 recommends identical parameters (the only exception is lack of ability to configure the parameter to identical values).

-=-=-=-=-=-=-=-=-=-

> Magnus wrote:

So let's assume that receiver measures two feedback intervals in a row that there where reordering events etc. So in case the first report is lost, then from the senders perspective would there be an advantage to know that both intervals indicated issues at this rate? It at least indicates that the measurement that one is over capacity is consistent. 

@@@@Answer: In the case above, the fast initial sender ramp-up would pause in response to the second report (Rx -= 1, like the single random loss case we discussed) and then resume at Rx += 10 steps. So, if the reordering in your example is caused by congestion we have continued to send at the ~rate that caused congestion during the missing status interval, and likely have a third message/measurement indicating congestion or two more measurements when the rate increases further.  
IOW, if measurement message loss is at the onset of congestion, rate increase will ensure more opportunities to communicate that state. And if measurement message loss is a random event, it just pauses the ramp-up for one FT interval.

-=-=-=-=-=-=-=-=- 

@@@@Answer: Yes, the running code (and a draft protocol proposal) has sequence numbers.

@@@@Answer: Yes, you could do that. Or, you could help us by adding the minimum security features that would pass IETF/SecDIR review into the running code and (currently identical) protocol proposal. Both of these paths have a set of unknowns when it comes to the final product, like will a small-CPU-power CPE sender still be able to achieve 1 or 2 Gbps? 

@@@@Offer: :-)
We now have the draft with the protocol description ready to go. If we can have a submission during black-out period, then people can look at it before IPPM's last-last session on Friday.  Otherwise, we'll wait till next Monday.

-=-=-=-=-=-=-=-=-=-

> opportunity to test >10Gbps.
...
> increases. It means we've taken some care to keep the network running. 
> We haven't broken any test path yet, and we bail-out quickly and 
> completely if something goes wrong.

@@@@Answer:  We added three sentences describing three ranges in response to your comment, IIRC. We can easily edit the sentence in the working text: 
    Above 10 Gbps, increments of 1 Gbps are RECOMMENDED.
	
and say something safer:
    Above 10 Gbps, increments of 100 Mbps are suggested in the absence of testing.
	
or we could delete the sentence entirely!  Your choice.

-=-=-=-=-=-=-=-=-=-=-=-

@@@@Answer: But we are in the phase of development where new readers suggest clarifications to the text, and we are more than willing to clarify anything we wrote. We spent several weeks doing exactly that with Martin before Last Call. To be fair, you did not have much time to review this memo before casting your ballot. 

@@@@Proposal: Would it help in your view to make the load adjustment pseudocode in the Appendix a normative supplement to the processes we have described in words in section 8.1 ?  We added the pseudocode during Martin's review.

-=-=-=-=-=-=-=-=-=-=-

@@@@Answer: Matt Mathis and I had lunch together at IETF the last time we were in Montreal, where he told me this BBRv1 story. Matt was very excited about what we were proposing because of the similarities to BBR, and after two hours Matt bought my lunch. During the moths that followed, we performed several comparison tests between our running code and BBR versions.

@@@@@Answer: Here's another Big Difference from the usual TCP algorithm deployment criteria, which you still seem be applying. BBR updates have been able to take advantage of vast server-ownership when they update the servers to use BBR-next-version and then the dominant downstream traffic is immediately controlled in the best way.  In the running code and the protocol proposal draft, the load control algorithm is a *server-only function*, and the server resides in the network where the algorithm can more easily be adjusted if necessary, and fewer hosts need the updates (as opposed to many-many more clients that are notoriously resistant to updates). There is a significant measure of safety provided by that strategy alone.

-=-=-=-=-=-=-=-=-=-=-=-=-

I do appreciate your work. However, I still don't think the load control algorithm description is sufficiently well specified and parameterized to be published in a standards track document. Maybe it can be made into that.

@@@@Answer: This is the path we were on, and prefer to remain on that path. Like every other reviewer, you tell us what you what to see clarified, and we will do it.

However, I think this would make an excellent experimental specification with some additional clarifications on parameters ranges and defined responses when feedback is not received in a timely fashion. 

@@@@Answer: If it were March 2019, we would agree with you that the specification is experimental back then. But now, two additional years of experimentation and rigorous review has occurred, and every other review indicates that we're ready for the Proposed Standard phase of metric and method standardization, with further evaluation according to BCP 176.  

Also I think you can clarify the protocol requirements for the measurement methodology. 

@@@@Answer: We can do more than that. The protocol requirements draft is ready for submission. It could be an Informative reference in this draft.

Then a crystal clear applicability statement for the load algorithm. 

@@@@Answer: I agree, and you can help us write the load algorithm-specific wording for the applicability section. In any case, that's part of a good compromise.

-=-=-=-=-=-=-=-=-

@@@@Answer: The IPPM precedent is that we specify both the metric and the method together on the Standards Track. Even if the methods are intentionally minimal, as they were at the beginning of IPPM work >20 years ago, the drafts went on the Standards Track, and IPPM was tasked to figure-out our own Standards Track advancement criteria (BCP 176). As a result, IPPM has developed a very complete formulation of our working methods over time.
 
-=-=-=-=-=-=-=-=-

@@@@Answer: Actually, experience tells me that aspects throughout the memo might benefit from updates after tests according to BCP 176 with additional experience and implementation. For example, the process to develop STD 81 (for delay) and STD 82 (for loss) involved improvements throughout the normative text, not just the methods:
https://tools.ietf.org/html/rfc7679#section-7
https://tools.ietf.org/html/rfc7680#section-6

Also, a load-adjustment algorithm specification on its own is more likely to be used beyond the original intent (and less-likely to be found if hidden inside an RFC on measurement, as-is the case now).

@@@@Conclusion: So, we have followed all previous IPPM precedents very closely and gone beyond them to provide additional specifications in many ways. We have certainly done more experimentation that was directly accessible to IETF participants than for any other IPPM metric and method (e.g., at the hackathons). We are open to all suggestions to embellish or clarify the current text. Let us work that aspect together now.

> -----Original Message-----
> From: Magnus Westerlund [mailto:[email-masked]]
> Sent: Tuesday, March 2, 2021 5:27 AM
> To: MORTON, ALFRED C (AL) <[email-masked]>; [email-masked]
> Cc: [email-masked]; [email-masked]; draft-ietf-ippm-capacity-
> [email-masked]; [email-masked]; [email-masked]
> Subject: RE: Magnus Westerlund's Discuss on draft-ietf-ippm-capacity-
> metric-method-06: (with DISCUSS)
> 
> Hi Al,
> 
> Would it be possible for you and your co-authors to call in tomorrow
> Wednesday at 16:30 CET to the TSV-Office hours to discuss this document?
> 
> Meeting link:
> 
> It was the above type of cases I was considering. I was worried that you
> don't detect some cases of issues if you reset and have no memory of last
> received packets. I think one obvious case that doesn't report as out of
> order for really short ones are this case when zero or single packets are
> delivered. Then a reordering could look like this.
> 
> || 1   || 3 || 2 || and without a memory of the last received this would
> not be an out of order event. Nor would a packet loss like this
> 
> || 1 ||     ||  3 ||
> 
> So when FB is on the same magnitude as the sent packet interval this
> measurement method appear to have issues. I have not analyzed if a single
> last seen packet number memory would solve most or all issues.
> 
> So, I think more details are needed to what is required on the feedback
> mechanism to detect these type of issues.
> 
> >
> Al, I hope you understand that my concern here is that the documents
> description of the control algorithm is incomplete and are missing
> important protection aspects. As well as its behavior to different
> parameterizations in various conditions.
> 
> So lets assume that receiver measures two feedback intervals in a row that
> there where reordering events etc. So in case the first report is lost,
> then from the senders perspective would there be an advantage to know that
> both intervals indicated issues at this rate? It at least indicates that
> the measurement that one is over capacity is consistent.
> 
> Also, will the sender know that feedback is missing. This is all depending
> on the protocol used for the measurements.
> 
> So I think there are two exiting protocols that I could reasonably easily
> implement this capacity measurement into. These two are QUIC and RTP/UDP.
> By taking of the shelf implementations I would have frameworks that would
> contain a number of features that appear needed to be able to do this
> measurement and receive feedback needed.
> 
> 
> >
> > >
> > > >
> > > >
> > > > >
> For example the last I am worried when you define the step size as 1 GBPS
> above 10 GBPS. My fear is that one 1 GBPS more traffic will increase the
> buffer occupancy so quickly that one alternates between not filled, and
> over filled between stepping up and receiving the feedback in the control
> cycle. We have to remember that with a FB of 50 ms the delay in the
> control cycle will be up to RTT + 50 ms.
> 
> So I am speculating and you haven't tested. So if you put in table
> construction recommendations, then I do think you should be very clear
> that these are untested.
> 
> 
> >
> > >
> Yes, my mistake, but let me replace one random loss with one congestion
> event due to a single intermittent transaction. So harder to trigger if
> the capacity is large, easier if it is low and then it matters less. So
> this reduced the issue significantly.
> 
> > The text says:
> So sender side transmission timestamping and the receiver measure delay
> variations based on these transmission times would suffice.
> 
> Good.
> 
> >
> > >
> > > >
> > > > -=-=-=-=-=-=-=-
> > > >
> > > > >
> I agree to misunderstandings and misinterpretation of what was defined.
> And I do have to say that some of these assumptions or guesses are due to
> the lack of clarity of the specification. Which from my perspective
> strengthens my position that this is not yet standards track quality
> around the load algorithm.
> 
> >
> So BBR is frequently discussed in ICCRG, and the BBR v1 was found to have
> some truly horrible stable state. Like one where it was running on at 30%
> packet loss without reacting as it only contained delay. BBRv2 has
> addressed this so it will react to higher degrees of packet loss.
> 
> So I do believe in running code to and as long as you are monitoring your
> impact I think experimentation is the right way here. However, if we put
> the label IETF proposed standard on the load algorithm people will assume
> that it is safe to use. That is my issue here and I don't think we can
> give that recommendation even with a limited applicability statement in
> the current form.
> 
> 
> >
> I do appreciate your work. However, I still don't think the load control
> algorithm description is sufficiently well specified and parameterized to
> be published in a standards track document. Maybe it can be made into
> that. However, I think this would make an excellent experimental
> specification with some additional clarifications on parameters ranges and
> defined responses when feedback is not received in a timely fashion. Also
> I think you can clarify the protocol requirements for the measurement
> methodology. Then a crystal clear applicability statement for the load
> algorithm.
> 
> Here I think we can see another point in actually splitting out the load
> algorithm and that is related to the applicability statement. Does the
> metric and the load algorithm have the same applicability? As a metric it
> is not truly restricted, it is just a question of the impact of measuring
> the metric across the internet.
> 
> Also, I think we should consider somewhat the update and evolution path
> that hopefully will occur here. I think the load algorithm is likely to
> benefit from further clarification in its specification in the future.
> Thus, having it in a separate specification makes it easier to update and
> maintain.
> 
> 
> Cheers
> 
> Magnus Westerlund

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Thu, 4 Mar 2021 23:47:54 +0000

Hi Magnus and all,

We have discussed and prepared new material for the memo that we hope
cover all the points raised at our meeting on Wednesday. The 3 major changes 
appear in Section 2 Scope and Applicability, and 8.1 on the Load Adjustment
Algorithm.

Let us know if these revisions are sufficient, or if we can add more material
that will help to resolve concerns.

thanks,
Al (for the co-authors)

New text in Section 2 Scope and Applicability:

   In addition, the use of the load adjustment algorithm described in
   section 8.1 has the following additional applicability limitations:

   - MUST only be used in the application of diagnostic and operations
   measurements as described in this memo

   - MUST only be used in circumstances consistent with Section 10,
   Security Considerations

New text adding "safety" requirements, and a new table of parameters in 
section 8.1 at the end:

   The Load Rate Adjustment Algorithm MUST include timers that stop the
   test when received packet streams cease unexpectedly. The timeout
   thresholds are provided in the table below, along with values for all
   other parameters and variables described in this section.

   +-------------+-------------+--------------+------------------------+
   | Parameter   | Default     | Tested Range | Expected Safe Range    |
   |             |             | or values    | (not entirely tested,  |
   |             |             |              | other values NOT       |
   |             |             |              | RECOMMENDED)           |
   +-------------+-------------+--------------+------------------------+
   | FT,         | 50ms        | 20ms, 100ms  | 5ms <= FT <= 250ms     |
   | feedback    |             |              | Larger values may slow |
   | time        |             |              | the rate increase and  |
   | interval    |             |              | fail to find the max   |
   +-------------+-------------+--------------+------------------------+
   | Feedback    | L*FT, L=10  | L=100 with   | 0.5sec <= L*FT <=      |
   | message     | (500ms)     | FT=50ms      | 30sec Upper limit for  |
   | timeout     |             | (5sec)       | very unreliable test   |
   | (stop test) |             |              | paths only             |
   +-------------+-------------+--------------+------------------------+
   | load packet | 1sec        | 5sec         | 0.250sec - 30sec Upper |
   | timeout     |             |              | limit for very         |
   | (stop test) |             |              | unreliable test paths  |
   |             |             |              | only                   |
   +-------------+-------------+--------------+------------------------+
   | table index | 0.5Mbps     | 0.5Mbps      | when testing <=10Gbps  |
   | 0           |             |              |                        |
   +-------------+-------------+--------------+------------------------+
   | table index | 1Mbps       | 1Mbps        | when testing <=10Gbps  |
   | 1           |             |              |                        |
   +-------------+-------------+--------------+------------------------+
   | table index | 1Mbps       | 1Mbps -      | same as tested         |
   | (step) size |             | 1Gbps        |                        |
   +-------------+-------------+--------------+------------------------+
   | table index | 100Mbps     | 1Gbps -      | same as tested         |
   | (step)      |             | 10Gbps       |                        |
   | size,       |             |              |                        |
   | rate>1Gbps  |             |              |                        |
   +-------------+-------------+--------------+------------------------+
   | table index | 1Gbps       | untested     | >10Gbps                |
   | (step)      |             |              |                        |
   | size,       |             |              |                        |
   | rate>10Gbps |             |              |                        |

Morton, et al.          Expires September 5, 2021              [Page 16]

Internet-Draft         IP Capacity Metrics/Methods            March 2021

   +-------------+-------------+--------------+------------------------+
   | ss, UDP     | none        | <=1222       | Recommend max at       |
   | payload     |             |              | largest value that     |
   | size, bytes |             |              | avoids fragmentation   |
   +-------------+-------------+--------------+------------------------+
   | cc, burst   | none        | 1 - 100      | same as tested         |
   | count       |             |              |                        |
   +-------------+-------------+--------------+------------------------+
   | tt, burst   | 100microsec | 100microsec, | available range of     |
   | interval    |             | 1msec        | "tick" values (HZ      |
   |             |             |              | param)                 |
   +-------------+-------------+--------------+------------------------+
   | low delay   | 30ms        | 5ms, 30ms    | same as tested         |
   | range       |             |              |                        |
   | threshold   |             |              |                        |
   +-------------+-------------+--------------+------------------------+
   | high delay  | 90ms        | 10ms, 90ms   | same as tested         |
   | range       |             |              |                        |
   | threshold   |             |              |                        |
   +-------------+-------------+--------------+------------------------+
   | sequence    | 0           | 0, 100       | same as tested         |
   | error       |             |              |                        |
   | threshold   |             |              |                        |
   +-------------+-------------+--------------+------------------------+
   | consecutive | 2           | 2            | Use values >1 to avoid |
   | errored     |             |              | misinterpreting        |
   | status      |             |              | transient loss         |
   | report      |             |              |                        |
   | threshold   |             |              |                        |
   +-------------+-------------+--------------+------------------------+
   | Fast mode   | 10          | 10           | 2 <= steps <= 30       |
   | increase,   |             |              |                        |
   | in table    |             |              |                        |
   | index steps |             |              |                        |
   +-------------+-------------+--------------+------------------------+
   | Fast mode   | 30          | 3 * Fast     | same as tested         |
   | decrease,   |             | mode         |                        |
   | in steps    |             | increase     |                        |
   +-------------+-------------+--------------+------------------------+
   | Number of   | 2000        | 2000         | same as tested         |
   | table steps |             |              |                        |
   | in total,   |             |              |                        |
   | <10Gbps     |             |              |                        |
   +-------------+-------------+--------------+------------------------+

               Parameters for Load Rate Adjustment Algorithm

> From: MORTON, ALFRED C (AL)
> Sent: Wednesday, March 3, 2021 8:40 AM
> To: 'Magnus Westerlund' <[email-masked]>; [email-masked]
> Also, assuming you are looking for recommended ranges for parameters like
> FB (=FT in the text), we can certainly provide them. FT is never = single
> packet send interval and << dt duration of the sub-interval.
> 
> @@@@Answer: We can add more details, but we need to agree on the list
> before we add the parameters, etc., and then we are done. To be clear, we
> have already added many default settings for parameter values. That's far-
> beyond what IPPM methods have required, and it recognizes both the number
> of parameters involved in this method/algorithm and the extensive testing
> we have completed (where we haven't broken anything in the 'net).
> 
> @@@@Also, it makes no sense to test for equivalent results with different
> parameter sets.  BCP 176 recommends identical parameters (the only
> exception is lack of ability to configure the parameter to identical
> values).
> 
> 
> -=-=-=-=-=-=-=-=-=-
> 
> > Magnus wrote:
> So let's assume that receiver measures two feedback intervals in a row
> that there where reordering events etc. So in case the first report is
> lost, then from the senders perspective would there be an advantage to
> know that both intervals indicated issues at this rate? It at least
> indicates that the measurement that one is over capacity is consistent.
> 
> @@@@Answer: In the case above, the fast initial sender ramp-up would pause
> in response to the second report (Rx -= 1, like the single random loss
> case we discussed) and then resume at Rx += 10 steps. So, if the
> reordering in your example is caused by congestion we have continued to
> send at the ~rate that caused congestion during the missing status
> interval, and likely have a third message/measurement indicating
> congestion or two more measurements when the rate increases further.
> IOW, if measurement message loss is at the onset of congestion, rate
> increase will ensure more opportunities to communicate that state. And if
> measurement message loss is a random event, it just pauses the ramp-up for
> one FT interval.
> 
> -=-=-=-=-=-=-=-=-
> 
> @@@@Answer: Yes, you could do that. Or, you could help us by adding the
> minimum security features that would pass IETF/SecDIR review into the
> running code and (currently identical) protocol proposal. Both of these
> paths have a set of unknowns when it comes to the final product, like will
> a small-CPU-power CPE sender still be able to achieve 1 or 2 Gbps?
> 
> @@@@Offer: :-)
> We now have the draft with the protocol description ready to go. If we can
> have a submission during black-out period, then people can look at it
> before IPPM's last-last session on Friday.  Otherwise, we'll wait till
> next Monday.
> 
> 
> -=-=-=-=-=-=-=-=-=-
> 
> > [acm]
> @@@@Answer:  We added three sentences describing three ranges in response
> to your comment, IIRC. We can easily edit the sentence in the working
> text:
> @@@@Answer: But we are in the phase of development where new readers
> suggest clarifications to the text, and we are more than willing to
> clarify anything we wrote. We spent several weeks doing exactly that with
> Martin before Last Call. To be fair, you did not have much time to review
> this memo before casting your ballot.
> 
> @@@@Proposal: Would it help in your view to make the load adjustment
> pseudocode in the Appendix a normative supplement to the processes we have
> described in words in section 8.1 ?  We added the pseudocode during
> Martin's review.
> 
> 
> -=-=-=-=-=-=-=-=-=-=-
> 
> @@@@Answer: Matt Mathis and I had lunch together at IETF the last time we
> were in Montreal, where he told me this BBRv1 story. Matt was very excited
> about what we were proposing because of the similarities to BBR, and after
> two hours Matt bought my lunch. During the moths that followed, we
> performed several comparison tests between our running code and BBR
> versions.
> 
> @@@@@Answer: Here's another Big Difference from the usual TCP algorithm
> deployment criteria, which you still seem be applying. BBR updates have
> been able to take advantage of vast server-ownership when they update the
> servers to use BBR-next-version and then the dominant downstream traffic
> is immediately controlled in the best way.  In the running code and the
> protocol proposal draft, the load control algorithm is a *server-only
> function*, and the server resides in the network where the algorithm can
> more easily be adjusted if necessary, and fewer hosts need the updates (as
> opposed to many-many more clients that are notoriously resistant to
> updates). There is a significant measure of safety provided by that
> strategy alone.
> 
> that.
> @@@@Answer: This is the path we were on, and prefer to remain on that
> path. Like every other reviewer, you tell us what you what to see
> clarified, and we will do it.
> 
> However, I think this would make an excellent experimental specification
> with some additional clarifications on parameters ranges and defined
> responses when feedback is not received in a timely fashion.
> 
> @@@@Answer: If it were March 2019, we would agree with you that the
> specification is experimental back then. But now, two additional years of
> experimentation and rigorous review has occurred, and every other review
> indicates that we're ready for the Proposed Standard phase of metric and
> method standardization, with further evaluation according to BCP 176.
> 
> @@@@Answer: We can do more than that. The protocol requirements draft is
> ready for submission. It could be an Informative reference in this draft.
> 
> @@@@Answer: I agree, and you can help us write the load algorithm-specific
> wording for the applicability section. In any case, that's part of a good
> compromise.
> 
> -=-=-=-=-=-=-=-=-
> 
> @@@@Answer: The IPPM precedent is that we specify both the metric and the
> method together on the Standards Track. Even if the methods are
> intentionally minimal, as they were at the beginning of IPPM work >20
> years ago, the drafts went on the Standards Track, and IPPM was tasked to
> figure-out our own Standards Track advancement criteria (BCP 176). As a
> result, IPPM has developed a very complete formulation of our working
> methods over time.
> 
> -=-=-=-=-=-=-=-=-
> 
> @@@@Answer: Actually, experience tells me that aspects throughout the memo
> might benefit from updates after tests according to BCP 176 with
> additional experience and implementation. For example, the process to
> develop STD 81 (for delay) and STD 82 (for loss) involved improvements
> throughout the normative text, not just the methods:
> Also, a load-adjustment algorithm specification on its own is more likely
> to be used beyond the original intent (and less-likely to be found if
> hidden inside an RFC on measurement, as-is the case now).
> 
> @@@@Conclusion: So, we have followed all previous IPPM precedents very
> closely and gone beyond them to provide additional specifications in many
> ways. We have certainly done more experimentation that was directly
> accessible to IETF participants than for any other IPPM metric and method
> (e.g., at the hackathons). We are open to all suggestions to embellish or
> clarify the current text. Let us work that aspect together now.
> 
> 
> > > > > also noticed, regarding variable F in section 8.1). So, I have
> added
> > FT as a
> > > parameter in section 4:
> > > > >
> > > > to loose information for reordering that crosses the time when the
> FT
> > timer
> > > fires, due to reset.
> > > [acm]
> > > > > > addition the algorithm discusses larger steps in the table
> without
> > any
> > > > If I have 1 GBPS line rate, there is a 1000 steps in the table to
> this
> > value.
> > > > Even if I increase with the suggested 10 steps until first
> congestion
> > seen, it
> detected
> > > > increase to reach likely capacity quickly. Then how big the
> adjustment
> > is
> > > > sufficiently short FT it will likely work in this algorithm.
> However,
> > I wonder
> > > Your example falls in the 1Gbps to 10Gbps range, where table
> increments
> > are
> > > 100Mbps.
> > >
> > > >
> > > > based on traffic capacity that doesn't cause more standing queue
> that
> > X,
> > > where X
> > > > is some additiona delay in ms compared to minimal one way delay. And
> X
> > is
> > > > > > delay and how it is related to unspecified delay thresholds.
> Also
> > no text
> > > > > the delay range was above the upper threshold, the offered load
> rate
> > is
> > > > > decreased.
> > > > > the delay range was above the upper threshold, the offered load
> rate
> > is
> > > decreased.
> > > > the capacity metric is. If I want to figure out the capacity for
> like
> > say XR or
> > > > > the metrics, as actions and operations the Src and Dst hosts
> perform
> > to
> > > > suspect it would take substantial amount of experimentation to get
> it
> > to
> &
> > > Yes, although it is a surprising at this point, we certainly
> understand
> > your
> > > current position.
> > >
> > >     It is not a blueprint for a congestion control algorithm (CCA) in
> a
> > bulk
> > > than specifying a CCA for global TCP deployment: transport area's
> usual
> > role.
> > >

---

From: Magnus Westerlund <[email-masked]>
Date: Fri, 5 Mar 2021 10:00:45 +0000

Hi Al,

Please see inline

> From: MORTON, ALFRED C (AL) <[email-masked]>
> Sent: den 5 mars 2021 00:48
> To: Magnus Westerlund <[email-masked]>; [email-masked]

I think this is a bit to unclear in regards to the limitation of scope. My worry are internet wide measurements. Reading what exists in the currently available draft in the Section 10, there are no limitation here described to only use this algorithm only across controlled networks. Especially as bullet 5 in Section 10 relies on the load algorithm what is currently written is fine across the whole internet as long as sender and receiver are okay with the measurement which is not what I at least thought we agreed. 

Can you please be explicit that the load algorithm is limited to use across networks path that are controlled or managed and not intended for Internet wide usage. 
 
 - MUST only be used as part of measurements within managed networks, and not across general Internet.

>    other parameters and variables described in this section.

I think this looks reasonable, however I need to consider that a bit more and I think I need to have it in context. I would recommend that resolving the applicability scope and then submit a new draft on Monday. Then I can look at in context next week. 

Cheers

Magnus Westerlund

> > 	The accumulated statistics are then reset by the receiver for
> the
> > next feedback interval.
> >
> > > > encountered many different technologies. Some subscriber rates were
> 5
> > to
> > > > 10 Mbps
> > > > on upstream.
> > > >
> > > > and the Maximum IP-Layer Capacity is determined from one (or more)
> of
> > > the
> > > > intervals.
> > >
> > >
> > >
> > > >
> > > >
> > > > >
> > > > > What else is needed?
> > > > >
> > > > > cloud gaming applications that maybe have much lower OWD
> variances
> > and
> > > > absolute
> > > > > > > level. However, the method for actually running the
> measurements
> > > are
> > > > not
> > > > > implemented a solutiont that works for some set of parameters. And
> I
> > > am
> > > > asking how
> > > > Now we refer you to the references in the memo, particularly
> Appendix
> > X
> > > > of Y.1540:
> > > >

---

From: <[email-masked]>
Date: Fri, 5 Mar 2021 11:12:52 +0000

<snip>

I think this is a bit to unclear in regards to the limitation of scope. My 
worry are internet wide measurements. Reading what exists in the currently 
available draft in the Section 10, there are no limitation here described to 
only use this algorithm only across controlled networks. Especially as bullet 
5 in Section 10 relies on the load algorithm what is currently written is fine 
across the whole internet as long as sender and receiver are okay with the 
measurement which is not what I at least thought we agreed.

Can you please be explicit that the load algorithm is limited to use across 
networks path that are controlled or managed and not intended for Internet 
wide usage.

 - MUST only be used as part of measurements within managed networks, and not 
across general Internet.

[RG] That's tough, as regulators are interested in this test and regulators 
aren't part of a domain. So they might resort to TCP speed tests, being less 
accurate and precise and not standardised (note that penalties are discussed 
to be linked to results). I'd be interested in finding agreement to have 
metric and method standardized for at least nations under authority of a 
single regulator. No idea how exactly. Would a bound on RTT/in some countries 
multiple instances and some additional parametrization information prior to 
start, e.g., a contracted access bandwidth, help?

[RG] I'm off for the weekend (I'm out for a lasting solution, not necessarily 
a speedy one) - regards,

Ruediger

---

From: Magnus Westerlund <[email-masked]>
Date: Fri, 5 Mar 2021 13:05:47 +0000

Hi,

Ruediger, if you can find a formulation that covers that national test case I 
am likely fine with it. If the involved parties know who to discuss issues 
with and can get them addressed I am not worried. I am worried where someone 
deploys a couple of servers, like the current TCP speed tests and users run it 
totally by themselves.

And to be clear I think with more experience with large scale deployment of 
the algorithm and more experiments beyond the intended deployment model this 
should be possible to update the specs to remove this type of limitation.

Cheers

Magnus

> From: [email-masked] <[email-masked]>
> Sent: den 5 mars 2021 12:13
> To: Magnus Westerlund <[email-masked]>
> [email-masked]; [email-masked]; [email-masked];
> [email-masked]; [email-masked]
> Subject: AW: Magnus Westerlund's Discuss on draft-ietf-ippm-capacity-

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Fri, 5 Mar 2021 13:22:06 +0000

Hi Magnus and Rüdiger,

The working text on the "applicability" limits that we added to the scope section *included an earlier agreement* to add the first sentence below and the bullet, so now the whole paragraph reads:

   The primary application of the metric and method of measurement
   described here is the same as in Section 2 of [RFC7479] where:

   o  The access portion of the network is the focus of this problem
      statement.  The user typically subscribes to a service with
      bidirectional access partly described by rates in bits per second.

We can make further edits to balance your comments but let's start here,
and sorry for not including the whole paragraph last night - it seems 
that I should have. 

Al

> Sent: Friday, March 5, 2021 8:06 AM
> To: [email-masked]
> [email-masked]; MORTON, ALFRED C (AL) <[email-masked]>
> > 5 in Section 10 relies on the load algorithm what is currently written
> is
> > fine

---

From: <[email-masked]>
Date: Mon, 8 Mar 2021 08:19:07 +0000

Hi Al, hi Magnus,

To make progress within the time left, it's likely best to leave things as they are. That produces a standard and a benchmark. A cross domain deployment may be tested against that benchmark to gain more experience on operational aspects.

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: MORTON, ALFRED C (AL) <[email-masked]> 
Gesendet: Freitag, 5. März 2021 14:22
An: Magnus Westerlund <[email-masked]>; Geib, Rüdiger <[email-masked]>
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Betreff: RE: Magnus Westerlund's Discuss on draft-ietf-ippm-capacity-metric-method-06: (with DISCUSS)

Hi Magnus and Rüdiger,

We can make further edits to balance your comments but let's start here, and sorry for not including the whole paragraph last night - it seems that I should have. 

Al

> Ruediger, if you can find a formulation that covers that national test 
> case I am likely fine with it. If the involved parties know who to 
> discuss issues with and can get them addressed I am not worried. I am 
> worried where someone deploys a couple of servers, like the current 
> TCP speed tests and users run it totally by themselves.
> 
> And to be clear I think with more experience with large scale 
> deployment of the algorithm and more experiments beyond the intended 
> deployment model this should be possible to update the specs to remove 
> this type of limitation.
> 
> Cheers
> 
> Magnus
> 
> > across the whole internet as long as sender and receiver are okay 
> > with
> the
> >  - MUST only be used as part of measurements within managed 
> > networks, and not across general Internet.
> >
> > aren't part of a domain. So they might resort to TCP speed tests, 
> > being
> less
> > to be linked to results). I'd be interested in finding agreement to 
> > have metric and method standardized for at least nations under 
> > authority of a single regulator. No idea how exactly. Would a bound 
> > on RTT/in some countries multiple instances and some additional 
> > parametrization information prior
> to
> > [RG] I'm off for the weekend (I'm out for a lasting solution, not 
> > necessarily a speedy one) - regards,
> >
> > Ruediger

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Mon, 8 Mar 2021 17:32:40 +0000

Hi all,

The working text was just submitted as version 07. [0] 
We can continue discussions from this point.

Thanks again for all the IESG reviews!

Al (for the co-authors)

[0] https://datatracker.ietf.org/doc/html/draft-ietf-ippm-capacity-metric-method

> From: [email-masked] [mailto:[email-masked]]
> Sent: Monday, March 8, 2021 3:19 AM
> To: MORTON, ALFRED C (AL) <[email-masked]>;
> [email-masked]
> [email-masked]; [email-masked]; [email-masked]; [email-masked]
> To make progress within the time left, it's likely best to leave things as
> they are. That produces a standard and a benchmark. A cross domain
> deployment may be tested against that benchmark to gain more experience on
> operational aspects.
> 
> Regards,
> 
> Ruediger
> 
> [email-masked]; [email-masked]; [email-masked]; [email-masked]
> The working text on the "applicability" limits that we added to the scope
> section *included an earlier agreement* to add the first sentence below
> and the bullet, so now the whole paragraph reads:
> 
> and sorry for not including the whole paragraph last night - it seems that
> I should have.
> 
> Al
> 
> > > I think this is a bit to unclear in regards to the limitation of
> scope.
> > My

---

From: Magnus Westerlund <[email-masked]>
Date: Thu, 11 Mar 2021 11:15:36 +0000

Hi,

I have reviewed the text changes in context. 

First of all I like to be clear that these changes do need review from the rest of the WG and a confirming consensus call as they are substantial. 

Section 2.

I think the scope text is still fairly open. Yes it is clear that the load algorithm is only intended for measurements. However, the usage is not particular limited, especially in regards to my main concern of edge to central nodes across multiple AS in the Internet like the different TCP speed tests often are deployed. The security consideration requirements are good for a number of reasons but put no limitations on that aspect. So I would prefer a more explicit statement here. I think an important aspect here is that any ISP seeing issues from these measurements should know who to talk to. I don't know how to best formulate that. 

Section 8.1:

So I am trying to understand the implication of the load algorithm at higher rates and how recommendations works out in relation to the definition of the rate. 

So the document says: 

    Each rate is defined as
   datagrams of size ss, sent as a burst of count cc, each time interval
   tt (default for tt is 1ms, a likely system tick-interval)

So I think this definition is fine for lower rate as the number of packets in each 1 ms burst is fairly small and the buffer it hits will likely be relatively large compared to the increase in load. However at higher rates like beyond 10 GBPS where 1 GBPS steps are recommended. So transmitting at bursts every 1 ms intervals means that one are transmitting 833 packets each burst at 10 GBP rate of 1500 bytes size, so likely even higher for more moderate 1200 byte size packets. That is almost 1,3 mb of data. So where pacing may be quite good at lower bit-rates < 1Gbps I wonder if it starts breaking down at higher rates, which appears to in the region where buffers becomes more shallow due to the cost of having large buffers and where good pacing reduces the need for buffering. I would also note that the reaction time for the control can be 1RTT + 50 ms which thus the increase in offered load for a step size becomes 10s of MB during an regulation period.

As the load algorithm hasn't been tested beyond 10Gbps and it appears that the numbers can start to become more problematic at these speeds, wouldn't it be better to say that this is not intended beyond 10 Gbps. 

On the table I have the following comments:

   +--------------+-------------+--------------+-----------------------+
   | Parameter    | Default     | Tested Range | Expected Safe Range   |
   |              |             | or values    | (not entirely tested, |
   |              |             |              | other values NOT      |
   |              |             |              | RECOMMENDED)          |
   +--------------+-------------+--------------+-----------------------+
   | FT, feedback | 50ms        | 20ms, 100ms  | 5ms <= FT <= 250ms    |
   | time         |             |              | Larger values may     |
   | interval     |             |              | slow the rate         |
   |              |             |              | increase and fail to  |
   |              |             |              | find the max          |
   +--------------+-------------+--------------+-----------------------+

I would note that a FT of 5 ms will have the potential to result in significant fluxtuations in some systems like mobile systems as the scheduler time is actually likely to be longer than 5 ms.

   +--------------+-------------+--------------+-----------------------+
   +--------------+-------------+--------------+-----------------------+

Even the default means that one looses 10 feedback packets in a row. That is a lot and shows that one have a serious interruption on the return path. Already loosing 3 feedback packets in a row indicates that one have significant outages if this is lost. 

Secondly, this is formulated only based on  intervals of FT. For startup the RTT is relevant factor. So I think there are several factors here for the timeout that maybe need to be teased apart? So initially one offers a very low load and one may not have a good measurement on base RTT. Thus, time to first feedback is okay to be fairly large and 500 ms is likely okay but quite longer than expected for an access to local internet exchange measurement. However, when one scale up the rate I think these values are way to long as the total amount of traffic sent without feedback becomes quite significant. Receiving no feedback for more than 10 reporting intervals are already way to long. And to state that 30 seconds would be an acceptable value I can't support even for a measurement tool.

The definition of what "Feedback message timeout" and "Load packet Timeout" is not defined. I assume that Feedback message timeout is the time without receiving any feedback messages after starting a measurement. Is the load packet timeout the time the receiver is waiting before using signalling channel to end the measurement without receiving any packets, or for the sender to receive feedback that says that no packets have been received? The roles here are not clear.

Sending packets for several seconds without seeing any result appears problematic and allowing values beyond several seconds looks broken. 

   +--------------+-------------+--------------+-----------------------+
   | table index  | 0.5Mbps     | 0.5Mbps      | when testing <=10Gbps |
   | 0            |             |              |                       |
   +--------------+-------------+--------------+-----------------------+
   | table index  | 1Mbps       | 1Mbps        | when testing <=10Gbps |
   | 1            |             |              |                       |
   +--------------+-------------+--------------+-----------------------+

Why is this value not relevant when testing beyond 10 Gbps, the ramp up time becomes to long with these values or?

   | ss, UDP      | none        | <=1222       | Recommend max at      |
   | payload      |             |              | largest value that    |
   | size, bytes  |             |              | avoids fragmentation  |
   +--------------+-------------+--------------+-----------------------+

So isn't there a mismatch between the metric and the load algorithm values here? With the rate definition in Section 8.1 being defined as based on "ss" that UDP payload bytes, rather than IP packet sizes that are used? 

I understand that one want to ensure that one measure using a size that actually works in the path. However, I think one should be warned that one might run into packet rate limitations rather than byte limits if one would use too small. 
`
   +--------------+-------------+--------------+-----------------------+
   | cc, burst    | none        | 1 - 100      | same as tested        |
   | count        |             |              |                       |
   +--------------+-------------+--------------+-----------------------+

So the cc value is dependent on target rate and the value of ss and tt. So should it be included in this table? Especially as 100 is not sufficient for multi-gigabit speeds with a tt of 1 ms. 

   +--------------+-------------+--------------+-----------------------+
   | low delay    | 30ms        | 5ms, 30ms    | same as tested        |
   | range        |             |              |                       |
   | threshold    |             |              |                       |
   +--------------+-------------+--------------+-----------------------+

So I think this value is highly dependent on several aspects and maybe should get more discussion. First for a measurement campaign it is relevant what one consider as the target additional latency that is acceptable when finding capacity. Secondly, the jitter in the network technology. For WIFI,  mobile and DOCIS a to low value may be shorter than the scheduling latencies that might occur. It is also a question about how precise the implementation are capable of measuring per packet latency variances. 

   +--------------+-------------+--------------+-----------------------+
   | high delay   | 90ms        | 10ms, 90ms   | same as tested        |
   | range        |             |              |                       |
   | threshold    |             |              |                       |
   +--------------+-------------+--------------+-----------------------+

Also here I wished there was a bit more discussion. So this value clearly must be above expected jitter for the network technology. It also needs to be sufficient large to represent a fair amount of queue to avoid measurement errors. I assume that if one would chose a value larger than available buffer depth one would drive the network into packet loss. And as long as there are some room between low delay range threshold and the actual delay causing loss or this higher one has a chance to regulate to that rate. 

   +--------------+-------------+--------------+-----------------------+
   | sequence     | 0           | 0, 100       | same as tested        |
   | error        |             |              |                       |
   | threshold    |             |              |                       |
   +--------------+-------------+--------------+-----------------------+

What is this value really? 

   +--------------+-------------+--------------+-----------------------+
   | consecutive  | 2           | 2            | Use values >1 to      |
   | errored      |             |              | avoid misinterpreting |
   | status       |             |              | transient loss        |
   | report       |             |              |                       |
   | threshold    |             |              |                       |
   +--------------+-------------+--------------+-----------------------+

Also here I am uncertain what is the criteria here?

   +--------------+-------------+--------------+-----------------------+
   | Fast mode    | 30          | 3 * Fast     | same as tested        |
   | decrease, in |             | mode         |                       |
   | table index  |             | increase     |                       |
   | steps        |             |              |                       |
   +--------------+-------------+--------------+-----------------------+

So is the recommended value 30 or 3*Fast mode increase? Should they be proportional or not?

The last entry appears to be a summary fact of the parameterization, and is it relevant?

What is the goal here in relation to push other congestion controlled traffic out of the way? It appears that it is likely to cause delay based congestion to be pushed out of the way. I am more uncertain how it interacts with loss based ones, as depending on situation it appears that it could avoid going into the loss regim. 

My conclusion is that some aspect of this do appear more clarifications on what they are and further assumptions on how the load algorithm will be deployed spelled out so that its function is more controlled. 

Cheers

Magnus

> Sent: den 8 mars 2021 18:33
> To: [email-masked]; Magnus Westerlund
> <[email-masked]>
> [email-masked]
> The working text was just submitted as version 07. [0] We can continue
> discussions from this point.
> 
> > [email-masked]
> > To make progress within the time left, it's likely best to leave
> > things as they are. That produces a standard and a benchmark. A cross
> > domain deployment may be tested against that benchmark to gain more
> > experience on operational aspects.
> >
> > Regards,
> >
> > Ruediger
> >
> > An: Magnus Westerlund <[email-masked]>; Geib,
> Rüdiger
> > <[email-masked]>
> > [email-masked]
> > The working text on the "applicability" limits that we added to the
> > scope section *included an earlier agreement* to add the first
> > sentence below and the bullet, so now the whole paragraph reads:
> >
> > We can make further edits to balance your comments but let's start
> > here, and sorry for not including the whole paragraph last night - it
> > seems that I should have.
> >
> > Al
> >
> > > Ruediger, if you can find a formulation that covers that national
> > > test case I am likely fine with it. If the involved parties know who
> > > to discuss issues with and can get them addressed I am not worried.
> > > I am worried where someone deploys a couple of servers, like the
> > > current TCP speed tests and users run it totally by themselves.
> > >
> > > deployment model this should be possible to update the specs to
> > > remove this type of limitation.
> > >
> > > Cheers
> > >
> > > Magnus
> > >
> > > > Cc: [email-masked]; [email-masked];
> > > > draft-ietf-ippm-capacity- [email-masked];
> > > > [email-masked]; [email-masked]; [email-masked];
> > > > [email-masked]
> > > > Can you please be explicit that the load algorithm is limited to
> > > > use
> > > across
> > > > to be linked to results). I'd be interested in finding agreement
> > > > to have metric and method standardized for at least nations under
> > > > authority of a single regulator. No idea how exactly. Would a
> > > > bound on RTT/in some countries multiple instances and some
> > > > additional parametrization information prior
> > > to

---

From: <[email-masked]>
Date: Thu, 11 Mar 2021 16:31:35 +0000

Hi Magnus,

Before going to discuss details some questions:

Would it help to insert text that person initiating test is required to parametrize an access bandwidth to be validated? 
I'm asking, as the intent of the draft might not be to capture a completely unknown access bandwidth (but I think it doesn't say so explicitely yet).

Do some of your comments address a concern that a test creates to much congestion during one feedback interval FT?
That may stem from the idea, that we've exactly reached access bandwidth AB during FT and badly congest the access during FT+1 (I don't deny that this is possible).

If that's your concern, would an approach to limit the maximum congestion created before FT+1 is reached being bound to a limit Buffer_max[ms]@AB[Bit/s] based on the last captured congestion free bandwidth AB during FT?
As an example, Buffer_max = 0,05 s if AB < 1 Gbit/s (smaller Buffer_max values may be applicable at higher rates). This would result in a formula determining the sending behaviour during st, rather than a table. 

Regards,

Rüdiger

-----Ursprüngliche Nachricht-----
Von: Magnus Westerlund <[email-masked]> 
Gesendet: Donnerstag, 11. März 2021 12:16
An: [email-masked]; Geib, Rüdiger <[email-masked]>
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Betreff: RE: Magnus Westerlund's Discuss on draft-ietf-ippm-capacity-metric-method-06: (with DISCUSS)

Hi,

Section 2.

Section 8.1:

So the document says: 

    Each rate is defined as

Cheers

Magnus

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Fri, 12 Mar 2021 03:15:58 +0000

Hi Magnus,

In addition to Rudiger's questions, please see in-line replies from me.

Al
> Sent: Thursday, March 11, 2021 11:32 AM
> To: [email-masked]
> Would it help to insert text that person initiating test is required to
> parametrize an access bandwidth to be validated?
> I'm asking, as the intent of the draft might not be to capture a
> completely unknown access bandwidth (but I think it doesn't say so
> explicitely yet).
> 
> Do some of your comments address a concern that a test creates to much
> congestion during one feedback interval FT?
> That may stem from the idea, that we've exactly reached access bandwidth
> AB during FT and badly congest the access during FT+1 (I don't deny that
> this is possible).
> 
> If that's your concern, would an approach to limit the maximum congestion
> created before FT+1 is reached being bound to a limit
> Buffer_max[ms]@AB[Bit/s] based on the last captured congestion free
> bandwidth AB during FT?
> As an example, Buffer_max = 0,05 s if AB < 1 Gbit/s (smaller Buffer_max
> values may be applicable at higher rates). This would result in a formula
> determining the sending behaviour during st, rather than a table.
> 
> Regards,
> 
> Rüdiger
> 
> 
> 
> First of all I like to be clear that these changes do need review from the
> rest of the WG and a confirming consensus call as they are substantial.
> 
> Section 2.
> 
> I think the scope text is still fairly open. Yes it is clear that the load
> algorithm is only intended for measurements. However, the usage is not
> particular limited, especially in regards to my main concern of edge to
> central nodes across multiple AS in the Internet like the different TCP
> speed tests often are deployed. The security consideration requirements
> are good for a number of reasons but put no limitations on that aspect. So
> I would prefer a more explicit statement here. I think an important aspect
> here is that any ISP seeing issues from these measurements should know who
> to talk to. I don't know how to best formulate that.
> 
> 
> Section 8.1:
> 
> So I am trying to understand the implication of the load algorithm at
> higher rates and how recommendations works out in relation to the
> definition of the rate.
> 
> So the document says:
> 
>     Each rate is defined as
> So I think this definition is fine for lower rate as the number of packets
> in each 1 ms burst is fairly small and the buffer it hits will likely be
> relatively large compared to the increase in load. However at higher rates
> like beyond 10 GBPS where 1 GBPS steps are recommended. So transmitting at
> bursts every 1 ms intervals means that one are transmitting 833 packets
> each burst at 10 GBP rate of 1500 bytes size, so likely even higher for
> more moderate 1200 byte size packets. That is almost 1,3 mb of data. So
> where pacing may be quite good at lower bit-rates < 1Gbps I wonder if it
> starts breaking down at higher rates, which appears to in the region where
> buffers becomes more shallow due to the cost of having large buffers and
> where good pacing reduces the need for buffering. I would also note that
> the reaction time for the control can be 1RTT + 50 ms which thus the
> increase in offered load for a step size becomes 10s of MB during an
> regulation period.
> 
> As the load algorithm hasn't been tested beyond 10Gbps and it appears that
> the numbers can start to become more problematic at these speeds, wouldn't
> it be better to say that this is not intended beyond 10 Gbps.
> 
> I would note that a FT of 5 ms will have the potential to result in
> significant fluxtuations in some systems like mobile systems as the
> scheduler time is actually likely to be longer than 5 ms.
[acm] 
Then we can increase the low end of the range, what value would you prefer??

> Even the default means that one looses 10 feedback packets in a row. That
> is a lot and shows that one have a serious interruption on the return
> path. Already loosing 3 feedback packets in a row indicates that one have
> significant outages if this is lost.
> 
> Secondly, this is formulated only based on  intervals of FT. For startup
> the RTT is relevant factor. So I think there are several factors here for
> the timeout that maybe need to be teased apart? So initially one offers a
> very low load and one may not have a good measurement on base RTT. 
[acm] 
All the timeouts are conducted on inter-packet arrival times at a single interface.
This removes the dependency on RTT. 

> Thus, time to first feedback is okay to be fairly large and 500 ms is likely
> okay but quite longer than expected for an access to local internet
> exchange measurement. However, when one scale up the rate I think these
> values are way to long as the total amount of traffic sent without
> feedback becomes quite significant. Receiving no feedback for more than 10
> reporting intervals are already way to long. And to state that 30 seconds
> would be an acceptable value I can't support even for a measurement tool.
[acm] 
We are willing to revise values, especially 30 sec, but would like to avoid 
prematurely shutting down measurements due to (what we consider to be)
short interruptions.

> 
> The definition of what "Feedback message timeout" and "Load packet
> Timeout" is not defined. I assume that Feedback message timeout is the
> time without receiving any feedback messages after starting a measurement.
[acm] 
That's close:
Operation: The load packet timeout SHALL be reset to the configured value 
each time a load packet received. If the timeout expires, the receiver 
SHALL be closed and no further feedback sent.

> Is the load packet timeout the time the receiver is waiting before using
> signalling channel to end the measurement without receiving any packets,
> or for the sender to receive feedback that says that no packets have been
> received? The roles here are not clear.
[acm] 
Operation: The feedback message timeout SHALL be reset to the configured 
value each time a feedback message is received. If the timeout expires, 
the sender SHALL be closed and no further load packets sent.

> 
> Sending packets for several seconds without seeing any result appears
> problematic and allowing values beyond several seconds looks broken.
[acm] 
Then let us make some revisions together. You've seen our proposals.
Our concern is stopping a test unnecessarily. There can be a happy 
conclusion.

> Why is this value not relevant when testing beyond 10 Gbps, the ramp up
> time becomes to long with these values or?
[acm] 
"not relevant" is different from the title of the column, which is:

Expected Safe Range:  when testing <=10Gbps

The parameters above and several others simply determine where a test starts.

This parameter:

> So isn't there a mismatch between the metric and the load algorithm values
> here? With the rate definition in Section 8.1 being defined as based on
> "ss" that UDP payload bytes, rather than IP packet sizes that are used?
[acm] 

Not really, UDP is mandatory in the metric definition.

> 
> I understand that one want to ensure that one measure using a size that
> actually works in the path. However, I think one should be warned that one
> might run into packet rate limitations rather than byte limits if one
> would use too small.
[acm] 
Ok
"Use of too-small payload size might result in unexpected sender limitations."

> So the cc value is dependent on target rate and the value of ss and tt. So
> should it be included in this table? Especially as 100 is not sufficient
> for multi-gigabit speeds with a tt of 1 ms.
[acm] 
We can remove it if the values cause confusion.

> So I think this value is highly dependent on several aspects and maybe
> should get more discussion. First for a measurement campaign it is
> relevant what one consider as the target additional latency that is
> acceptable when finding capacity. Secondly, the jitter in the network
> technology. For WIFI,  mobile and DOCIS a to low value may be shorter than
> the scheduling latencies that might occur. It is also a question about how
> precise the implementation are capable of measuring per packet latency
> variances.
[acm] 
As we discussed much earlier in this long thread, we arrived at both the 
delay threshold values after testing with the WIFI, mobile and DOCIS access
services we could use in production, and many others.

> Also here I wished there was a bit more discussion. So this value clearly
> must be above expected jitter for the network technology. It also needs to
> be sufficient large to represent a fair amount of queue to avoid
> measurement errors. I assume that if one would chose a value larger than
> available buffer depth one would drive the network into packet loss. And
> as long as there are some room between low delay range threshold and the
> actual delay causing loss or this higher one has a chance to regulate to
> that rate.
> 
[acm] 
When loss or reordering occur, initially these impairments appear as missing 
or unexpected sequence numbers in the stream, or sequence errors.

[acm] 
From the draft, where consecutive status reports are sent at feedback intervals:

   Lastly, the method for inferring congestion is that there were
   sequence number anomalies AND/OR the delay range was above the upper
   threshold for two consecutive feedback intervals.

[acm] 
The Default can be 3 * Fast mode increase if you want, that's what we tested.

> The last entry appears to be a summary fact of the parameterization, and
[acm] 
It might be removed, we thought it was useful info.
> 
> 
> What is the goal here in relation to push other congestion controlled
> traffic out of the way? It appears that it is likely to cause delay based
> congestion to be pushed out of the way. I am more uncertain how it
> interacts with loss based ones, as depending on situation it appears that
> it could avoid going into the loss regim.
[acm] 
The goal is to measure the true maximum rate during the test duration.

> 
> My conclusion is that some aspect of this do appear more clarifications on
> what they are 
[acm] 
These ASCII tables don't provide much space for explanation without becoming
awkward due to row height.  We'll add some definitions elsewhere.

> and further assumptions on how the load algorithm will be
> deployed spelled out so that its function is more controlled.
[acm] 
We could use some text suggestions to continue the discussion productively, 
having already tried several times.

thanks,
Al

> > [0] https://datatracker.ietf.org/doc/html/draft-ietf-ippm-capacity-
> metric-
> > method
> >
> >

---

From: Magnus Westerlund <[email-masked]>
Date: Fri, 12 Mar 2021 13:10:53 +0000

Hi Reudiger,

> Sent: den 11 mars 2021 17:32
> I'm asking, as the intent of the draft might not be to capture a completely
> unknown access bandwidth (but I think it doesn't say so explicitely yet).

I think that would help significantly.

> is possible).

Yes, what I am worried in some cases is the amount of overshoot the load 
algorithm can cause. And that is not just one FT interval. The regulation loop 
is somewhere between one RTT and RTT + FT long without feedback losses, and 
with losses it could be as long as RTT + m*FT where m is the number of 
feedback interval until one makes it in the feedback. Thus the congestion 
volume in a static system becomes (R[x+1]-R[x])*(RTT + m*FT) and where RTT 
+m*FT is capped by the longest non responsive interval. So have much data this 
represents, i.e. how badly one affect the on path with overload. So this is 
highly dependent on the parameters which is why I think this is so hard to 
evaluate.

> determining the sending behaviour during st, rather than a table.

I am not certain if I understand this correctly. Are you proposing to have a 
congestion window for the increase, or the full transmission, so that it will 
throttle if it doesn't get feedback in a timely way?

I think the main problem here is that one need to analyze the parameter space 
for the limiter that are proposed and what impact the parameters have. That 
has been part of my problem analyzing the updated text.

Cheers

Magnus Westerlund

---

From: <[email-masked]>
Date: Fri, 12 Mar 2021 14:26:17 +0000

Hi Magnus,

Thanks. My comments refer to the ramp up phase of the measurement only, an important aspect I forgot to mention.

The sender has a  congestion budget Buffer_max[ms]@R[x] based on the last sending rate R[x] confirmed by FT[x] without congestion, when it starts to send by R[x+1] . 
- It takes RTT + FT, before that rate is adapted as a minimum. 
- As the smallest optimistic waiting time, after st[x+1] + Buffer_max[ms], there's no more queue at the receiver's access (if there's no competing traffic). 

That would help to determine a function when to earliest start the next st[x+2]. A behaviour definition specifying how to react after FT + RTT + Tolerance without any feedback may be as follows: 
- Repeat after Timeout by sending another st at rate R[x+1]. 
- Timeout = max ( st[x+1] + T_secure *  Buffer_max[ms] , FT + RTT + Tolerance ) with T_secure being an integer. "Tolerance" [s] on the order of some 10 [ms], I'd think.

Design doesn't have to be exactly like that, this is about the concept. Len, Al, please contact me if I missed an important part of the implementation (or if I here suggest to slow down the measurement to eternal).

Regards,

Rudiger (stick with "u", if the "ue" puzzles you)

-----Ursprüngliche Nachricht-----
Von: Magnus Westerlund <[email-masked]> 
Gesendet: Freitag, 12. März 2021 14:11
An: Geib, Rüdiger <[email-masked]>
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Betreff: RE: Magnus Westerlund's Discuss on draft-ietf-ippm-capacity-metric-method-06: (with DISCUSS)

Hi Reudiger,

Cheers

Magnus Westerlund

---

From: Magnus Westerlund <[email-masked]>
Date: Fri, 12 Mar 2021 17:35:56 +0000

Hi,

I do provide some comments inline.

As I am no longer a member of the IESG list, you will need to ensure that I am 
in to or cc field to ensure that I do receive future emails on this topic that 
you want my feedback on.

> >
> > Section 2.
> >
> > I think the scope text is still fairly open. Yes it is clear that the
> > load algorithm is only intended for measurements. However, the usage
> > is not particular limited, especially in regards to my main concern of
> > edge to central nodes across multiple AS in the Internet like the
> > different TCP speed tests often are deployed. The security
> > consideration requirements are good for a number of reasons but put no
> > limitations on that aspect. So I would prefer a more explicit
> > statement here. I think an important aspect here is that any ISP
> > seeing issues from these measurements should know who to talk to. I
> don't know how to best formulate that.

Okay, I will try again to reword the scope next week.

> > So I think this definition is fine for lower rate as the number of
> > packets in each 1 ms burst is fairly small and the buffer it hits will
> > likely be relatively large compared to the increase in load. However
> > at higher rates like beyond 10 GBPS where 1 GBPS steps are
> > recommended. So transmitting at bursts every 1 ms intervals means that
> > one are transmitting 833 packets each burst at 10 GBP rate of 1500
> > bytes size, so likely even higher for more moderate 1200 byte size
> > packets. That is almost 1,3 mb of data. So where pacing may be quite
> > good at lower bit-rates < 1Gbps I wonder if it starts breaking down at
> > higher rates, which appears to in the region where buffers becomes
> > more shallow due to the cost of having large buffers and where good
> > pacing reduces the need for buffering. I would also note that the
> > reaction time for the control can be 1RTT + 50 ms which thus the
> > As the load algorithm hasn't been tested beyond 10Gbps and it appears
> > that the numbers can start to become more problematic at these speeds,
> > wouldn't it be better to say that this is not intended beyond 10 Gbps.
> >

So, I would prefer to raise this at least to 20 ms. However, I think that is 
for general robustness without knowledge about the network one is measuring.

> > Even the default means that one looses 10 feedback packets in a row.
> > That is a lot and shows that one have a serious interruption on the
> > return path. Already loosing 3 feedback packets in a row indicates
> > that one have significant outages if this is lost.
> >
> > Secondly, this is formulated only based on  intervals of FT. For
> > startup the RTT is relevant factor. So I think there are several
> > factors here for the timeout that maybe need to be teased apart? So
> > initially one offers a very low load and one may not have a good
> measurement on base RTT.
> [acm]

Yes, I think you can define this as time, since you last received a feedback 
message. However, my concern with what the appropriate for a value is actually 
dependent on the RTT as it effects the control loop. So the amount of damage a 
sequence of lost feedback message is dependent on the total time as the RTT is 
effecting the reaction time. So clearly receiving no feedback message for 10 
FT intervals is bad as it represents a lot of missing packets.

>
> > Thus, time to first feedback is okay to be fairly large and 500 ms is
> > likely okay but quite longer than expected for an access to local
> > internet exchange measurement. However, when one scale up the rate I
> > think these values are way to long as the total amount of traffic sent
> > without feedback becomes quite significant. Receiving no feedback for
> > more than 10 reporting intervals are already way to long. And to state
> > that 30 seconds would be an acceptable value I can't support even for a
> measurement tool.
> [acm]

I Understand the view that the interruption is a short time. However, a 
several second without feedback transmission continuing to transmit is also a 
significant issue. So I think everything beyond 1 second is not really 
acceptable.

> each time a load packet received. If the timeout expires, the receiver SHALL
> be closed and no further feedback sent.

Ok, that needs to be included.

>
>
> > Is the load packet timeout the time the receiver is waiting before
> > using signalling channel to end the measurement without receiving any
> > packets, or for the sender to receive feedback that says that no
> > packets have been received? The roles here are not clear.
> [acm]
> value each time a feedback message is received. If the timeout expires, the
> sender SHALL be closed and no further load packets sent.

Ok. So I think there are two aspects here. The separation between something 
that backs off a transmission, and terminates the whole measurement. Do you 
really like to have these two clumped together.

> Our concern is stopping a test unnecessarily. There can be a happy

Yes, and as I said above do you really want to have timeout of measurement be 
connected to a rate reductions timeout?

> The parameters above and several others simply determine where a test

Okay, I will consider if I have any proposal for how to make this clearer.

> Not really, UDP is mandatory in the metric definition.

Hmm, I think then there is a mismatch here. Section 6.3 states:

n0 is the total number of IP-layer header and payload bits that
      can be transmitted in standard-formed packets from the Src host
      and correctly received by the Dst host during one contiguous sub-
      interval, dt in length, during the interval [T, T+I],

So the metric appears to be defined based on the IP packets, not the UDP 
payload size. Thus, one need to convert between the value of ss and the actual 
packet size sent. Thus, I think the rate value in the table will be 
misinterpreted as a rate of 560 mbit/s based on ss = 1210 bytes would in fact 
be an IPv6 capacity of 582,2 mbps.

> We can remove it if the values cause confusion.

I think that should be done, unless it has a real purpose here.

> services we could use in production, and many others.

Understood, did really 5 ms work well in DOCIS and 4G Mobile? Or is it 30 ms 
that works well?

> or unexpected sequence numbers in the stream, or sequence errors.

So, this is the amount of change beyond the expected next sequential value 
should be used. Did you actually use 100 as threshold, i.e. that you need 
burst loss of 100 packets or reordering that moved a packet 100 out of 
sequence for it to be considered an error? What was the purpose of using a so 
high value?

>    threshold for two consecutive feedback intervals.

Okay I get it.

> The Default can be 3 * Fast mode increase if you want, that's what we

Yes, I think that makes more sense and avoid causing the values to not be 
related to.

> It might be removed, we thought it was useful info.

It would be good if these values are not input parameters, rather a 
consequence of others would be separated to its own category.

> The goal is to measure the true maximum rate during the test duration.

So pushing traffic out of the way during the test period. I think that should 
be made more explicit and with that stated explicit it is easier to make clear 
why this only should be deployed for measurement within cooperating 
administrative domains.

> awkward due to row height.  We'll add some definitions elsewhere.

Yes, please do. I would also recommend that you try doing these as XMLv3 
tables so they look much better in the HTML version.

> having already tried several times.

I starting to see how we can get this into where I personally will find it 
acceptable.

I think rewriting the applicability and make the intention clear will be the 
main part. I also think there should be a paragraph in the security 
consideration that makes it clear that deployments should prevent metrics to 
be run by clients that are outside of the intended administrative domains to 
prevent that this traffic can be used to interfere with other administrative 
domains traffic.

Cheers

Magnus Westerlund

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Tue, 16 Mar 2021 23:43:42 +0000

Hi Magnus,

We are continuing discussion of some points among the co-authors, 
but here are some replies for this particular thread.

Al 

Open Issues:  

Wording of Applicability in section 2. (rely on the reference 
to Section 2 of [RFC7479] )

Feedback message timeout (stop test) 
  - MW: Default 500ms too high (L=10, FT=50ms), 1sec upper limit (30sec too long)

Len proposes a safety measure to back-off the rate when there is no
feedback in the time-scale of 0.1 seconds or so.
Provides clear separation between this safety measure, normal operation, 
and a timeout to terminate the test. This process is untested/not implemented.

?? Remove table entry for cc, burst count ??

> Sent: Friday, March 12, 2021 12:36 PM
> To: MORTON, ALFRED C (AL) <[email-masked]>; [email-masked]
> As I am no longer a member of the IESG list, you will need to ensure that
> I am in to or cc field to ensure that I do receive future emails on this topic
> that you want my feedback on.
[acm] 
Yes, of course.
> Okay, I will try again to reword the scope next week.
[acm] 
I think the last paragraphs in Section 2 on Applicability are where you 
want to concentrate.  As I mentioned during the IPPM meeting, the reference 
to Section 2 of [RFC7479] provides an explicit and non-Internet-wide 
applicability, and has already been agreed by IPPM WG for the problem 
statement where our memo provides a solution.

> > >    |              |             |              | RECOMMENDED)
> |
> > >    |              |             |              | find the max
> |
> So, I would prefer to raise this at least to 20 ms. However, I think that
> is for general robustness without knowledge about the network one is
> measuring.
[acm] 
Ok,  20ms <= FT <= 250ms, it's in the working text.

[acm]
 
In the present wording, rate adjustments only take place when new feedback 
arrives. Sometimes the algorithm determines that no rate change is warranted.

Len's thought is that we could introduce a safety measure: rate reduction
each time a feedback status message is declared lost. We would need to wait
*some amount of time* to avoid over-reaction (2*FT ?), and then execute Rx-1
at 2*FT, again at 3*FT, again at 4*FT, etc. until L*FT when the procedure 
to stop a test begins.

A point is that safety measures like the above and other time-outs go
beyond the method somewhat. For example, the Load Rate Adjustment Algorithm
begins when a feedback status message arrives and ends when the new rate 
has been determined, waiting for the next message.  
The safety measure described above begins when feedback status messages 
are absent for a period of time, and haven't been tested (yet).

> several second without feedback transmission continuing to transmit is
> also a significant issue. So I think everything beyond 1 second is not really
> acceptable.
[acm] 
This seems to be a very conservative opinion. How many applications 
like streaming video, VoiP, etc. will ride-out a 1 second outage and 
keep trying to restore communication on their own (so that the user
doesn't have to do it) ??

And, is there value to measure the duration of an outage when it 
occurs during testing? If you always quit quickly, because the 
Expected Safe Range demands it, you'll won't know whether an 
application would self-restore or not.  And I know that we haven't
talked about outage measurement in the draft, but we clearly intend
to measure Capacity and additional metrics.

> Ok, that needs to be included.
[acm] 
OK it's in the working text.

> that backs off a transmission, and terminates the whole measurement. Do
> you really like to have these two clumped together.
[acm] 
We were satisfied with that strategy *when this timeout = 5 seconds*.

Now that you would likely prefer to see the timeout in the < 0.5 seconds
range, Len proposes a safety measure to back-off the rate when there is no
feedback (a.k.a. ACK of working path), in the time-scale of 0.1 seconds
or so.

There is clear separation between this safety measure, normal operation, 
and a timeout to terminate the test.

What do you think?

> Yes, and as I said above do you really want to have timeout of measurement
> be connected to a rate reductions timeout?
[acm] 
The timeout setting of 5 seconds during a 10 second test removed any
ambiguity in our minds.  In Half of randomly occurring 5 second outages,
the sender would be terminated by the test duration anyway.

So, let's see what we can do with some separation and a new safety measure
using absence of feedback at the sender.

> > >    | 0            |             |              |
> |
> > >    | 1            |             |              |
> |
[acm] 
BTW this row seems clearer if we write the tested range as:

+--------------+-------------+--------------+-----------------------+
| table index  | 1Mbps       | 1Mbps<=rate  | same as tested        |
| (step) size  |             | <=1Gbps      |                       |
+--------------+-------------+--------------+-----------------------+

> misinterpreted as a rate of 560 mbit/s based on ss = 1210 bytes would in
> fact be an IPv6 capacity of 582,2 mbps.
[acm] 

We can certainly calculate the IP-layer rates that each payload size 
produces using UDP and IP headers, and those are the rates we are 
referring to when talking about 1Mbps step sizes, starting rates,
transition rates, etc.

Sure, we need a table for v4 and v6. The running code can use v6 
address family, too.

> > >    | count        |             |              |
> |
[acm] 

OK , will check with co-authors on this first.

> > >    | range        |             |              |
> |
> > >    | threshold    |             |              |
> |
> Understood, did really 5 ms work well in DOCIS and 4G Mobile? Or is it 30
> ms that works well?
[acm] 

5ms worked well with PON, and ok with DOCSIS, 30ms worked well with 4G 
and **DSL, and still worked accurately with others having less delay 
variation.

> > >    | range        |             |              |
> |
> > >    | threshold    |             |              |
> |
> > >    | error        |             |              |
> |
> > >    | threshold    |             |              |
> |
> sequence for it to be considered an error? What was the purpose of using a
> so high value?
[acm] 

One of our open-source collaborators brought in measurements with dispersed
losses and reordered packets (not bursts of anything). The threshold of 
100 worked well in these unexpected circumstances.  All we are claiming is
that we HAVE tested at 100 using the running code.

> > >    | report       |             |              |
> |
> > >    | threshold    |             |              |
> |
> > >    | steps        |             |              |
> |
[acm] 

OK, done in the working text.

[acm] 

OK, I turned that row into a sentence in the working text.

> So pushing traffic out of the way during the test period. 
[acm] 

Aren't you assuming the measurement system allows other traffic during a test?
The instructions can tell a naïve user to stop other traffic, or they may 
not see accurate results ...  All the ad hoc tools FAQ that, and then launch
9 or 10 TCP streams to a low-RTT host that pushes all the non-measurement 
traffic out the way during the test period.

Like I said at the IPPM session, we pound hard for a few seconds, then go away.
There were some testing scenarios where our load adjustment did not push
all other traffic out of the way (likely 100s of connections were active).

> I think that should
[acm] 

Because my current tool is v2, I'm a v2 guy. But we added the definitions 
agreed so far.

> I think rewriting the applicability and make the intention clear will be
> the main part. I also think there should be a paragraph in the security
> prevent that this traffic can be used to interfere with other
> administrative domains traffic.
[acm] 

the reference to Section 2 of [RFC7479] will help you.