Subject: Re: [ippm] New QUIC Packet Loss Measurement (draft-cfb-ippm-spinbit-measurements)

From: "Lubashev, Igor" <[email-masked]>
Date: Sat, 9 May 2020 06:00:10 +0000

Fabio,

I think we are in a complete agreement that any loss reporting is a very useful extension – both for monitoring the network and for troubleshooting individual flows. I think we agree that L-bit and R-bit are better than nothing.  The disagreement is about the relative fitness for the purpose and ease of deployability.

I would also welcome other people to weigh in with novel insights.  Maybe more people will catch up on both drafts before the next IETF.

I am happy to get into the details of each point (inline), but my biggest concern is aligned with Dmitri’s concern that:

   (a) Simultaneous captures/observations by different organizations of infrequently occurring events is a futile proposition, and
   (b) A very large percentage of internet flows is asymmetric.

As a result, a method that relies of either of these properties is limiting its utility for troubleshooting customer problems.

(The rest of the comments are inline)

> On Tue, April 28, 2020 at 9:51 AM, Bulgarella Fabio (Guest) <fabio.bulgarella=[email-masked]> wrote:
>
> Hello Igor,
> thanks for your reply.
>
> This exchange of e-mails is very useful to understand which of the two solutions has the best balance between positive and negative aspects,
> at least in the measurement scenarios of interest for telecommunication operators. The important thing is that one of the two solution is
> chosen and a packet loss feature is finally included in the Quic protocol (after the necessary checks in terms of privacy and security).
> However, we can already say, from the point of view of Telecom Italia and Orange, that the Qbit is a fixed point of this type of measures.
> It remains to decide which is better between L-bit and R-bit.
>
> First of all, let's remember the reasons that led us to design the Rbit.
> We found some weaknesses in the Lbit:
>
> - the LBit logic is strictly tied to the protocol implementation.
> For example in LSQUIC there are 6 different hook points where this variable is set. QL performance mechanism logic is intertwined with
> the protocol logic. So, the concern here is that future changes in the protocol or in new implementations need a careful analysis of the
> code to avoid different behaviors and different measurement results;

The quality of the implementation of each stack is mostly related to the implementor.  The behavior for both L-bit and R-bit is specified by the standards, so if the standards are implemented correctly, no analysis of the implementation should be needed.

> - loss of packets carrying only ACK frames (are detected?).
> We saw in the Montreal demo (using Ericsson SpinDump tool), the last July, that if Quic packets only contain ACK frames the Lbit is not set.
> Actually, during a file download we inserted losses in the upload direction that were not detected by the system.
> Alexandre confirmed this behavior. Have you changed something since then?

With QL-bits, each endpoint is free to implement or NOT implement the marking.  The beauty of the QL scheme is that if an endpoint chooses to mark packets, those markings are sufficient to signal upstream and downstream loss of all packets sent by THAT endpoint. In the implementation you are referring to, it looks like the client endpoint did NOT choose to implement QL markings, so its packets (pure ACKs) did not provide loss signaling.  If the client did implement QL, those pure ACKs would report losses all the same.

> - the loss measurement signal is inaccurate in case of losses:
> The Lbit is a simple signaling protocol. If you miss a Lbit marked packet you miss a signal of loss. If losses are not equally distributed between
> L-marked and not L-marked packets, it's very likely that the produced measurements are wrong or at least inaccurate. The two approaches
> are completely different. Rbit works exactly as the Qbit. It counts the lost packets, it's not a signaling. So, as the Qbit, the correct percentage
> of lost packets is detected.

There is no difference between QL- and QR- schemes in regard to the signal quality in case a loss of packets with the signal. As you note, we discuss in the draft that it is conceivable for a somewhat different loss ratio for packets w/ L=0 and L=1 for QL-bit scheme. Likewise, it is conceivable for a different loss ratio for packets belonging to “shorter” and “longer” R-trains for QR-bit scheme. After all, when there are losses to report (shorter R trains), the ACKs will be sent more frequently, increasing the chance of the return path congestion.

> Now some inline answers to your considerations, even though at this point it would be convenient to organize a comparison involving also
> other people interested in the topic, maybe in a next IETF meeting. We are also available to carry out comparative tests on the same flows
> (marking simultaneously with both solutions) using for example the Ericsson SpinDump tool which has already been used in the previous
> hackathons as QUIC performance measurement tool.
>
>> Strong reordering exists.  That’s why we are recommending the minimum Q to be 64. This comes from real data, not lab data.
>> If 5 packets were enough to tell the end of marking periods, we would had recommended minimum Q to be 8.
>
> The length of the period is not chosen only in function of reordering but also to avoid that burst losses completely cancel one or more periods.

That's right, that is a factor also.  However, we have not observed losses that obliterate several dozens of consecutive packets, though I am sure they exist somewhere.  Consistent packet reordering of that magnitude we did see.

> However, stimulated by Igor's concerns, we thought of an improvement regarding the tolerance of our system against out of sequence.
> In a nutshell, it would be possible to intercept reordered packets received after the detection of an incoming "Qbit marking period", using them
> to update the computed average until the current "Rbit marking period" is completed in generation.

Sure, though the receiver endpoint has access to the packet numbers and knows the loss rate exactly and can signal it directly.

> After the necessary laboratory tests, we could add this new feature to the next revision of our draft.
>
> R signal is still going to be significantly delayed, especially with slower connections (not every connection is going to be streaming media at a
> sustained rate of 15Mbps). So if for a 15Mbps average-rate connection you can expect to see R-signal with a delay between 8-RTTs (1:2 ACK)
> and 38-RTTs (1:10 ACK), the delay increases to 24-RTTs and 114-RTTs for a 5Mbps average-rate connection.
>
> As already said in the previous mail, based on our tests, this is not a big problem because it's still possible to obtain enough measurements to
> evaluate the packet loss of a session. Moreover, the measurement delay is shortened when packet loss occurs.

It depends.  For a large media download it is probably enough data for generic network monitoring.  It would not work for shorter flows.  Also, as I showed above, even with an ack2 policy, you may get a 38-RTT delay for very realistic network media flows, which will make troubleshooting specific flows difficult.

>> L-bit signal provides a crisp shape of the loss – whether it is random loss, or packets are lost of batches (and how many packets are in a loss batch).
> By contrast, R-signal only indicates an average loss over 2 to 10 Q-bit marking periods.  The share of the loss is erased. I think for that alone, L-signal
> is more useful for troubleshooting.
>
> Although we calculate the E2E loss through an average (only if the reflection is on a slower channel than the signal reception channel), we are still
> able to accurately calculate the loss using a correction factor (see https: //tools.ietf.org/html/draft-cfb-ippm-spinbit-measurements-00#section-7.4).

I think you did not understand the concern here.  The specific shape of the loss may be very useful for a network troubleshooting.  QL-signal allows one to observe the precise shape of the loss.  QR-scheme only gets you an average rate over multiple RTTs.

In fact, even if the loss is on the slower channel and reflection is on the faster channel, QR only gets an average loss rate rather then the precise shape of the loss.

>> This is an important difference, and it seems accurate on its surface.  In reality, every QUIC implementation will keep track of packets and declare them
>> lost for the purposes of data retransmission and/or congestion control.  The timing and details of when packets are declared lost (L-bit signal) will vary.
>> Likewise, every QUIC implementation will send some ACKs, but their timing and frequency (R-signal) will vary.  R-signal quality will also heavily depend
>> on the “marking period detection” implementation quality and “Q-signal averaging” implementation details.
>
> See first point in the L-bit weeknesses paragraph. Moreover, the two functions "marking period detection" and "Q-signal averaging", really simple in
> implementation, are completely detached from the rest of the protocol logic and their behaviors are expressly indicated in the draft.

There is no particular L-bit weakness here.  See my comment above – the precise timings of the signals will vary in both approaches.  The timing of L-bit will vary based on the sender’s loss detection.  The timing of R-bit will vary based on the receiver’s ACK policy (and also the details of their “reordering mitigation” algorithms).

>> Also, accurate accounting with QR scheme requires correlation of both directions, with the R-bit direction significantly lagging the Q-bit direction.
>> It should be understood how this scheme behaves in cases of changing Connection IDs.
>
> We do not see the problem of implementing the solution on both endpoints since generally the client, unlike the server, does not have scalability problems.
> So if you implement it on the server it costs nothing to implement it on the client as well.

There is no scalability concern for the QL implementation at all.  It is just two counters that are incremented and decremented.

The real concern is developer skills and developer resources.  Servers typically have substantial development organizations that will get it right. Except for a handful of most popular desktop and phone clients, however, clients have very limited developer support and update story. A lot of bits are pushed by clients built into various streaming devices, smart TVs, set top boxes, BluRay players, and some such.  Their implementations tend to be minimal and somewhat sloppy, and their upgrade path is sporadic at best.

Best regards,

- Igor

________________________________________
Da: Lubashev, Igor <mailto:ilubashe=[email-masked]>
Inviato: giovedì 23 aprile 2020 18:41
A: Bulgarella Fabio (Guest); Ian Swett; Cociglio Mauro
Cc: Riccardo Sisto; mailto:[email-masked]; mailto:[email-masked]; mailto:[email-masked]; IETF IPPM WG (mailto:[email-masked])
Oggetto: [EXT] RE: [ippm] New QUIC Packet Loss Measurement (draft-cfb-ippm-spinbit-measurements) 
 
Fabio, thanks for the reply.
 
• “Moreover, with a strong amount of reordering the Qbit (and the SpinBit) would not work either.”
 
Strong reordering exists.  That’s why we are recommending the minimum Q to be 64.  This comes from real data, not lab data.  If 5 packets were enough to tell the end of marking periods, we would had recommended minimum Q to be 8.

• R-bit signal delay.
 
R signal is still going to be significantly delayed, especially with slower connections (not every connection is going to be streaming media at a sustained rate of 15Mbps).  So if for a 15Mbps average-rate connection you can expect to see R-signal with a delay between 8-RTTs (1:2 ACK) and 38-RTTs (1:10 ACK), the delay increases to 24-RTTs and 114-RTTs for a 5Mbps average-rate connection.
 
• R-bit signal quality.
 
L-bit signal provides a crisp shape of the loss – whether it is random loss, or packets are lost of batches (and how many packets are in a loss batch).  By contrast, R-signal only indicates an average loss over 2 to 10 Q-bit marking periods.  The share of the loss is erased.
 
I think for that alone, L-signal is more useful for troubleshooting.

• L-bit signal depends on each implementation’s determination of when to “declare a packet lost” (which may differ for each implementation), while R-signal depends only on more generic behaviors.
 
This is an important difference, and it seems accurate on its surface.  In reality, every QUIC implementation will keep track of packets and declare them lost for the purposes of data retransmission and/or congestion control.  The timing and details of when packets are declared lost (L-bit signal) will vary.  Likewise, every QUIC implementation will send some ACKs, but their timing and frequency (R-signal) will vary.  R-signal quality will also heavily depend on the “marking period detection” implementation quality and “Q-signal averaging” implementation details.

• End-to-end loss: “We can also place two independent observers wherever on the two directions and they will still produce coherent E2E measures.”
• End-to-end loss: “by observing only one direction we can measure the […] the end-to-end in the opposite direction (Rbit).
 
Correct, as long as both endpoints implement the entire QR scheme (even if this is an unusual setup – measuring path A requires observing path B).
 
Also, accurate accounting with QR scheme requires correlation of both directions, with the R-bit direction significantly lagging the Q-bit direction.  It should be understood how this scheme behaves in cases of changing Connection IDs.
 
Best wishes,
 
• Igor

From: Bulgarella Fabio (Guest) <mailto:fabio.bulgarella=[email-masked]> 
Sent: Wednesday, April 22, 2020 3:12 PM
To: Lubashev, Igor <mailto:[email-masked]>; Ian Swett <mailto:[email-masked]>; Cociglio Mauro <mailto:[email-masked]>
Cc: Riccardo Sisto <mailto:[email-masked]>; mailto:[email-masked]; mailto:[email-masked]; mailto:[email-masked]; IETF IPPM WG (mailto:[email-masked]) <mailto:[email-masked]>
Subject: Re: [ippm] New QUIC Packet Loss Measurement (draft-cfb-ippm-spinbit-measurements)
 
Hello Igor,
here some thoughts about your concerns.
 
1. If big differences in delay are present between two parallel paths on which the same connection is divided, this is certainly an unwanted behavior, because it could create problems to the protocol itself even before damaging the performance measurements. As a network operator, we try to avoid such scenarios. However, our mechanism works correctly in most cases and is still resistant to normal reordering. Moreover, with a strong amount of reordering the Qbit (and the SpinBit) would not work either.

2. It's true that your mechanism allows the measurement of E2E and upstream loss of the observed direction. However, these measurement methodologies are born with the idea of providing information on the performance of the entire connection, as was done with the SpinBit. Actually with our mechanism, when observing a single direction, we still are able to measure the E2E loss of the opposite channel (using the R-bit).

3. The ACK ratio of 1:10 should only occur in the absence of loss. By introducing loss, the ratio between ACK and data is certainly better. So the quality of our signal improves just when it is needed; we don't care if we get delayed measurements when everything is working great. This behavior is confirmed by our laboratory tests where we have seen that loss can be easily detected in an extremely accurate manner even in the case of highly unbalanced flows.
 
L-Bit has an RTO delay but to evaluate an accurate loss ratio you have still to observe a significant number of packets.

4. First of all, our solution is not dependent on any specific protocol implementation unlike the QL and therefore there shouldn’t be problems related to the quality of the implementation. In other words, our portion of code is always the same, in every protocol implementation.
Secondly, as anticipated in point 2, by observing only one direction we can measure the losses in the operator's domain (Qbit) and the end-to-end in the opposite direction (Rbit). So, our solution does work also for asymmetric path segments. We can also place two indipendent observers wherever on the two directions and they will still produce coherent E2E measures.
 
The QL solution does not give any information about the opposite direction, it only measures what it observes, that is the current direction. Having two independent measures, one per direction, can be an advantage in some aspects, but a disadvantage for others.

Best regards,
Fabio B..

________________________________________
Da: ippm <mailto:[email-masked]> per conto di Lubashev, Igor <mailto:ilubashe=[email-masked]>
Inviato: martedì 21 aprile 2020 18:31
A: Ian Swett; Cociglio Mauro
Cc: Riccardo Sisto; mailto:[email-masked]; mailto:[email-masked]; mailto:[email-masked]; IETF IPPM WG (mailto:[email-masked])
Oggetto: [EXT] Re: [ippm] New QUIC Packet Loss Measurement (draft-cfb-ippm-spinbit-measurements) 
 
I like the QR draft from the perspective that it mostly works (and lots of credit is due for things that work).
 
I have a few concerns, though.
 
1. The most trivial one is that QR is actually harder to implement correctly for endpoints, given the tricky processing required to identify when a marking interval has ended.  Experimental data that Akamai & Orange compiled shows that there are networks that do ECMP among links with significantly different latencies.  They will cause a lot more reordering to a burst of packets than the recommended “observe 5 packets (in a row?) of the opposite markings” rule can address.  You might need to potentially defer the decision on where exactly the marking period boundary is until you’ve seen almost half of the next marking period.  Requiring every endpoint to implement this right is tough.
 
2. Since the most important direction (at least for downloads and media streaming) is server-to-client, to get this signal with QR, you need both endpoints implementing the logic, with the heaviest lift on the clients (see above).  For QL, a client that does not want to implement QL logic but is ok with the loss data reporting can just update the TP and header protection mask – all the lift is on the server side.
 
3. I am concerned with the quality of the signal you get from QR.  While the upstream loss signal is strong (same Q-bit), the downstream/end-to-end loss signal is weak and is likely to result in very delayed and inconclusive signal.  Google’s experience (as I understand it) and protocol research (https://urldefense.proofpoint.com/v2/url?u=https-3A__erg.abdn.ac.uk_-7Edownloads_ackscaling.pdf&d=DwMF-g&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=u3hS0QnHLn0-P-Lmy8-Yhm7L0G310RjwTQfw2XLvfJk&s=8qb_Zcm3YASCVrMZiZ5Le_3_nPxd8AC7bUbVwXl4L9I&e=) point to the benefit of ACK-ing only 1:10 packets after the initial slow start (draft-fairhurst-quic-ack-scaling).  Since such behavior has a beneficial impact on networks, mobile devices, and endpoint cpu, and no negative impact on the protocol performance, I expect it to be in common use.  With 1:10 ACK ratio, R-bit signal is significantly delayed as it is lagging the loss by at least 1400*(Q=64)*2*10=1.8MB of data transferred, which is at least 38 RTTs for a 15Mbps stream over a 25ms link (L-bit signal is delayed by an RTO).
 
4. QR scheme only works for symmetric path segments, when the observer can capture both directions of the flow. You also cannot compute end-to-end loss in any direction, unless both endpoints implement the entire QR algorithm.  In contrast, QL works for single-direction observers, and whichever endpoint implements QL algorithm ensures end-to-end loss signal for the packets it sends, and the quality of that loss signal solely depends on the quality of that endpoint’s implementation (not the quality of the other endpoint’s implementation).  QL seems more deployable.
 
• Igor

From: Ian Swett <mailto:[email-masked]> 
Sent: Wednesday, April 8, 2020 3:44 PM
To: Cociglio Mauro <mailto:mauro.cociglio=[email-masked]>
Cc: mailto:[email-masked]; Lubashev, Igor <mailto:[email-masked]>; mailto:[email-masked]; mailto:[email-masked]; Riccardo Sisto <mailto:[email-masked]>; IETF IPPM WG (mailto:[email-masked]) <mailto:[email-masked]>
Subject: Re: [ippm] New QUIC Packet Loss Measurement (draft-cfb-ippm-spinbit-measurements)
 
Thanks for sharing this.  From my perspective, this is an improvement upon the previous proposal in terms of robustness.  This design feels very similar to the spin bit, and seems trivial to implement, which are also nice properties.
 
Also, QUIC doesn't really have retransmissions, so the 'R bit' name always made me a bit uncomfortable.
 
I'm not saying QUIC should adopt this yet, but I'd be interested in seeing a privacy analysis completed for it.
 
Ian
 
On Wed, Apr 8, 2020 at 4:25 AM Cociglio Mauro <mauro.cociglio=mailto:[email-masked]> wrote:
Dear QUIC WG Members.
 
We submitted to IPPM WG a draft where we described a new 2 bit packet loss methodology (https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dcfb-2Dippm-2Dspinbit-2Dmeasurements-2D01&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=e_BQ3UDNSWkXu_ID1rkGv3CCZs89sezu-6Ibkvltf7Y&s=UeTW_fLvxHy_ewVvm0y6kIqZoXUR8PYWgxAOxmQScYw&e=).
The measurement of packet loss is under discussion in QUIC WG and our draft introduces two alternatives about it.
The first one is a spin-bit dependent signal and uses a single bit. The second one, described in the following linked slides, is a standalone solution based on a two bits loss signal and on alternate marking (RFC8321)..
This last methodology improves, in our opinion, the algorithm proposed by Orange and Akamai described in https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dferrieuxhamchaoui-2Dquic-2Dlossbits-2D03&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=e_BQ3UDNSWkXu_ID1rkGv3CCZs89sezu-6Ibkvltf7Y&s=V649Uqu8mfuBWvMRBTUkpEWfpEqlLfufGs67XXzvNFo&e=.
 
The 2 bits are the sQuare bit (Q-bit) and the Reflection square bit (R-bit).
The Q-bit doesn’t change from the Ferrieux-Hamchaoui draft but the R-bit substitutes the L-bit.
This avoids the L-bit dependence from an internal protocol variable, a problem raised in the last QUIC interim meeting.
 
You can find the slides, we prepared for the IPPM interim meeting, at the following link: https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ietf-2Dippm_meeting-2Dmaterials_blob_aa486f7ead28b9f55c5ef499e5c9ed33ab06daf9_ietf107-2Dvirtual_Slides_07-2Dspinbit-2Dmeasurements.pdf&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=e_BQ3UDNSWkXu_ID1rkGv3CCZs89sezu-6Ibkvltf7Y&s=3l6t3FYQn0iz_xX0sxNFKUnh7bwYFMOpQ0z_zNhXHpg&e=
 
Comments and suggestions are always welcome.
 
Of course we are available to present our proposals in the next QUIC WG meeting or to arrange a Webex side meeting if needed.
 
Best regards.
 
Mauro, Fabio, Giuseppe, Massimo and Riccardo

_____________________
Mauro Cociglio
TIM - CT.TA.EI
Via G. Reiss Romoli, 274
10148 - Torino (Italy)
Tel.: tel:+39%20011%20228%205028
Mobile: tel:+39%20335%20766%209751
_____________________

https://urldefense.proofpoint.com/v2/url?u=https-3A__on.tim.it_banner-2Dmail-2Ddip&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=e_BQ3UDNSWkXu_ID1rkGv3CCZs89sezu-6Ibkvltf7Y&s=z55QpRg48UYrhUvomUNAlkMCUxdQT20Mz1VhZkNrLQI&e=
Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione, Grazie. 

This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised. If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail, Thanks... 

Rispetta l'ambiente. Non stampare questa mail se non è necessario. 
_______________________________________________
ippm mailing list
mailto:[email-masked]
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_ippm&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=e_BQ3UDNSWkXu_ID1rkGv3CCZs89sezu-6Ibkvltf7Y&s=YF4GnFEgwmSakC8Bv_763sIZZR3g8t7wdsuiOl6C8fU&e=

---

From: "Bulgarella Fabio (Guest)" <[email-masked]>
Date: Thu, 14 May 2020 13:08:35 +0000

Hi Igor and all QUIC/IPPM WG members,

I think it's clear to everyone that both systems have advantages and disadvantages. In this email exchange, we both expressed our views on it.
At this point we would ask to WG members if they think it would be interesting to see a comparative test between the two methodologies.

Our proposal is to implement both solutions on a common implementation of the QUIC protocol so that the same data flow can be marked using both techniques (i.e., Qbit, Rbit and Lbit - borrowing the third most significant bit occupied by the spinbit) and analyzed using the two observers.

Waiting for your feedbacks on this proposal,
Best regards.

Fabio, Mauro, Massimo, Giuseppe and Riccardo

________________________________________
Da: Lubashev, Igor <ilubashe=[email-masked]>
Inviato: sabato 9 maggio 2020 08:00
A: Bulgarella Fabio (Guest); Ian Swett; [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]; IETF IPPM WG ([email-masked])
Oggetto: [EXT] RE: [ippm] New QUIC Packet Loss Measurement (draft-cfb-ippm-spinbit-measurements)

Fabio,

Best regards,

- Igor

• R-bit signal delay.

• R-bit signal quality.

Best wishes,

• Igor

Hello Igor,

Best regards,
Fabio B..

• Igor

Ian

Best regards.

_____________________
Mauro Cociglio
TIM - CT.TA.EI
Via G. Reiss Romoli, 274
10148 - Torino (Italy)

---

From: Marcus Ihlar <[email-masked]>
Date: Thu, 14 May 2020 13:52:15 +0000

Hi,
I agree with the original problem that we need a good way to do packet loss measurements for both aggregates and individual flows.
I'm not yet in a position to say which methodology is the most beneficial one from the set of cases I'm interested in (SLA assurance and troubleshooting mainly).

In Spindump we have support for both measurement methodologies and could quite easily extend it to observe loss using both methodologies simultaneously.
So I like the approach that Fabio is describing here, perhaps this could be a good Hackathon project?

Marcus Ihlar

-----Original Message-----
From: QUIC <[email-masked]> On Behalf Of Bulgarella Fabio (Guest)
Sent: den 14 maj 2020 15:09
To: Lubashev, Igor <ilubashe=[email-masked]>; Ian Swett <[email-masked]>; [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]; IETF IPPM WG ([email-masked]) <[email-masked]>
Subject: Re: [ippm] New QUIC Packet Loss Measurement (draft-cfb-ippm-spinbit-measurements)

Waiting for your feedbacks on this proposal, Best regards.

Cc: [email-masked]; [email-masked]; alexandre.ferrieux@orange..com; IETF IPPM WG ([email-masked])

Fabio,

I think we are in a complete agreement that any loss reporting is a very useful extension - both for monitoring the network and for troubleshooting individual flows. I think we agree that L-bit and R-bit are better than nothing.  The disagreement is about the relative fitness for the purpose and ease of deployability.

I am happy to get into the details of each point (inline), but my biggest concern is aligned with Dmitri's concern that:

> This exchange of e-mails is very useful to understand which of the two
> solutions has the best balance between positive and negative aspects,
> at least in the measurement scenarios of interest for telecommunication operators. The important thing is that one of the two solution is chosen and a packet loss feature is finally included in the Quic protocol (after the necessary checks in terms of privacy and security).
> For example in LSQUIC there are 6 different hook points where this
> variable is set. QL performance mechanism logic is intertwined with
> the protocol logic. So, the concern here is that future changes in the
> protocol or in new implementations need a careful analysis of the code
> to avoid different behaviors and different measurement results;

> The Lbit is a simple signaling protocol. If you miss a Lbit marked
> packet you miss a signal of loss. If losses are not equally
> distributed between L-marked and not L-marked packets, it's very
> likely that the produced measurements are wrong or at least inaccurate. The two approaches are completely different. Rbit works exactly as the Qbit. It counts the lost packets, it's not a signaling. So, as the Qbit, the correct percentage of lost packets is detected.

There is no difference between QL- and QR- schemes in regard to the signal quality in case a loss of packets with the signal. As you note, we discuss in the draft that it is conceivable for a somewhat different loss ratio for packets w/ L=0 and L=1 for QL-bit scheme. Likewise, it is conceivable for a different loss ratio for packets belonging to "shorter" and "longer" R-trains for QR-bit scheme. After all, when there are losses to report (shorter R trains), the ACKs will be sent more frequently, increasing the chance of the return path congestion.

> Now some inline answers to your considerations, even though at this
> point it would be convenient to organize a comparison involving also
> other people interested in the topic, maybe in a next IETF meeting. We
> are also available to carry out comparative tests on the same flows (marking simultaneously with both solutions) using for example the Ericsson SpinDump tool which has already been used in the previous hackathons as QUIC performance measurement tool.
>
>> Strong reordering exists.  That's why we are recommending the minimum Q to be 64. This comes from real data, not lab data.

> In a nutshell, it would be possible to intercept reordered packets
> received after the detection of an incoming "Qbit marking period", using them to update the computed average until the current "Rbit marking period" is completed in generation.

> R signal is still going to be significantly delayed, especially with
> slower connections (not every connection is going to be streaming
> media at a sustained rate of 15Mbps). So if for a 15Mbps average-rate connection you can expect to see R-signal with a delay between 8-RTTs (1:2 ACK) and 38-RTTs (1:10 ACK), the delay increases to 24-RTTs and 114-RTTs for a 5Mbps average-rate connection.
>
> As already said in the previous mail, based on our tests, this is not
> a big problem because it's still possible to obtain enough measurements to evaluate the packet loss of a session. Moreover, the measurement delay is shortened when packet loss occurs.

>> L-bit signal provides a crisp shape of the loss - whether it is random loss, or packets are lost of batches (and how many packets are in a loss batch).
> By contrast, R-signal only indicates an average loss over 2 to 10
> Q-bit marking periods.  The share of the loss is erased. I think for that alone, L-signal is more useful for troubleshooting.
>
> Although we calculate the E2E loss through an average (only if the
> reflection is on a slower channel than the signal reception channel), we are still able to accurately calculate the loss using a correction factor (see https: //tools.ietf.org/html/draft-cfb-ippm-spinbit-measurements-00#section-7.4).

>> This is an important difference, and it seems accurate on its
>> surface.  In reality, every QUIC implementation will keep track of packets and declare them lost for the purposes of data retransmission and/or congestion control.  The timing and details of when packets are declared lost (L-bit signal) will vary.
>> Likewise, every QUIC implementation will send some ACKs, but their
>> timing and frequency (R-signal) will vary.  R-signal quality will also heavily depend on the "marking period detection" implementation quality and "Q-signal averaging" implementation details.
>
> See first point in the L-bit weeknesses paragraph. Moreover, the two
> functions "marking period detection" and "Q-signal averaging", really simple in implementation, are completely detached from the rest of the protocol logic and their behaviors are expressly indicated in the draft.

There is no particular L-bit weakness here.  See my comment above - the precise timings of the signals will vary in both approaches.  The timing of L-bit will vary based on the sender's loss detection.  The timing of R-bit will vary based on the receiver's ACK policy (and also the details of their "reordering mitigation" algorithms).

Best regards,

- Igor

* "Moreover, with a strong amount of reordering the Qbit (and the SpinBit) would not work either."

Strong reordering exists.  That's why we are recommending the minimum Q to be 64.  This comes from real data, not lab data.  If 5 packets were enough to tell the end of marking periods, we would had recommended minimum Q to be 8.

* R-bit signal delay.

* R-bit signal quality.

L-bit signal provides a crisp shape of the loss - whether it is random loss, or packets are lost of batches (and how many packets are in a loss batch).  By contrast, R-signal only indicates an average loss over 2 to 10 Q-bit marking periods.  The share of the loss is erased.

* L-bit signal depends on each implementation's determination of when to "declare a packet lost" (which may differ for each implementation), while R-signal depends only on more generic behaviors.

This is an important difference, and it seems accurate on its surface.  In reality, every QUIC implementation will keep track of packets and declare them lost for the purposes of data retransmission and/or congestion control.  The timing and details of when packets are declared lost (L-bit signal) will vary.  Likewise, every QUIC implementation will send some ACKs, but their timing and frequency (R-signal) will vary.  R-signal quality will also heavily depend on the "marking period detection" implementation quality and "Q-signal averaging" implementation details.

* End-to-end loss: "We can also place two independent observers wherever on the two directions and they will still produce coherent E2E measures."
* End-to-end loss: "by observing only one direction we can measure the [...] the end-to-end in the opposite direction (Rbit).

Correct, as long as both endpoints implement the entire QR scheme (even if this is an unusual setup - measuring path A requires observing path B).

Best wishes,

* Igor

Hello Igor,

4. First of all, our solution is not dependent on any specific protocol implementation unlike the QL and therefore there shouldn't be problems related to the quality of the implementation. In other words, our portion of code is always the same, in every protocol implementation.
Secondly, as anticipated in point 2, by observing only one direction we can measure the losses in the operator's domain (Qbit) and the end-to-end in the opposite direction (Rbit). So, our solution does work also for asymmetric path segments. We can also place two indipendent observers wherever on the two directions and they will still produce coherent E2E measures.

Best regards,
Fabio B..

1. The most trivial one is that QR is actually harder to implement correctly for endpoints, given the tricky processing required to identify when a marking interval has ended.  Experimental data that Akamai & Orange compiled shows that there are networks that do ECMP among links with significantly different latencies.  They will cause a lot more reordering to a burst of packets than the recommended "observe 5 packets (in a row?) of the opposite markings" rule can address.  You might need to potentially defer the decision on where exactly the marking period boundary is until you've seen almost half of the next marking period.  Requiring every endpoint to implement this right is tough.

2. Since the most important direction (at least for downloads and media streaming) is server-to-client, to get this signal with QR, you need both endpoints implementing the logic, with the heaviest lift on the clients (see above).  For QL, a client that does not want to implement QL logic but is ok with the loss data reporting can just update the TP and header protection mask - all the lift is on the server side.

3. I am concerned with the quality of the signal you get from QR.  While the upstream loss signal is strong (same Q-bit), the downstream/end-to-end loss signal is weak and is likely to result in very delayed and inconclusive signal.  Google's experience (as I understand it) and protocol research (https://urldefense.proofpoint.com/v2/url?u=https-3A__erg.abdn.ac.uk_-7Edownloads_ackscaling.pdf&d=DwMF-g&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=u3hS0QnHLn0-P-Lmy8-Yhm7L0G310RjwTQfw2XLvfJk&s=8qb_Zcm3YASCVrMZiZ5Le_3_nPxd8AC7bUbVwXl4L9I&e=) point to the benefit of ACK-ing only 1:10 packets after the initial slow start (draft-fairhurst-quic-ack-scaling).  Since such behavior has a beneficial impact on networks, mobile devices, and endpoint cpu, and no negative impact on the protocol performance, I expect it to be in common use.  With 1:10 ACK ratio, R-bit signal is significantly delayed as it is lagging the loss by at least 1400*(Q=64)*2*10=1.8MB of data transferred, which is at least 38 RTTs for a 15Mbps stream over a 25ms link (L-bit signal is delayed by an RTO).

4. QR scheme only works for symmetric path segments, when the observer can capture both directions of the flow. You also cannot compute end-to-end loss in any direction, unless both endpoints implement the entire QR algorithm..  In contrast, QL works for single-direction observers, and whichever endpoint implements QL algorithm ensures end-to-end loss signal for the packets it sends, and the quality of that loss signal solely depends on the quality of that endpoint's implementation (not the quality of the other endpoint's implementation).  QL seems more deployable.

* Igor

Ian

The 2 bits are the sQuare bit (Q-bit) and the Reflection square bit (R-bit)..
The Q-bit doesn't change from the Ferrieux-Hamchaoui draft but the R-bit substitutes the L-bit.
This avoids the L-bit dependence from an internal protocol variable, a problem raised in the last QUIC interim meeting.

Best regards.

_____________________
Mauro Cociglio
TIM - CT.TA.EI
Via G. Reiss Romoli, 274
10148 - Torino (Italy)