Subject: minutes for IPPM WG, 13 July 1999

From: "Will E. Leland" <[email-masked]>
Date: Mon, 9 Aug 1999 17:56:19 -0400 (EDT)

Here's the minutes from our hour at Oslo; the presentations are
available in PDF format by http and ftp from
    http://www.advanced.org/IPPM/Meetings/ietf45/
-- Will

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Tuesday, 13 July 1999 at 14:15-15:15
====================================

The meeting was chaired by the working group chairs, Will Leland
and Matt Zekauskas, and was reported by Paul Love.

AGENDA
1. Introduction & Status
2. IPDV
3. Loss pattern update
4. Futures and Milestones

IETF Home page: http://www.ietf.org/html.charters/ippm-charter.html
IPPM Home page: http://www.advanced.org/IPPM/

1. Introduction & Status, Matt Zekauskas (Advanced Network & Services)

Within the last two weeks a number of metrics were approved as
Proposed Standard.  Vern Paxson and Scott Bradner will write a
document that explains the meaning of a metric to be on "standards
track" (similar to the document for MIBs).  Connectivity (RFC 2498)
will be reclassified as a Proposed Standard, and the one-way delay,
loss, and round-trip delay drafts were all approved as Proposed
Standards.

Since the last meeting, the bulk transfer framework, IP delay
variation (jitter) and loss pattern drafts were all updated.
If anyone has comments on the bulk transfer framework, please
take them to the mailing list; it seems to the chairs that
the bulk transfer document is close to being done (at least
until we have experience with some individual bulk transfer
metrics).  The IPDV document has been picked up by Phil Chimento
(see below).  The loss patterns document has been updated,
but mostly cosmetically, and will be discussed today.

2 IPDV Update, Phil Chimento (University of Twente)

Since the last meeting, Phil Chimento has taken over the
IP Delay Variation draft after Carlo Demichelis switched
positions; Phil has released version 3 of the draft.
The major changes are that the "discontinuous samples"
definition was deleted from the draft, and the old discussion
of IPDV mean value was also deleted.  A new section was
added to clarify distributions of variation, and a new statistic
was added for jitter that follows Van Jacobson's definition
from DiffServ.  In addition, a new section on the relationship
with other standards (ITU-T) was added, as well as a short
section on security.  The next version will clean up the
discussion of clock skew and the effects of taking IPDV
measurements when the clocks are not synchronized.

Phil then went through a discussion on the components of delay
(see slide) to ensure that we all understood what delay variation
really means, as presented in this draft.  The specific emphasis was
on the relationship between pair-wise delay variation versus change
in delay from some absolute:  even where each pair-wise delay variation
is small, the total variation can be large.

There was also a discussion of the ITU's delay variation measures
for 1-point and 2-point delay variation of ATM cells, and the
non-normative appendix to the IP measurement specification
which discusses 2-point delay for IP.  The ITU's specification
is based on absolute variation from a reference, either a
particular value (e.g., the first) or the average of a set
of values.

Finally, Phil turned to questions for the working group --
essentially is the IPDV metric headed in the right direction?

In particular, what is the purpose of IPDV?
- Jitter Estimation?  If so, in the EE sense (deviation from
  absolute) or the Computer Science sense (relative differences;
  for example, Van's definition from the expedited forwarding
  specification, or min & max differentiation with distribution)?	
- Sizing Buffers?
- Traffic characterization?
- Estimation of queuing delay?

What would you do if you had reams and reams of numbers?

One ISP in the audience said that their need was for measuring real
time voice characteristics and capacity planning.  These kinds
of uses imply that summaries and trends over time are important.
Another audience member mentioned that delay variation metrics
can serve to determine how large a token bucket is required to
provide unimpeded access to a packet stream that is intended to
be constant (or nearly constant) rate. Will mentioned that such
use relates to so-called "R/S statistics" developed in hydrology
for sizing dams.

3. Loss pattern update
	Introduction by Matt Zekauskas,
	Update by Steven Glass (Nokia)

Matt's introduction gave the basics about loss patterns and why
they are interesting, and about the loss pattern metrics which build
on one-way loss streams. [See slides.]

Steve pointed out the changes to the document, which include
many minor changes.  Section 3.3 added a clarifying example,
but it may have a typo.  There is a new Loss-distance-stream
metric.

Finally, Steve led a discussion about potential controversial
items and whether the metrics are ready to be submitted to the
IESG.  It is felt that the draft needs polish, but it's complete
and pretty close to being ready to go.

A question from the audience asked about the effect of the number
of drops causing retransmits and timeouts that cause retransmits
(specifically, TCP).  This particular item is out of scope for
the group, except as part of a bulk transfer capacity metric.
Matt Z. noted that Matt Mathis has talked about a "TCP MIB" --
exporting TCP variables.  However, that is out-of-scope for us.

There was a discussion about "Noticeable Rate" -- some applications
are sensitive to even two adjacent packet losses.

There was also a discussion about the fact that a burst loss
is considered terminated by a single good packet arrival.
A burst followed by a single good packet and another burst
would seem to be better characterized as one long burst.
Will noted that what is being measured could provide the
raw data for constructing a "less sensitive" metric.
Would some sort of "errored period" be a meaningful thing
to define?

Finally, there was a discussion about what to do with this draft.
Is it useful enough to move forward?  Should it be standards track?
Consensus from the group was that it should be cleaned up
and moved forward as an Experimental RFC.

4. Milestones and futures, Matt Zekauskas and Will Leland (Telcordia).

There wasn't much time left, but Matt presented the current thinking
on milestones.  [See the slides.] Key to the table:  The columns are
target dates -- months in 1999 and 1Q 2000.  The work items are on the
left.  Within the table, N means new draft, R means revised draft,
S means submit to the IESG.  "IPPM and ITU" is shorthand for a draft
that compares the IPPM metrics with the ITU metrics based on Will
Leland's presentation at the last IETF in Minneapolis.  The Delay-Loss
BCP is a best-current-practices document for the one-way delay and
loss metrics, based on at least the experience with the Advanced
Network & Services implementation and the RIPE NCC implementation.
The milestones are fairly aggressive, but the goal is to have all
the basic metrics done by 1Q 2000.

There were two questions from the floor: should Internet weather
maps be discussed here, and should we define MIBs for metrics.  Both
topics are apparently in-scope, as long as the maps are restricted
to the specific metrics under discussion.  However, they are not
currently specifically part of the charter.  There has been a
preference in the IETF for MIBs to be developed by the working group
that originates the related standards, so MIBs for metrics naturally
fall under our responsibility. We should look for help from people
already experienced in MIB design and use.

In continued informal discussions after the official WG slot ended,
the broad question was raised about the applicability of IPPM's
proposed metrics for SLAs: are our current metrics of use for SLAs?
Are there other metrics within our scope that could be useful for
SLA definitions? The Chairs urged that this discussion be raised
on the mailing list.

---

From: "Henk Uijterwaal (RIPE-NCC)" <[email-masked]>
Date: Tue, 10 Aug 1999 10:22:24 +0200 (CEST)

(Editted the distribution list a bit).

On Mon, 9 Aug 1999, Will E. Leland wrote:

> Here's the minutes from our hour at Oslo; 

> 4. Milestones and futures, Matt Zekauskas and Will Leland (Telcordia).

I don't see it anywhere in the minutes and I'm not even sure anymore if I
made the comment in Oslo :-), but I think that we need a (BCP?) document
describing how to compare numbers and distributions.  Contrary to to
protocols and other binary stuff, this is NOT immediately obvious. 

For example:  Suppose two implementations measure a delay over the same
path for a time-interval and quote that as (10%, median, 90%).  Now
implementation 1 finds (15ms, 30ms, 45ms) and #2 finds (13ms, 26ms, 39ms),
does that mean that the values are comparable and that the difference is
due to statistical fluctuations and sampling errors _or_ is there a
significant difference between the two?

The same goes for distributions.  Look back at the slides I showed in
M'polis.  I think we really need some agreed method better than just
looking at them, to say that 2 distributions are compatible. 

> currently specifically part of the charter.  

I believe the charter should be modified in this way.  During the last
months, I attended a number of meetings with the customers of ISP's that
are (or will) host one of our test-boxes, and the two things that always
came up were: 

1. Everybody seems interested in a "quality of the Internet" map, but 
   nobody seems to be able to define what should go in there (and if
   the IPPM metrics are indeed useful for this).  This group might be
   able to define what we can (not) provide for an Internet Weather Map

2. Can we use the results of your (delay/loss) measurements in our
   SLA.  People put values for delays/losses in SLA's but nobody 
   seems to put any method for measuring them in the SLA.  I think
   it would be useful if we could provide some "building blocks" 
   for SLA's.  Not the values, more something along the lines of

    - Metric
    - Method of measuring the metric
    - Numbers that are quoted
    - When is the result acceptable.  (This is related to the previous
      point, if I put "Median Delay = X ms", in an SLA is (X-1, X+0.1,
      X+100) as (10%, median, 50%) in or outside that limit?)

   That way, both vendor and customer know what is being measured,
   how it is measured and when the SLA isn't met.

Does anybody else think that this is useful?

And finally, during my vacation after Oslo, I read a paper on Pathchar and
started to wonder if we could measure the effective throughput of a link
using 1-way delays with different packetsizes.  Has anybody ever tried
this?

Henk

------------------------------------------------------------------------------
Henk Uijterwaal                    Email: [email-masked]
RIPE Network Coordination Centre     WWW: http://www.ripe.net/home/henk
Singel 258                         Phone: +31.20.535-4414,  Fax -4445
1016 AB Amsterdam                   Home: +31.20.4195305
The Netherlands                   Mobile: +31.6.55861746  
------------------------------------------------------------------------------

The Committee (...) was unable to reach a consensus that substantial merit was
lacking. Thus, the appeal was deemed meritorious.          (Orlando NABC #19).

---

From: Andy Scherrer <[email-masked]>
Date: Tue, 10 Aug 1999 08:47:31 -0500 (CDT)

> protocols and other binary stuff, this is NOT immediately obvious. 

Not immediately obvious, and also could get hairy. 
How to compare numbers depends on the numbers and on the distribution, not
to mention some measure of how the data were collected. 

But I agree that it makes a lot of sense to explore and document the realm
of cases that IPPM interests tend to encounter. 

> significant difference between the two?

Adequate comparison needs even more information, including sample sizes.
All statistical comparisons are deemed "significant" or not on the
basis of variation around the points you want to compare (usually the
center). While the 10% and 90% values give a range that is useful for
quantifying variation, the weighting of such a quantity depends on sample
size. And also on how independent your samples are. 

And then, defining what qualifies as "significant" is an entirely
different problem, since at the very least there's a distinction between
"detectible" with a certain reliability and "important" as far as, say, a
user or application would be concerned. 

> looking at them, to say that 2 distributions are compatible. 

To come up with an improvement on just looking at them (noting that humans
are really good at pattern recognition), you'd have to identify features
of the distributions that are important. Once again it's a question of
weighting: how important are deviations in the tails, vs. deviations in
the center, or some other region?

[SLA-related questions.

> Does anybody else think that this is useful?

FWIW, I do. 
Looks like a lot happened in Oslo; sorry I missed it!

Andy Scherrer
Statistician
Matrix Information and Directory Services, Inc.
1106 Clayton Lane, Ste. 501W, Austin, TX 78723
http://www.mids.org/
+1-512-451-7602

---

From: "Henk Uijterwaal (RIPE-NCC)" <[email-masked]>
Date: Wed, 11 Aug 1999 14:52:53 +0200 (CEST)

On Tue, 10 Aug 1999, Andy Scherrer wrote:
> I wrote:
> > Matt and Will wrote:

> Adequate comparison needs even more information, including sample sizes.

OK, but then let's at least define which numbers have to be provided (or
have to be constant) in order to be able to compare them.   Besides
comparing implementations, I can also see applications when comparing
today's results for the delays along a certain path against the results
from last week. 

> user or application would be concerned. 

I think we should define "significant" as "detectible", or the point when
there is a certain probability that the different results from two
measurements are caused by a real effect.  A user might want to set a
higher threshold depending on his application.

> > M'polis.  I think we really need some agreed method better than just >
> are really good at pattern recognition),

But the human brain quickly tends to get bored.  To compare a handful of
distributions against each other can easily be done by eye.  However, to
compare, say, today's delay patterns for all 40^2 combinations between 40
hosts against yesterday's results, is something that you want to automate. 
Even reducing the 1600 plots for this case to a few % that don't seem to
agree and have to be looked at by a human is already useful. 

> you'd have to identify features

Agreed.

Henk