Subject: [alto] ALTO Interoperability test cases (draft)

From: "Vijay K. Gurbani" <[email-masked]>
Date: Wed, 22 Jun 2011 10:21:38 -0500

Folks: Here is the first draft of the ALTO interoperability test
we are scheduling in Quebec City.

Please go through this and suggest any fine-tuning we need to do
on the test suite.  Some of the test cases are more fleshed out
than others (complete with request and response), others need some
more work, and if anyone will like to pick some of the test
cases that need some more work and complete them, please let
the chairs know.

Similarly, if we need to add any additional test cases or you feel
that we need to remove some of the existing ones, please feel
free to indicate so on the list.

If there is anything egregiously wrong with the IP address blocks
or costs, please indicate so.

I suspect that we will go through a couple of iterations to fully
finish the test cases, but the this should give you a good idea
of what to interoperate.

---- BEGIN ALTO INTEROPERABILITY DOCUMENT (Version 0.1)

For the ALTO interoperability, we will consider the following topology
from the view point of a service provider:

PID            IP Address Block
-------------------------------
mypid1         10.0.0.0/8
mypid2         192.168.0.0/16
mypid3         192.168.10.0/24
peeringpid1    128.0.0.0/16
peeringpid2    130.0.0.0/16
transitpid1    132.0.0.0/16
transitpid2    135.0.0.0/16
defaultpid     0.0.0.0/0

And here is the cost map for the above topology:

Source PID         Destination PID            Cost (mode: numerical)
--------------------------------------------------------------------
mypid1             mypid1                               0
   "                mypid2                               0
   "                mypid3                               0
   "                peeringpid1                          0
   "                peeringpid2                          0
   "                transitpid1                          5
   "                transitpid2                         10
   "                defaultpid                           4
mypid2             mypid1                               0
   "                mypid2                               0
   "                mypid3                               0
   "                peeringpid1                          0
   "                peeringpid2                          0
   "                transitpid1                          7
   "                transitpid2                          8
   "                defaultpid                           4
mypid3             mypid1                               0
   "                mypid2                               0
   "                mypid3                               0
   "                peeringpid1                          0
   "                peeringpid2                          0
   "                transitpid1                          8
   "                transitpid2                          8
   "                defaultpid                           5.1

Test Cases

1) Retrieving the Information Resource Directory (IRD, Section 7.6)

    - Client retrieve IRD through a well known URI:
         http://alto.ietf.org/directory
    - Server returns the following directory:
      (TBD)

2) Longest prefix match (c.f. Section 4.2.1)

    - Client retrieves a PID for IP address 192.168.1.23
      Use the following request:

      POST /endpointprop/lookup HTTP/1.1
      Host: alto.ietf.org
      Content-Length: ...
      Accept: application/alto-endpointprop+json

      {
         "properties" : [ "pid" ],
         "endpoints" : [ "ipv4:192.168.1.23" ]
       }

    - Server returns mypid2 using the following response:

    HTTP/1.1 200 OK
    Content-Length: ...
    Content-Type: application/alto-endpointprop+json

    {
      "meta" : {},
      "data": {
        "ipv4:192.168.1.23"    : { "pid": "mypid2" }
      }
    }

3) Longest prefix match (c.f. Section 4.2.1)

    - Client retrieves a PID for IP address 192.168.10.23
      Use the following request:

      POST /endpointprop/lookup HTTP/1.1
      Host: alto.ietf.org
      Content-Length: ...
      Accept: application/alto-endpointprop+json

      {
         "properties" : [ "pid" ],
         "endpoints" : [ "ipv4:192.168.10.23" ]
       }

    - Server returns mypid3 using the following response:

    HTTP/1.1 200 OK
    Content-Length: ...
    Content-Type: application/alto-endpointprop+json

    {
      "meta" : {},
      "data": {
        "ipv4:192.168.10.23"    : { "pid": "mypid3" }
      }
    }

4) Returning default PID:

    - Client retrieves a PID for IP address 201.1.13.12
      Use the following request:

      POST /endpointprop/lookup HTTP/1.1
      Host: alto.ietf.org
      Content-Length: ...
      Accept: application/alto-endpointprop+json

      {
         "properties" : [ "pid" ],
         "endpoints" : [ "ipv4:201.1.13.12" ]
       }

    - Server returns defaultpid using the following response:

    HTTP/1.1 200 OK
    Content-Length: ...
    Content-Type: application/alto-endpointprop+json

    {
      "meta" : {},
      "data": {
        "ipv4:201.1.13.12"    : { "pid": "defaultpid" }
      }
    }

5) Retrieving a network map (Section 4)

    - Client uses information from IRD to retrieve a network map.
    - Server returns the following:
      (TBD)

6) Retrieving a cost map (Section 5)

    - Client uses information from IRD to retrieve a cost map.
    - Server returns the following:
     (TBD)

     NOTE: Make sure that the vtag in the response for the cost
     map is the same as the vtag in the response for Test case
     5.

7) Error: JSON parsing error (Section 7)

    - Client sends a malformed JSON body in the request (details TBD)
    - Server returns HTTP response code of 400 with ALTO error code
      of E_JSON_SYNTAX (c.f., Table 1).

8) Error: Required field missing (Section 7)

    - Client sends a request with a missing field (details TBD)
    - Server returns HTTP response code of 400 with ALTO error code
      of E_JSON_FIELD_MISSING (c.f., Table 1).

9) Error: Unexpected JSON value (Section 7)

    - Client sends a request with a JSON value of unexpected type
      (details TBD)
    - Server returns HTTP response code of 400 with ALTO error code
      of E_JSON_VALUE_TYPE (c.f., Table 1).

10) Error: Invalid JSON cost mode (Section 7)

    - Client sends a request with a cost mode of 'foo'
    - Server returns HTTP response code of 400 with ALTO error code
      of E_JSON_COST_MODE (c.f., Table 1).

11) Error: Invalid JSON cost type (Section 7)

    - Client sends a request with a cost type of 'foo'
    - Server returns HTTP response code of 400 with ALTO error code
      of E_JSON_COST_TYPE (c.f., Table 1).

12) Error: Invalid JSON property type (Section 7)

    - Client sends a request with an invalid property type (details TBD)
    - Server returns HTTP response code of 400 with ALTO error code
      of E_JSON_PROPERTY_TYPE (c.f., Table 1).

13) Error: multiple errors in request (Section 7)

     - Client sends a request with a missing required field and an
       invalid cost mode ('foo')
     - Server should detect at least one error and return that in a
       response.  The server chooses which error to report.

      Question: If the server is capable of detecting and reporting
      multiple errors, then what?  How does it let the client know?

14) Error: send a request with a bad URI

     - Client sends the following request:

       GET /foo/bar HTTP/1.1
       ...

     - Server responds with:
       (TBD)

15) Change in network map (Section 5.3)

     - Add a new network, changing the map.  For example, add the
       following subnet 201.1.2.0/24 to peeringpid2 giving the new
       network map entry for peeringpid2 as follows:

       peeringpid2    130.0.0.0/16, 201.1.2.0/24

     - Have a client retrieve the new network map.

     - The Version tag on the network map should be different than the
       one retrieved in Test Case 5.

16) Filtered network map (Section 7.7.3)
     TBD

17) Filtered cost map (Section 7.7.3.2, Section 7.7.2.2)
     TBD

18) Endpoint cost service (Section 7.7.5)
     TBD

19)  IPv6 test cases
      (TBD)

Miscellaneous notes

1) Section 4.2.1 says that "Each endpoint MUST map into exactly one
    PID."

    What happens if a client finds out that the above is not true?

---- END ALTO INTEROPERABILITY DOCUMENT (Version 0.1)

---

From: Ben Niven-Jenkins <[email-masked]>
Date: Thu, 23 Jun 2011 18:00:18 +0100

Vijay,

I took a quick look over the test cases and although the details are not complete for some of them they look a fairly reasonable set of tests.

Some comments:

1) I wonder whether it is worthwhile to define at least one of the PIDs as containing multiple IP subnets as in real life PIDs are likely to contain >1 subnet.

2) As part of the IPv6 test cases I think some PIDs should contain a mixture of both IPv4 & IPv6 PIDs while other PIDs contain purely IPv4 or IPv6.

3) I assume that the specific hostnames and URLs used in the test cases are purely used for example purposes and the expectation is that the client will be configured with the location of the IRD and then discover the other services based on the contents on the obtained IRD? As the IRD is the key resource for discovering ALTO services and therefore obtaining data from ALTO servers dynamic service discovery based on the IRD is pretty critical for interoperability between clients/servers.

>   What happens if a client finds out that the above is not true?

IMO:

If a client discovers that a given CIDR block is present in more than one PID then the client should be free to pick any of the PIDs that the block is a member of according to local configuration/policy.

If a server discovers that a given CIDR block is present in more than one PID then the server should be free to pick any of the PIDs the block is a member of according to local configuration/policy. However for a given resource the same decision should be made for a given version of the underlying map (i.e. subsequent requests generate the exact same result and URLs that are served by multiple implementation instances also generate the exact same result regardless of which implementation instance is returning the response).

Ben

On 22 Jun 2011, at 16:21, Vijay K. Gurbani wrote:

> _______________________________________________
> alto mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/alto

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Thu, 23 Jun 2011 21:19:47 -0500

> Vijay,

Ben: Thanks for your feedback.

> I took a quick look over the test cases and although the details are
> not complete for some of them they look a fairly reasonable set of
> tests.

Right, the intent is to get the test cases out so
folks can see the nature of the tests, while simultaneously
fine tuning them.

More inline.

> Some comments:
>
> 1) I wonder whether it is worthwhile to define at least one of the
> PIDs as containing multiple IP subnets as in real life PIDs are
> likely to contain>1 subnet.

Good idea.  I will put multiple subnets in a PID.

> 2) As part of the IPv6 test cases I think some PIDs should contain a
> mixture of both IPv4&  IPv6 PIDs while other PIDs contain purely IPv4
> or IPv6.

What triggers the choice of an ALTO server returning an IPv6
address?  An IPv6 address sent by the client and indicated as
the source address?  Returning IPv6 addresses to clients who
have indicated a source address of IPv4, obviously, does not
make sense.

I have put a placeholder for an IPv6 test, but decided to
concentrate on IPv4 first to get the flavour of the tests
out as a first cut.

> 3) I assume that the specific hostnames and URLs used in the test
> cases are purely used for example purposes and the expectation is
> that the client will be configured with the location of the IRD and
> then discover the other services based on the contents on the
> obtained IRD?

Yes, indeed.

More below.

> You also asked:
>> Miscellaneous notes
>>
>> 1) Section 4.2.1 says that "Each endpoint MUST map into exactly
>> one PID."
>>
> If a client discovers that a given CIDR block is present in more than
> one PID then the client should be free to pick any of the PIDs that
> the block is a member of according to local configuration/policy.
>
> If a server discovers that a given CIDR block is present in more than
> one PID then the server should be free to pick any of the PIDs the
> block is a member of according to local configuration/policy. However
> for a given resource the same decision should be made for a given
> version of the underlying map (i.e. subsequent requests generate the
> exact same result and URLs that are served by multiple implementation
> instances also generate the exact same result regardless of which
> implementation instance is returning the response).

This all sounds reasonable.  The question I have is how much of
this should be captured in the protocol document.  That document
simply states that there is a 1-1 relationship between an
endpoint and PID.  We can leave it at that and not worry about
the question I ask above.  However, if we go down the route of
providing heuristics to deal with 1-N mapping, then we should
just as well put it in the document itself.

Thoughts?

Thanks,

- vijay
-- 
Vijay K. Gurbani, Bell Laboratories, Alcatel-Lucent
1960 Lucent Lane, Rm. 9C-533, Naperville, Illinois 60566 (USA)
Email: vkg@{bell-labs.com,acm.org} / [email-masked]
Web:   http://ect.bell-labs.com/who/vkg/

---

From: Ben Niven-Jenkins <[email-masked]>
Date: Fri, 24 Jun 2011 16:46:13 +0100

Vijay,

> have indicated a source address of IPv4, obviously, does not

Good questions and operation of ALTO in mixed IPv4 & IPv6 environments is not something I have given much thought to.

For the Network Map (and filtered network map) service it doesn't seem unreasonable to say that the ALTO server returns all address types it has - the protocol certainly allows for that already.

For the Cost Map (and filtered cost map) service it seems less obvious. For example, if two PIDs both contain a mixture of IPv4 & IPv6 prefixes what does that mean? Maybe it doesn't matter because ALTO isn;t saying anything about routing reachability only that any combination of endpoint addresses from each PID are the same cost apart.

However it does also raise the question of how an ALTO server can indicate that the cost between two PIDs is infinite (i.e. connectivity between those PIDs is not possible) - or is that what a cost of 0 means (in which case that needs explicitly mentioning in the protocol draft because I couldn't find it in there).

> just as well put it in the document itself.

I think we should say something along the lines of what I suggest (it probably doesn't need to go in much, if any, more detail either) otherwise we risk interoperability issues where one client may interpret such a map as invalid while others do not.

Ben

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Mon, 27 Jun 2011 11:23:08 -0500

As individual ...

In reference to [1], where we discuss whether or not there
should be a 1-1 relationship between an IP address and
a PID, and if it is not, how should we handle it?

On 06/24/2011 10:46 AM, Ben Niven-Jenkins wrote:
> I think we should say something along the lines of what I suggest (it
> probably doesn't need to go in much, if any, more detail either)
> otherwise we risk interoperability issues where one client may
> interpret such a map as invalid while others do not.

Ben: The bigger question to me is: is it indeed valid to have
a 1:N mapping between an IP address and a PID?  If it is not,
then we leave this to the category of configuration errors and
not discuss how to handle it in the draft (the draft cannot
provide hedges against every MUST and MUST NOT).

So, the larger question is:

   Do folks see a 1:1 relationship between a PID and an IP
   address?  Or do folks envision one IP address belonging
   to more than on PID?

My personal answer is to leave it at a 1:1 relationship, essentially
as it is defined in the draft right now.

Thanks,

[1] See the bottom half of
   http://www.ietf.org/mail-archive/web/alto/current/msg01058.html

- vijay

---

From: Reinaldo Penno <[email-masked]>
Date: Mon, 27 Jun 2011 12:28:52 -0400

I second having 1:1 relationship. We explored having an IP address in more
than one PID during the protocol design and there were some difficult cases.

Having said that, a _private_ IPv4 can and would certainly appear in more
than one PID if these represent different VRFs. It goes back to some of the
gaps identified in penno-cdn

Thanks,

reinaldo

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Mon, 27 Jun 2011 11:49:00 -0500

As individual ...

> gaps identified in penno-cdn

So ... are you saying that we should mandate a 1:1 but have some
flexibility?  If so, this puts us in the grey area of having to explain
why a MUST may sometimes not quite be so authoritative.

As I said, in the end we appear to have some idea on how to deal with
an IP address appearing in multiple PIDs, however, the bigger question
remains on whether we should allow it at all.

I am trying to find out an equitable answer to that particular question
to determine if we change the MUST to SHOULD and explain what to do when
an IP address does have a 1:N relationship with PIDs.

Thanks,

- vijay

---

From: Reinaldo Penno <[email-masked]>
Date: Mon, 27 Jun 2011 13:02:19 -0400

> why a MUST may sometimes not quite be so authoritative.

I'm saying we should take it later when we allow PIDs to have some
structure, such as (do not take this literally): Asnumber.VRF.IP

---

From: Robert Varga <[email-masked]>
Date: Mon, 27 Jun 2011 20:19:45 +0200

> an IP address does have a 1:N relationship with PIDs.

I think we should keep the 1:1 relationship because of consistency
reasons in the cost maps. Given the network map:

PIDA = { 192.168.2.0/24; 10.0.0.0/8; }
PIDB = { 192.168.1.0/24; 10.0.0.0/8; }

and cost map:

PIDA->PIDB = 10
PIDB->PIDB = 0

I cannot see how the following queries can be kept consistent with each
other:
- endpoint cost from 10.0.0.0/8 to 192.168.1.0/24
- cost map
- network map
- endpoint property 'pid' of 10.0.0.0/8

e.g. combining the cost map and network map queries, the client should
calculate exactly the same cost the server returns in endpoint cost query.

Bye,
Robert

---

From: Ben Niven-Jenkins <[email-masked]>
Date: Mon, 27 Jun 2011 21:43:03 +0100

Vijay,

> provide hedges against every MUST and MUST NOT).

I have no issue with the current requirement that for a given IP address a client should be able to resolve one (and only one) PID via longest prefix matching and I'm not suggesting we change that.

However I do think that a sentence or two on what it means when such a scenario happens is useful for interoperability.

For example, if my client uses the network map service to obtain the complete network map and discovers that a particular IP address exists in two PIDs (i.e. a longest prefix match would return two PIDs), how is my client supposed to interpret the network map:

a) Does it mean the entire network map is invalid and I should throw it all away?
b) Does it mean those PIDs containing the duplicate IP address are invalid and I must disregard their complete contents but the rest of the map is useable?
c) Does it mean that those PIDs containing the duplicate IP address are valid (along with the rest of the network map) provided I disregard the duplicate IP addresses?
d) Does it mean I am allowed to make a local decision as to which PID is the correct PID for that IP address?

IMO (a) & (b) require a client to fully validate the network map looking for duplicate IP addresses before using any portion of it can be considered useable. (c) & (d) allow a client to discover & recover from an inconsistency at "run time".

Validating the entire network map before using it may not have a significant performance impact for an ALTO client in some P2P software where the number of PIDs and IP prefixes in the network map is likely to be small, however for some CDN use cases it could well be the case that there are thousands of PIDs and millions of individual prefixes in a single network map. A simple example would be if one wanted to represent in an ALTO network map a geo-location database containing the entire global routing table.

Ben

---

From: Ben Niven-Jenkins <[email-masked]>
Date: Mon, 27 Jun 2011 21:54:43 +0100

Reinaldo,

> gaps identified in penno-cdn

Probably something to pick up and discuss in more depth at a later date probably but IMO different VRFs are different networks with different topologies and should probably be best represented by independent ALTO network maps.

It's not just restricted to private IP addresses either if one considers the cases of extranet VPNs or of CDNs that use multiple topology maps.

Ben

---

From: Reinaldo Penno <[email-masked]>
Date: Mon, 27 Jun 2011 17:12:40 -0400

On 6/27/11 1:54 PM, "Ben Niven-Jenkins" <[email-masked]> wrote:

> Probably something to pick up and discuss in more depth at a later date
> probably but IMO different VRFs are different networks with different
> topologies and should probably be best represented by independent ALTO network
> maps.

They could, but you will loose in terms localization. Meaning different
clients (whether P2P or CDN) in overlapping networks can not use resources
of another.

> 
> It's not just restricted to private IP addresses either if one considers the
> cases of extranet VPNs or of CDNs that use multiple topology maps.

Agreed. I believe it is something we should take up. I touched on many of
these points in

http://potaroo.net/ietf/all-ids/draft-penno-alto-ipv4v6-00.txt

---

From: Richard Alimi <[email-masked]>
Date: Mon, 27 Jun 2011 23:33:15 -0700

Hi Ben,

> Validating the entire network map before using it may not have a significant performance impact for an ALTO client in some P2P software where the number of PIDs and IP prefixes in the network map i…

If I were implementing a client, I would choose (d) in the interest of
Postel's Law.  I would also send an email to the ALTO Server's
provider or implementer notifying them that it is not compliant and
that they should stop it :)

Saying something in the protocol is okay with me, but a concern is
getting into the habit of trying to find and suggest possible
behaviors in case any of the MUST / MUST NOT's in the document are
violated.

Rich

---

From: Ben Niven-Jenkins <[email-masked]>
Date: Tue, 28 Jun 2011 16:47:17 +0100

Rich,

> behaviors in case any of the MUST / MUST NOT's in the document are

IMO it's not about trying to find & suggest possible behaviours in case of all MUST/MUST NOTs being violated, it's about providing guidance on what constitutes an invalid ALTO response and whether a violation of the specification in a response means the entire response should be treated as invalid or not.

I'd suggest that the fact the original question was raised as part of documenting a set of interoperability test cases would suggest the current draft could benefit from guidance on what does/does not constitute and invalid response, however if the WG's preference is that clients should "just apply Postel's law" and define for themselves what constitutes an invalid response I'll accept that.

Ben

---

From: Richard Alimi <[email-masked]>
Date: Tue, 28 Jun 2011 09:09:43 -0700

Hi Ben,

> I'd suggest that the fact the original question was raised as part of documenting a set of interoperability test cases would suggest the current draft could benefit from guidance on what does/does…

For the purposes of the interop test, i think such responses should be
considered invalid, but I wouldn't fault an ALTO Client implementation
for not doing the validation. I'm just not convinced we should be
attempting to give such advice for real-world deployments in the draft
though. But, as I mentioned, adding something in the draft if the WG
wishes.

Thanks,
Rich