Subject: [dtn-interest] DTN "test track"?

From: "Burleigh, Scott C (313B)" <[email-masked]>
Date: Wed, 9 Nov 2011 09:49:00 -0800

At the CHANTS workshop in September I was talking with the guys from IBR about the BP implementation performance benchmarking they had done, and we converged on an idea.

We agreed, I think, that everybody in the DTN community would benefit from being able to consult comprehensive benchmarking and interoperability testing results for all the various implementations of Bundle Protocol.  It would help potential users make informed decisions on which implementations would best fit their problem domains, improving everybody's chances of deployment success.

IBR's CHANTS paper was a great first step toward this, and the more detailed technical report behind it (at http://www.ibr.cs.tu-bs.de/papers/poettner-tr201108.pdf) goes much further.  But because the IBR implementation was one of those examined, it's possible for a reader to be uncertain -- however unfairly -- about the objectivity of the results; the same would obviously be true if a similar benchmarking study were performed at JPL's Protocol Technology Laboratory, and so on.  Anyway, objectivity aside, it's asking quite a lot to require any testing team to master all the technical intricacies of configuring and operating all available BP implementations, especially considering the imperfect maturity of the documentation for many of them.

What we think is needed is an independent DTN testing laboratory -- somewhat analogous to an automotive "test track" -- that would be a center for designing, conducting, and documenting thorough, standardized performance and interoperation benchmark tests on a common, standard Linux-based test platform.  Implementations written in C, C++, C#, Java, Python, whatever would be run head-to-head on the same systems, using the same convergence-layer stack through the same NICs on the same LAN.

The thinking here is that the test designs and platform specifications would be made freely available to all implementers, who would themselves develop and submit all the appropriate configuration files and runtime scripts to conduct the tests in configurations that best demonstrated the capabilities of their own implementations.  The lab staff would simply run the tests on the standard testbench and publish the results on a web site.  As new versions of implementations were developed, the tests could be re-run so that potential users could see how the performance of a given implementation has changed over time, in addition to how it compares to its contemporaries.

The site for such a laboratory would preferably be an organization that hasn't developed a BP implementation of their own and hasn't otherwise made a deep commitment to any single implementation, and probably it should be a university so that costs can be kept down by engaging students to do most of the work.  I think this doesn't have to entail a huge funding outlay, so it might be a nice opportunity for a resource-limited participant in the DTN community to take on a larger role.

What do you think?

Scott