April 19 from 2:30 - 4:00 pm (EDT)


Russ Hobby hosted the 4th E2E Tools Tutorial; he introduced the speakers:

SPEAKERS
John Moore, NC-ITEC [HTML] [PPT]
Ralf Kleineisel, DFN [PDF]
Jeff Boote, Internet2 [HTML]


John Moore presented information on two of the network test tools used in the NC-ITEC (SmartBits and AX/4000 Broadband Tests System). Both of the tools, from Spirent Communications (one of their lab partners), focus on end-to-end (E2E) testing. Common characteristics include a GUI mainframe version (that sits in a chassis), and portable versions that are deployed to various locations on the network. There is a variety of interface types. In the remote testing scenario, NC-ITEC has been using them because of the easy availability of GPS and CDMA time syncs. They’ve been working closely with Spirent to develop the types of tools that are needed for the ITEC.

John provided information on the differences between the two tools. Spirent is a conglomeration of two different companies that have come together, so each previous company brought a product to the table. With SmartBits, which fits the mainframe slot; the type of testing one can do focuses on flow-based analysis, related to the methodology of the RFC-style test. Primarily, the tests happen in a sequence; with a throughput test, John would run a test with 100% to see if there is a loss; if there is, the algorithm gradually reduces the percentage until it finds a percentage at which there is zero loss.

AX/4000 is very different; it has lower port density and provides a richer set of analysis tools – a generator and analysis tool provides much more detail. At NC-ITEC, they turn on the bit stream and then, whatever the particular analysis is (latency, packet loss, fragmentation, or other similar issues), they identify what they were looking for and AX/4000 would focus in on that. John showed images of the chassis and described what the slots were focused on. There is a separate GPS module at the top. He noted that there are several versions of the portables with two slots to work with.

John also provided an indepth description of a Lightpath demo conducted between NC-State, Chicago, New York, and CERN. He had both IPv4 and IPv6 traffic coming between Chicago and New York so he had to install and route into the Juniper Routing Table to point to a specific path (one hop). The first procedure was to ensure the traffic was making it into the proper queues (especially if there was under-represented traffic). (The “Best Effort” was having the biggest problem.)

He provided statistics for the average transfer delay, average packet size, packet loss, etc. data for the demo. In the packet definition window, he defined a traffic stream definition that had four different modes of distribution for traffic. In general, a couple of different traffic matrices have been developed to show the spread of packet sizes in a normal Internet environment. In the demo, they created a different distribution of traffic for over 900 Mbps so they could support the higher MTUs. They can also capture traffic in the analyzer.

Using the AX/4000 platform, they did an IGP resiliency test to measure the recovery time of an E2E traffic stream after a link outage. The test-bed was at the NC-ITEC lab (with multiple contrived paths).

They did a third test for MPLS Service to examine performance of their inter-AS MPLS path over Abilene. The test bed was on Abilene, between NC-ITEC and the Indiana University NOC. They used the SmartBits platform for this test. John noted that both of the tester tools do a good job of arcing to determine the best routing path.

John showed an example of the output – he noted that, at 7 MB, they were losing some packets; he had seen some scenarios where they were losing packets on the backbone when the size gets too large. In an area of the table with zero loss of frames, there are a number of different rows that show a little bit of reordering to avoid the loss. The next scenario is to compare a similar IP path to see if there is similar loss. In one of the tests, they added several other paths to see if there were similar problems – several streams were not getting through until there was some rearrangements of the routers.


Ralf Kleineisel talked about the IPPM Measurement System developed at the University of Erlanger. They developed and created an accounting system and a performance measurement system for the German research network. Ralf showed the German networking map and the regularly scheduled testing that is going on therein. (See slides which appear to be very comprehensive.) One reason for the testing was to determine the correlation between active and passive measurement.

They began one-way delay measurement in 2000 and have improved upon the system each year since. Their system includes a time-stamp and they send packets in groups so they can distinguish between shorter and longer periods of delay. A primary focus for DFN is to give their customers more information so they can make the necessary changes to allow better use of the network capabilities. Russ Hobby asked if they were also monitoring packet loss; Ralf said they were.


Jeff Boote provided an overview of the Bandwidth Control (BWCTL) tool and discussed the scheduling details. A participant asked if there was a method to determine the scheduling by an event; Jeff responded that, although there is not a specific opportunity to identify an event at which you would like to start a test, it would be possible to program for it. He reported that, although he is only scheduling one test at a time (so that there is no overlap), it would be possible to allow more than one test to run, concurrently, so that no test overlapped with another of its own type.

When it comes to resource allocation, if the user is only interested in Iperf testing, then larger spheres of control can be offered. A user might come up with a different set of resources at each level that needs to be limited during the authorization phase. Jeff noted that, at each level of control, BWCTL uses a hierarchical model for doling out the resources; the user would allocate ‘x’ to each of several groups; within those groups, the user could provide different levels of resources to specific individuals/groups as necessary.

The client contacts the main bwctl daemon (which acts as the main resource scheduler) and that daemon checks with the server to a) determine time offsets between the two locations, b) check for authorization (DOS protection) and c), when the time has been verified, each of the servers (at the two endpoints) starts the testing.

Jeff showed a demonstration of the test; the server responded that it was busy – he commented that the BWCTL default is to respond ‘busy’ if the test cannot start to run within 10 seconds of your request. He added that there is a command line modification which allows the user to set a wider window for start time; he demonstrated this to see if the test would run. Within the 1-minute period he specified, the test began to run. Jeff informed the group that, because of the full mesh of tests already scheduled on Abilene, users will get the ‘busy’ signal nearly 50% of the time.

He pointed out that there are Abilene test points at this time – a few AES keys have been handed out so beta tests could be done. The policy is in flux and Jeff recommended that people apply ‘early’ because they are not sure how limited the resources will be. He ran through the basic steps for using BWCTL and noted that we are currently developing documentation for this process (see http://e2epi.internet2.edu/pipes/pmp/bwctl-use.html.)

Jeff commented on the difficulties encountered – for example, with UDP, Iperf doesn’t always send at the requested rate. For TCP, when you are crossing a large pipe to a small pipe, you fill up your buffer very fast. With Iperf, you find out nothing and the server kills the program before the test can complete.

Jeff talked about future steps – he would like to develop a server-less client side for end-hosts, an integrated tester, a 3-part test (where the client is not one of the end-points), and to stop using Iperf (after developing a similar tool without the glitches).

He provided information about the beta version and the mailing lists for BWCTL. John Moore asked him about what he does with the data collected off Abilene. Jeff responded that the data was being used by the Abilene Observatory to show the status of the network. The current system is not collecting data off the edges; he’d like to develop a Top 10 BEST list to highlight the best connections.


ATTENDEES
Tommy Jacobson, Justin Church, Russ Hobby, Terri Saarinen, Sub Ramaurisnan, Dale Smith, Yasuichi Kitamura, Roman Jumenez, Dave Pokorney, Stephan Kraft, Rencke Schroeder, Trevor Corbett, James Deaton, Javier Muňoz, Dan Eklund, Chas DiFatta, W. Marti, G. Pearson, B. Josefsson, Eric Boyd, Dan McCarrier, Stephen Barrington, Shumon Huque, Stanislav Shalunov, John Streck, Deke Kassabian, Joe Askins, George Brett, Jerry Sobieski, Rich Carlson, Cheryl Munn-Fremon, and Susan Evett.