April 19 from 1:00 – 2:30 pm (CST)

SPEAKERS:
Steve Wallace, Indiana University [PPT]
Matt Zekauskas, Internet2: Agenda [PPT] | Authorization [PPT] | PMA Workshop [PPT]


Matt Zekauskas hosted the Measurement SIG at the 2004 Internet2 Spring Member Meeting in Arlington, Virginia. Matt gave a brief overview of the agenda: Steve Wallace would give a talk about some interesting IP address count results from Abilene, followed by Matt’s presentation on an authentication and authorization for measurement infrastructures, some “Lightpath” demo testing, and a performance debugging measurement architecture workshop (PMA).

Steve Wallace introduced himself. He has a live netflow feed from all the core routers in Abilene; they collect and analyze the data in various ways. An Abilene requirement is that the data be anonymized before it goes into any storage areas. Steve worked with Stanislav Shalunov to develop a “black box” to crunch the data without violating policy about anonymouzation – using data before all defining data has been stripped out.

The purpose of the research was to determine how many different machines are using Abilene. The initial estimate was about 5 million users (based on various analyses). They built a black box that tried to determine how many unique IP addresses were traversing Abilene.

Steve created a 4-Gigabit bit array – 4 million rows with 2 bits per row; one bit is set if it is a source address and the second bit is set if it is a destination address. If they saw the same IP number in both sources and destinations, they assumed it was a real IP address.

Steve described how they separated out multicast addresses (never a source, only a destination) – they had a large number of unique destinations that they assumed were multicast (because that would account for all the numbers). There were a number of unique sources, which they determined must be source “spoofing” on Abilene.

Then, he showed data of unique matched IP addresses over Abilene in a 10-day period – about 10 million addresses. Steve showed data that indicated that there were periods with upwards of 100 million unmatched IP addresses. John Streck asked why they killed the process 4-days early; Steve reported that, because there would be lots of multicast during the Member Meeting and he wanted to end the test before they got the huge spikes.

After the data was collected, it was anonymized and returned to the Abilene data storage. A couple years ago, they took the Abilene routing table and calculated the number of routing addresses and then took the global routing table and compared the numbers. Abilene was getting 500,000 (addresses advertised), 15% of the Internet; it suggested that the Abilene community is 15% of the global Internet. Abilene doesn’t get 15% of the traffic, which is probably because there is less traffic between R&E sites vs. commodity sites. Steve is interested in getting more ideas to add to the black box.


Matt Zekauskas talked about measurement infrastructures – he covered the threats and risks which include DoS and DDoS attacks as well as unintentionally swamping a local network (which looks like a DoS attack). When implementing a measurement infrastructure, Matt recommends having an audit trail.

Matt noted that he’s interested in “testing to the middle” – Jeff Boote asked if he was interested in identities (roles) for authentication. Matt said there were several ways to address risks, including rate- or time-limiting streams, limiting sources and/or destinations, or controlling the scheduling (as with BWCTL) as well as the Matt-preferred way – testing based on identity. Matt noted that he’d like to create a “model” policy that Internet2 members could agree upon – one would suggest various levels of authorization for different roles, but still incorporate local authority and allow locations to set their own levels of authorization based on security concerns.

John Moore noted that a group on campus might be responsible for measurement infrastructure; they supply a machine and they want some control over who gets to use it. This is just a sample of campus level concerns that need to be addressed in any “model” policy. Matt noted that levels need to differentiate between the people who are at/near the node that are known, other network administrators that are trusted, and then many more people at the campus level. Each level has different levels of trust and, thus, different levels of authorization for testing.

Jeff Boote felt there should be some identified roles where users could agree upon what each person would do; he noted that Abilene was interested in providing policy for people to use when testing into Abilene – if a campus sets one up, they would get to provide a set of policy. John Moore commented that it would be good to setup a “model” policy for campuses to use. Eric Boyd wanted to have a fairly liberal, low-incursion policy for everyone and have only a small “special friends” group that would be allowed to do more intrusive testing. Jeff said that “network engineer” is an easy-to-identify” role but there would be other roles that also would need more capabilities than the general public is allowed.

John Streck wanted to know how you had out tokens – how do you keep it fair? He also noted that, by doing the measurements, you change the environment and you have to be sure you don’t step on someones method of testing and changing the results of their test. Matt mentioned that we were thinking about having the tests node-specific; individual nodes would not let people stomp on each other.

John Streck mentioned that there are prime-times each day when people like to do things and non- prime-times, depending on the tester (i.e., staff vs. graduate students).

Jeff Boote stated that there’s always a “limited” service; you need a scheduler component for each of the steps (node, sub domain, campus domain, WAN, etc.) – you worry about your most limited resources first. John Moore said he could see there would be different classes of nodes – the one at the edge you’d like to have everyone be able to reach but the local nodes would be harder to authorize. Jeff thought the Middleware mantra “authenticate globally, authorize locally” would help keep the control over central nodes.

Matt mentioned several sites with open connections –one at JPL, one in DREN, and the NDT servers. Joe St Sauver mentioned that there are servers you can refuse service to – when you have 10,000 servers trying to test to the same location, someone has to be denied.

Matt talked about the traditional methods of testing which are getting more difficult because institutions are concerned about security and the number of folks who want to do testing is increasing. Then, he talked about current tools – Jeff Boote has developed a scheduler tool (BWCTL) that rate-, time-, and frequency-limits the tests. Also, LBL did some Iperf tests based on x.509 certs and the University of Michigan developed tools based on kerberos, ks509, AFS pts and globus. Shawn McKee mentioned that this work has been extended.

Matt described his “sample use” vision; he mentioned that there are test points in Abilene and want to deploy test points on campuses. It schedules local tests to nearest test point and any other tests were the data is stale, the results are used to “divide and conquer,” pointing to a suspect network segment and contact point. The contact is then given the results so they can do some research and, ideally, solve the problem.

Other possible uses for the piPEs work is for the NOC, Abilene engineers to schedule longer tests and for connecters (testing buddies) to run tests to the core. Unauthenticated folks would be able to do some limited tests and have access to the data collected from regularly scheduled tests. John Streck asked “Why? And what if someone is doing something unusual?” John called it a “Catch-22” – if you haven’t done anything, you aren’t known and, therefore, can’t be trusted; and if you can’t be trusted, you don’t get to try anything so you never get known so that you can be trusted. Matt mentioned that he wanted to have a red switch to STOP people who aren’t “trusted” if they begin to do things that are untrustworthy.

Jeff Boote mentioned that he wanted to have fairly liberal limits of authorization requirements for testing – first, it lowers overhead, and second, it allows a history to be obtained. Once people have a history, you can give them access to more tests. John Streck mentioned that there are new people every year on a campus who might need to push the edges; when you get used to today’s grad students, they graduate and you get new ones. Eric Boyd noted that, within the NC network, you would trust John Moore for some things you wouldn’t trust Matt Zekauskas for, and others you’d trust Matt for as well, and still others you’d trust more generic students to do. If you keep the limits within what you can take – say one Iperf test/week – there is no concern about who they are. John Streck noted that, even if you only give someone a small amount of ability they can still blast out the network. Eric Boyd commented that the only thing that has changed is that, with the piPEs system, there’s an audit trail. Chas DiFatta commented that that is very important – if people know there is an audit trail it might deter them. If they are a student, they have provided personal and financial information to the University so they are vulnerable. He also noted that, if people know you have a regular schedule for tests, they can sneak in and tweak their numbers so they look better.

Matt commented on the 5 questions that need to be addressed:

1. Authentication needs – Where to? Each node? Virtual organizations? Devices? What degree of trust – do you accept someone who has authenticated to their own university? Or do you need them to re-authenticate in your organization? When do you do it?
2. Type of authentication – Global? Federated? Ad-hoc? Matt prefers the Federated version (network engineer, Abilene, Internet2)
3. Authorization – static factors (based on role or remote domain, i.e., piPEs user) and dynamic factors (delegation, preemption – red switch locally, and time frame – 24 hours)
4. Privacy – for active tests, not worried about it but encryption on wire -- yes for passive, not worried about anonymity of traffic or users and not sure about international users! There are interesting governmental policies one might have to deal with – Matt stated that he’ll know more when he’s worked with GEANT.
5. Auditing and Accounting – tie actions to users, correlate use to end-users for measurement points (not user data) and allow owners of measurement points and framework to view audit logs. Allocate based on bandwidth, time, and frequeny.

Chas DiFatta noted that if you keep the active material around for a while, you could be subject to a subpeona. Matt commented that it would be useful for comparing if you are having problems with running tests – “hey, I can never do authorization when these guys are doing throughput tests!”

Matt talked about policy decision points for authorization – a policy might be based on local resource availability – i.e., “I have too much scheduled to let anyone else in” vs. “you’re a network engineer so I’d better make time for you to run tests”.

When discussing policy enforcement points, Matt recommended many – most likely one per node. Jeff Boote talked about the meta-authorization daemon above them both. There are two dimensions – there are two groups and you can sub-limit each group. There are limits based on each program and limits for each host. That’s how he would think about limiting a bottleneck link for Abilene.

Matt gave an overview of the Performance Measurement Architecture (PMA) Workshop. He noted that Internet2’s E2Epi has been working with a lot of different groups (including GEANT, UCL, NLANR, CERN Caltech, several of the ITECs, the VLBI community and the MonALISA project) and wanted to see if there was an overall plan of architecture and see what’s common/missing. Using NSF-grant funds, Matt hosted a workshop in December 2003 (at SDSC) to bring together a bunch of NSF-funded projects. He gave an overview of the exiting architectures (grid measurement, piPEs, TF-NGN measurement group, and others). Middleware is also working on E2E – authorize to campus and use a resource on another campus because the two institutions “trust” each other. If there is a failure – what went wrong? You want to get a handle on problem without giving up the privacy information if possible. Need a diagnostic backplane with a common event record. It is very applications-focused – looking for failures, but not necessarily performance failures.

Matt also provided a list of current projects (see slides). He provided his interpretation of the results of the workshop (see slides).


ATTENDEES
Chris Tracy, Shawn McKee, Dan Eklund, Stev Senger, Mike Gill, Sami Chatterjee, Ralf Kleineisel, Stephan Kraft, Dave Pokorney, Cindi Dunn, Joe St Sauver, Cindy Shuman, John Moore, John Streck, Sub Ramaurisnan, Lawrence Kirchmeier, Warren Matthews, Javier Muňoz, Shumon Huque, Justin Church, Tommy Jacobson, Susan Evett, George Mallick, Boyd Knosp, Stanislav Shalunov, Rich Carlson, Paul Love, B. Joseffson, Matt Mathis, Cheryl Munn-Fremon, Chas DiFatta, Eric Boyd, Jeff Boote, and Jonathan Tyman.