Peering Policy Analysis

Peering or Settlement-free Interconnect (SFI), is a contentious subject as can be seen here and here. Having been involved in a few SFI negotiations and disputes I thought it might be instructive to  use an existing SFI policy as a vehicle for analysis.

First, what is SFI? Simply put it is the bilateral exchange of  two service provider (SP)’s customer routes without payment by either side (settlement-free). A more detailed explanation can be found here.

The technical details and various modes of the peering definition could go on for quite some time, but the question at the heart of the matter is: “will provider X interconnect with me on a settlement-free basis?” Network Service Providers want to connect to other networks on a settlement-free basis because it allows them to exchange traffic for free with them, without having to pay an upstream to carry their traffic. The upstream providers do not want to interconnect on a settlement-free basis because they lose revenue.

Geoff Huston has a very good statement of what settlement-free interconnection really means.

The bottom line is that a true peer relationship is based on the supposition that either party can terminate the interconnection relationship and that the other party does not consider such an action a competitively hostile act. If one party has a high reliance on the interconnection arrangement and the other does not, then the most stable business outcome is that this reliance is expressed in terms of a service contract with the other party, and a provider/client relationship is established.

Like taking margin in the retail industry, SFI will only be granted if the benefits of interconnection outweigh the cost. It really is that simple.

With that in mind, let us take a current SFI Policy and analyze the technical aspects. To ground the discussion in reality, I will use the Comcast SFI Policy as of September 2009. It is a good example of a well-written, modern SFI policy. Comcast policy text is in blue.

Applicant must operate a US-wide IP backbone whose links are primarily 10 Gbps or greater.

This is to ensure that the applicant’s network is similar to Comcast in size and has a similar cost basis. Traffic engineering and management are simplified due to similar bandwidth on the interconnecting backbones as traffic flows tend to be of similar size. There have been people who have interconnected at 10G with the backhaul restricted to STM-1/OC-3 links, causing saturation and a poor user experience.

Applicant must meet Comcast at a minimum of four mutually agreeable geographically diverse points in the US. Interconnection points must include at least one city on the US east coast, one in the central region, and one on the US west coast, and must currently be chosen from Comcast peering points in the following list of metropolitan areas: New York City/Newark NJ, Ashburn, Atlanta, Miami, Chicago, Denver, Dallas, Los Angeles, Palo Alto/San Jose, and Seattle.

This clause ensures that the applicants network is similar to Comcast in scope (and has a similar cost basis) and has the same redundancy, size, and diversity of connection that allows Comcast to easily integrate the interconnection and session management into their traffic engineering and operational procedures.

Applicant’s traffic to/from the Comcast network must be on-net only and must amount to at least 7 Gbps peak in the dominant direction. Interconnection bandwidth must be at least 10 Gbps at each interconnection point.

This requirement ensures that the network is at par with other SFI networks, making traffic engineering and operational management easier.  It should be subject to change regularly based on network evolution. The only thing I would change in the requirement is to substitute average for peak. With peak and 95th percentile a small number of samples dominate the calculation.  With average, that is not the case. Peak and 95th percentile are relatively easy to game, not so with average. Any metric that allows dominance of the outcome by a small set of samples is contraindicated in peering calculations, whereas in customer/provider relationships they are preferred by providers. The former situation is optimized for volume and the latter is optimized for rate.

A network (ASN) that is a customer of a Comcast network for any dedicated IP services may not simultaneously be a settlement-free network peer.

This requirement has caused more confusion than any other clause to my knowledge. Most people interpret this to mean “once a customer, always a customer, with no possibility of getting SFI in the future.”  This is quite incorrect. What it actually means is that if you are a customer, you cannot simultaneously interconnect for free for on-net routes. This comes up when customers want only to pay for “off-net” traffic and is implemented by the provider by setting up multiple interconnections.  Announce customer routes (the on-net traffic) on some interconnections and only announce  peer (or off-net) routes on others. If the provider offers this option there are many ways to game it. This requirement is self-defense and eliminates operational complexity.

Applicant must have a professionally managed 24×7 NOC and agree to repair or otherwise remedy any problems within a reasonable timeframe. Applicant must also agree to actively cooperate to resolve security incidents, denial of service attacks, and other operational problems.

Applicant must maintain responsive abuse contacts for reporting and dealing with UCE (Unsolicited Commercial Email), technical contact information for capacity planning and provisioning and administrative contacts for all legal notices.

This requirement ensures that there is a good point of contact that is reachable at any time, considerably simplifying technical and policy coordination between networks.

Applicant must agree to participate in joint capacity reviews at pre-set intervals and work towards timely augments as identified.

Traffic forecasting and pre-planning for capital expenditutures, metro and PoP upgrades is essential as they take time to get deployed in the field.

Applicant must maintain a traffic scale between its network and Comcast that enables a general balance of inbound versus outbound traffic. The network cost burden for carrying traffic between networks shall be similar to justify SFI.

This  is another very controversial requirement – the so-called ‘Ratio clause.’  The best way to look at it is via the Geoff Huston definition above, any other way of looking at this is doomed to failure. This requirement serves as another way to ensure that the interconnection applicant has a similar scale and scope network as Comcast, with a similar cost basis as measured by the cost of carriage of a bit/mile.

Applicant must abide by the following routing policy:
Applicant must use the same peering AS at each US interconnection point and must announce a consistent set of routes at each point, unless otherwise mutually agreed.

Consistent route announcements are useful to prevent gaming (see ratio requirement mentioned earlier), help in troubleshooting and traffic engineering.

No transit or third party routes are to be announced; all routes exchanged must be Applicant’s and Applicant’s customers’ routes.

If a network starts announcing transit or third party routes, those prefixes will interfere with normal routing and traffic engineering, potentially severely disrupting Internet connectivity for customers. Sending a large amount of transit routes can also potentially double or triple the number of paths in the routers, causing them to run out of resources and crash.

Applicant must filter route announcements from their customers by prefix.

Customer routes are preferred in most networks, and are announced to other SFI networks as the best path to reach that customer. If the customer makes an error such as leaking another providers upstream routes, it can cause significant disruption. For example, by making the customer look like it has the the best route to that upstream provider. The wrong information may be propagated to Comcast and their SFI networks, causing traffic to to be incorrectly routed.

Neither party shall abuse the SFI network peering relationship by engaging in activities such as, but not limited to: pointing a default route at the other or otherwise forwarding traffic for destinations not explicitly advertised, resetting next-hop, selling or giving next-hop to others.
Applicant should be willing to enter into an NDA before formal discussions begin.

The abuse requirement simply says do not try to steal service by pointing a default, or faking next-hops.  The NDA requirement is quite standard when entering into negotiations for something as sensitive as SFI.

Applicant should be advised that the SFI processes will start with a 90 day trial.  On successful completion of that trial, a formal interconnect agreement will be processed.  This agreement will renew annually, subject to the then current SFI Policy.  During the year if there is a violation of the policy, the agreement and interconnections may be terminated upon written notice to the contacts specified in the agreement.
A 90 Day trial to verify that the traffic, ratio and other technical conditions are satisfied is reasonable. It allows for sufficient time to verify the claims for volume and ratio, but is not so long that it starts looking like  a revenue generation mechanism.

Applicant shall not be permitted to offer or sell any IP transit services providing only AS7922.

This particular requirement prevents networks that meet the SFI requirements from selling cheap, direct access to the Comcast network to networks who otherwise do not meet Comcast SFI requirements.  This violates the equivalent cost basis argument for SFI.

Applicant must be financially stable.
Comcast requires that Applicants seeking SFI in the United States agree to provide reciprocal SFI arrangement with Comcast in the Applicant’s home market.

Excellent clauses. Comcast is US centric (for now). If they ever expand out to different geographies, there is a ready-made interconnection system in place.

This is a good, rigorous policy that sets out a fair, even-handed system of evaluation for SFI with Comcast. The requirements are clear, well articulated and make technical sense and that makes a sensible trade-off between of cost of interconnection and the value to the Comcast customer base.

Article was vastly improved thanks to editing and wordsmithing help from Ben Black.


21 Responses to Peering Policy Analysis

  1. Anonymous says:

    Comcast has an excellently written and very reasonable peering policy. Unfortunately they don’t actually follow it, instead opting for intentional congestion of their transit providers so that content customers will pay them for direct transit to get decent performance.

    If you look at their routing table, you’ll see their only direct peers are Cogent and a handful of medium sized European networks, everyone else is a transit provider. They appear to be “holding out” for only tier 1’s in the US, regardless of whether a perspective peer meets their published policy. But hey, at least the policy is nice.

  2. director of bump says:

    Peering policies are nice and fine. They provide a bar for those who seek to attain peering, but far too many try goofy tactics to meet the requirements.


    The “national 10G network” clause is bogus. The peering applicant is not submitting their PO’s from whatever L1 transport carrier they have to prove they have 10G. In reality, anyone can lie and compose their network of zero commit pseudowires to meet the goal.

    Ratios are controversial, mainly because of a certain “tier 1” network that used them in the past where it made no sense. For example, Comcast used to be single-homed behind AT&T (AS7018) for years. Now, Comcast being mainly eyeballs (inbound-heavy) would expect that if there was a ratio requirement, it would take this into account? However, the reality was AT&T enforced a strict ratio of around 1:1 on its peers and routinely threatened them with depeering. That seems a bit silly if your number one customer is a broadband network who pulls more than it pushes. Unless of course, the ratio requirement was just a method to force people into settlement-free contracts and wield the depeering saber at peers you felt were doing you “wrong”.

    In any event, the requirements are merely guidelines. When it comes to establishing peering with a large carrier, the decision is made well before anyone actually reads the peering request application. This is because the large networks already know who they want to peer with. Anything else is usually some sort of a deal made in exchange for other services (transport, etc), legally required (government entities) or some sort of a freebie (conference, infrastructure related services such as DNS roots).

    • director of pump says:

      Why try to enforce a ratio of 1:1 with your peers when your own traffic ratios aren’t 1:1? Because then you can claim “unfairness”, and make them honor MEDs and haul the traffic for you. Eyeball networks have all the power, they’re the ones with the locked in customer base tied to physical infrastructure.

      The ironic thing about Comcast intentionally congesting their upstreams so hosting customers will buy from them is that they were the victims of this same play when it was done to them by AT&T a few years ago. Now the victim has become the abuser. Funny how that cycle works.

      • vijaygill says:

        I wasn’t expecting this to be such a comcast hatefest. further hate on comcast will be not be approved.

      • director of bump says:

        What is also interesting is how AT&T is starting to get into the content game. I’m not just referring to their CDN product, but also the fact that they do have some amount of content providers purchasing transit for them.

        What if a peer of AT&T decides to turn the tables around after years of the ratio police?

        Should be an interesting next few years.

  3. vijaygill says:

    They can lie, but if it comes to light, it is an easy opt out clause. If you have good monitoring, you can figure out whats going on in the other network.

    The comcast push vs. pull might be surprising.
    The rest of the comment is spot on.

  4. […] Peering Policy Analysis « Vijay Gill's Blog – view page – cached #RSS 2.0 Vijay Gill's Blog » Peering Policy Analysis Comments Feed Vijay Gill's Blog Moving to WordPress Femtocells — From the page […]

  5. JZP says:

    Regarding average VS peak: average of 7G is close enough to 10 that you will be hampering microbursts. I would imagine that they just want to have some time to make use of the 10G links before having to throw more in for aggregates, get a historical sense of the AS-relationship’s growth rates, and still protect against bogus applicant with a decent enough threshold.

    As for the weenies: it takes two to tango^Wcongest, so I find it highly amusing that anyone would bother to take anonymous blather purporting “intentionality” seriously. Data or it didn’t happen.

    • Director of Transit says:

      JZP, Comcast is purchasing transit from a number of “tier 1” ISPs. If they’re having congestion issues, they can buy more transit. They don’t need to obtain approval from their peering partner, they just need to pony up some cash.

      In short, Comcast doesn’t care about its customers.

    • director of front says:

      Are you kidding? Last I looked it took one party to refuse to upgrade to congest a peer, or one party to refuse to buy more transit capacity when their pipes are full. Of course then you need someone to be the peering equivalent of the Iraq Information Minister, to declare that it is their transit providers’ fault for not agreeing to give them more transit for free… Wonder who that would be.

    • vijaygill says:

      JZP, very reasonable. I wasn’t saying keep the average at the same number as the peak (if your peak to average ratio normally is 2:1),so if the original was 7g peak, I’d move it down to 3.5 gig average.

  6. Chris says:

    @JZP: I read that clause as requiring minimum of 7Gbps total, not per interconnect. Since a minimum of four 10G interconnects are required, there shouldn’t be a congestion risk at the minimum qualification point assuming reasonable traffic distribution between the regional interconnects.

  7. davidu says:

    Geoff Huston’s definition of SFI is the only one that matters. Unfortunately, Comcast can choose to de-peer anyone they want regardless of policy since they own the eyeballs and eyeballs matter more to content networks than content networks matter to eyeball networks.

    In other words, Comcast having a crummy circuit to Google will hurt Google a lot more than it’ll hurt Comcast.

    • director of bump says:

      The impact varies depending on the type of the customer at an eyeball provider.

      Broadband users paying $20-50 a month can easily be told that “oh, its a Google problem, write your congressman. sorry”.

      A business customer paying big money on the other hand isn’t going to accept this as an acceptable answer. They want a solution, not excuses.

      Either way, the eyeball provider can tell any story to their customers, true or false. Most of them aren’t smart enough to know the difference.

    • ras says:

      Anyone can choose to de-peer (or not peer) anyone they want, regardless of policy. Peering policies are not contracts, nothing says they have to be truthful or that you have to follow them. The historical example of this is networks who set policies so strict that they themselves could not meet it, and then simply made “exceptions” for those they wished to peer with. Comcast’s policy is actually perfectly reasonable and fair, whether they follow it or not is a different question.

      But yes, Comcast absolutely controls the long term peering situation because they control the eyeballs. Quite simply, high speed eyeballs mean you own physical infrastructure somewhere, and in the US that physical infrastructure is often a monopoly or duopoly. Comcast customers simply have no real alternatives in most cases, so as long as they manage to steer clear of regulators they can pretty much do whatever they want to their customers without significantly impacting the bottom line. Content/hoster networks on the other hand have dozens of choices, all at roughly the same prices, and can switch providers in the blink of an eye if called upon to do so. In any kind of peering battle, Comcast will be the last to cave.

      As far as a Google/Comcast situation, why should either one be hurt? Last I checked both networks were buying significant amounts of transit, probably at pretty good rates too. I’m pretty certain the price of transit isn’t going to have any real impact to the operations of either company, so if for example they couldn’t get a mutually agreeable peering arrangement they could just both pay for some transit until they change their minds. And hey, Level 3 can certainly use the business :P.

  8. Joe Beets says:

    VJ, nice walk-through. Obviously being on both ends of these for many years has provided a certain clarity of understanding not often seen. Things like this need transparent explanation as much as they need policy transparency. Nice job.

  9. rf says:

    A well-written (read useful) SFI policy should be simple in it’s purpose and specific in it’s requirements. Ultimately it is an agreement to exchange goods of similar, perceived value. If one party fails to meet the agreed upon terms then the chest pounding and check writing can begin.

    Having sat on both sides of the “eyeballs” vs “content provider” table I have used all of the standard arguments to justify the network’s “value”. However, in the end I think the “eyeballs” have the upper hand. Content providers are being paid to house and deliver the widgets. They will incur distribution costs in delivering those widgets to the wholesale market.

  10. Vijay,

    Great post. You are dead on about the operational complexity of mixing customer transit and non-transit SFI connections to one entity. This mix is very challenging to maintain, particularly if you are operating a global network with a single AS. Customers tend to frown upon transoceanic tromboning and correcting usually requires one-offs that complicate routing policy.

    Jeff L.

  11. Aditya says:

    I clicked on the “peering disputes” search link in the article. 187000 results.

    Repeated search repeated with “peering disputes -comcast” gave 59000 results.

    Just an observation.

  12. […] peering agreement. (For detailed analysis of Comcast’s peering agreement check out this post from Vijay […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: