© Cybertelecom ::
- Balanced Value :: Settlement Free Peering
- Slightly Imbalanced Value
- Imbalanced Value :: Interconnection for a Fee
- Value in Dispute :: Peering Disputes
- Other Interconnection Information
- Internet eXchange Points
When two networks interconnect, both networks benefit. The reachability to and potential communication opportunities with end-users, services, applications, and content has grown by merit of the interconnection, and thus the value of both networks to their end-users has grown. This is known as “network effect.” The benefit of interconnection to the respective networks may be balanced or imbalanced.
Whether one network "should" pay another network for interconnection, and who pays whom can be difficult to resolve. Geoff Huston wrote
given two ISPs that interconnect, the decision as to which party should assume the upstream provider role and which party should assume the downstream customer role is not always immediately obvious to either party, or even to an outside observer. …In many ways, the outcome of these discussions can be likened to two animals meeting in the jungle at night. Each animal sees only the eyes of the other, and from this limited input, the two animals must determine which animal should attempt to eat the other!
[Houston INET'99 Sec. 4.]
In a functioning market, who pays whom is determined by who has to pay whom (not by subjective determinations of who should pay whom, or what is fair). A consumer pays a baker for bread because the consumer has to pay the baker in order to induce the baker to sell the bread; the baker has no other incentive to give the bread to the consumer and the consumer needs to eat. Where the value of interconnection is relatively balanced, interconnection advances both providers' business plans relatively equally. Where the value of interconnection is imbalanced, one provider needs the other more than the latter needs the former. One provider cannot pursue its business plan without purchasing services from the other, while the other may be able to get along just fine without the later. Who has paid whom has been determined fundamentally by who has had to pay whom in order to receive services and pursue their business plan.
Arrangement provides access to partner's Peering Transit Network
Y Y Transit Customers
Y Y Peers
N Y Transit Vendors
Internet interconnection arrangements involves the following characteristics: [Charter Order para. 98] [AT&T Comments Docket 10-90 at 12 (“This peering and transit regime encompasses interconnection agreements among all of the Internet’s constituent IP networks, not just those between “backbone” networks”)] [PCH Survey 2016 at 1 ("Each of the interconnecting links takes one of two forms: transit or peering")]
- Financial terms (settlement-free or for a fee)
- Traffic exchanged
A network announces On-Net routes when it announces to its interconnection partner that it is a route to itself and to its customers. A network announces Off-Net routes when it announces that it is a route to its other peering partners and to its transit vendors.
Interconnection arrangements involves backbone service providers, internet access service providers, content delivery networks, and end users (edge providers, individual users). Interconnection between networks commonly takes place at an Internet eXchange Points. [BEREC p. 10 Dec. 6 2012] [OECD p. 7 2014 (providing players in the "content distribution value chain")] [Verizon v. FCC, No. 11-1355, Slip at 5 (D.C. Cir. 2014) ("Four major participants in the Internet marketplace are relevant to the issues before us: backbone networks, broadband providers, edge providers, and end users").]
Balanced Value :: Settlement-Free Peering
Financial Terms: Where the perceived value of interconnection between networks is balanced, network providers may enter into settlement-free peering arrangements. Both providers perceive that the benefit gained from interconnection exceeds potential revenue gained from attempting to charge for interconnection, or that they would be unable to impose an interconnection fee on their interconnection partner. [ENISA Report at 93 (“In practice ASes will peer if they both believe that the benefit to themselves outweighs any perceived (or, indeed, actual) inequality in costs or benefits.”)] [NRIC Report Sec. 2.6 ("ISPs tend to peer with ISPs of a similar scale (as this often allows for a perceived rough equality of value).")] [Brock, Price Structure Issues in Interconnection Fees at 3 ("We should expect to see 'sender keep all' arrangements develop in a competitive communications market if either of two conditions are met: Traffic flows are very roughly balanced among the companies so that each sees a clear benefit for its customers in both sending and receiving traffic from other companies; OR The cost to a company of terminating traffic is low in relationship to the transaction costs of measuring and charging for traffic so that even with unbalanced traffic companies find the simple 'sender keep all' approach superior to efforts to develop appropriate cost-based terminating charges.")]. [AT&T, What Does it Mean to be a “Peer”? (“Peering works between networks that are of similar size and scope and have similar traffic characteristics. Such networks are able to assess whether in exchanging traffic on behalf of their users, they obtain equivalent value to what they provide”)] [ENISA Report at 75 (“This is a generally symmetrical and mutually beneficial arrangement.”)].
Both providers cover their own costs and neither charges the other based on traffic exchanged. Both providers benefit from the interconnection and the value of their service grows. [BEREC p. 22 Dec. 6, 2012 (noting costs associated with "settlement free" peering)]
Settlement-free peering avoids transaction costs, improves QoS and resiliency, and focuses on selling Internet access to customers as the primary source of revenue. [Brock, Economics of Interconnection at iv ("'Sender keeps all' is an administratively simple mutual compensation scheme with zero prices for terminating service.")] [Norton Internet Service Providers and Peering] [BEREC Draft Report 21] [Moyle-Croft] [Sutrisno] [Norton Remote Peering] Settlement-free peering is also attractive because it creates an incentive for innovation and efficiency; a provider that brings traffic to the exchange point efficiently gets to keep the savings. [Brock, Interconnection and Mutual Compensation with Partial Competition at 13 (Under sender keep all, each company has an incentive to increase the efficiency of its operations in order to reduce its costs and to maximize its outgoing traffic relative to its incoming traffic because outgoing traffic is the most profitable.).]
Traffic Exchanged: Peering is an interconnection arrangement where two parties (generally networks) exchange traffic between (1) themselves and (2) their transit customers (a.k.a. On-Net traffic), but do not provide transit through their networks to their other peering partners or transit vendors. [Golding] [BEREC Draft p. 19 2012] [BEREC p. 21 Dec. 6, 2012] [Norton ISPs and Peering ("Internet Peering is the business relationship whereby companies reciprocally provide access to each others’ customers.")] [AT&T Slide 2] [NRIC Sec. 2.5 ("Peering is an agreement between ISPs to carry traffic for each other and for their respective customers. Peering does not include the obligation to carry traffic to third parties. Peering is usually a bilateral business and technical arrangement, where two providers agree to accept traffic from one another, and from one another’s customers (and thus from their customers’ customers)."] [ISOC p. 3 ("Peering agreements are where two network providers agree to exchange traffic between each other's customers (but not use each other's transit connections). Peering is the act of exchanging traffic with a peer.")] [AT&T Mair Declaration para 11 ("Large ISPs often interconnect with AT&T through peering. Peering is a private commercial arrangement under which two “peer” ISPs connect and exchange traffic. Each peer provides the other with access only to its own customers – not to the entire Internet.") MCI WCOM / Sprint Merger DOJ Complaint ¶ 24; AT&T / Bell South Merger Order 2007, para 123. [SBC / AT&T Merger Order ¶ 110 2005] [WCOM MCI Merger Order para 145][TWC Peering Policy ("“Peering” is the interaction between two distinct ASN’s using BGP to exchange TCP/IP routing information between service providers operating Internet Networks")] [First Sec. 706 Report, 1999, ¶ 105 & n 240 ("an arrangement in which two Internet backbone providers exchange traffic that originates from an end user connected to one of the providers and terminates with an end user connected to the other provider.")] [WCOM / MCI, 1998, ¶ 145 ("In a peering arrangement, two IBPs agree to exchange traffic that originates from an end user connected to one IBP and terminates with an end user connected to another IBP.")]
Settlement-free peering has also been referred to as
- a barter exchange where both parties are involved in a mutual exchange of value (each other's networks, infrastructure, and customers)
- [AT&T OI NPRM Reply at 95 n 343 (“[P]eering is equivalent to a mutually-agreed-upon barter transaction. Although some mischaracterize peering as “free” interconnection, in reality the two networks agree to exchange traffic without an associated financial transaction because the resources they use on each other’s networks are roughly equal.”)] [AT&T July 30, 2014 Ex Parte, Attach., Dkt. 14-18, at 3 (explaining that peering is a “commercially negotiated barter transaction” where “parties’ perceived value of arrangement is equal”)] [Opposition of AT&T to Petition to Deny at 42-44, MB 14-90 (referring to SFP as the exchange of in-kind traffic in a rough balance)] [AT&T Mair Decl. ¶ 6, 12 ("Although peering arrangements are “settlement-free,” in the sense that the two parties typically do not exchange monetary payment, peering is not “free.” These arrangements are barter transactions under which each peer network agrees to exchange roughly equal amounts of traffic")] [Yoo (“peering is better understood as a form of barter ... when value is no longer equal on both sides of the transaction, barter no longer makes sense”)] [Norton, Evolution of Peering Ecosystem (“Internet Peering is the business relationship whereby companies reciprocally provide access to each others’ customers.”)] [Singer (“While one well-established model of interconnection involves settlement-free peering arrangements with no money exchanged, that model generally has been confined to those situations where the parties recognize that there is an exchange of roughly equal value between them in other forms. For example, where two networks exchange roughly equal amounts of traffic and are otherwise similarly situated, they have been more likely to interconnect without bothering to exchange money.”)] [ENISA p. 100 (discussing “the value of traffic”)] [Zachem Nov. 3, 2014 at 3 (“Comcast looks to its settlement-free peers to provide mutual value reflected in the form of investment in network infrastructure.”)]
- Bill and Keep [Noam Sec. 5.1.1. ("Two networks might agree to a zero-charge where traffic and costs between the entrant and the incumbent are balanced, and where it therefore would be administratively easier to impose no charge. Since costs are not passed on, each carrier has an economic incentive to increase the efficiency of its own network.")] [Degraba (notes that telephone networks followed a similar model, with local exchange carriers, which were like-networks, interconnecting on a bill-and-keep basis)] [ENISA p. 75 (“In any form of peering arrangement the parties exchange traffic destined only for their own users and customers, including transit customers. The Traffic is paid for by the two parties’ users and customers – it is part of the ‘Internet Access’ that they are buying.”]
- Sender Keeps All
Networks have different approaches to settlement free peering. These approaches include
- Open peering (will peer with anyone);
- Permissive peering (will peer with those who meet criteria);
- Restrictive peering (will peer with only a select few); and
- Closed peering (generally will not peer)
[Norton, Open Peering Policy]
See also History of Settlement Free Peering.
(1) Financial Terms: Settlement Free
(2)(a) Route Announced: On-Net
(2)(b) Long-Haul Transport: Yes (shared)
Two backbone service providers may "peer." They announce to each other that they are routes to their own networks and to their customers (On-Net traffic). Because backbones are large, these peering arrangements involve the exchange of a large amount of routes and traffic.
Traditionally, backbone providers interconnect at multiple IXPs throughout the nation. The backbone provider of the originating traffic will deliver the traffic to the partner backbone at the closest point of interconnection to the source of traffic (a.k.a. "Hot Potato Routing). Thus the terminating network will provide long haul transport to the destination. Since traffic is relatively balanced and multidirectional, the partners will share responsibilities for the long haul transport.
Donut Peering SFP:
(1) Financial Terms: Settlement Free
(2)(a) Route Announced: On-Net
(2)(b) Long-Haul Transport: No
Networks that are not backbones may peer. Two local networks can directly peer, exchanging traffic between themselves, avoiding sending traffic to a far off interconnection point to be exchanged (known as "tromboning") and avoiding buying transit from a backbone provider for that traffic. This is known a “donut-peering” because the networks are interconnecting around the core backbone providers. This peering arrangement is limited and controlled. The routes exchanged are only those between the two partners and generally neither party is agreeing to long haul transport for the other's traffic. [Levy Slide 12 OECD 2011 (noting that it can be quite expensive for a network to build its own peered network)] [Marcus Slide 17 OECD 2011 ("Any self-supply is presumably cheaper than buying transit")] [Krogfoss Slide 5 OECD 2011 ("50 global traceroute servers showed 60% of routes to large ASPs avoided T1 ISPs" (backbones)]] [Daniel Golding, AOL, Peering Evolution, Presentation at NANOG 27, Slide 14 (2003), ("The current "hot idea" in peering is the donut - networks with open peering going 'around the core' to peer.")] [Houston INET99 Sec. Sec. 5.2 ("Local ISPs participate at exchanges and announce local routes at the exchange on an SKA basis of interconnection with peer ISPs. Such ISPs are strongly motivated to prefer to use all routes presented at the exchange within such peering sessions, as the ISP is not charged any transit cost for the traffic under an SKA settlement structure.")] [BEREC Report 2012 at 24 (discussing regional or donut peering)] [Kende Assessment of the Impact of IXPs (discussing the benefits of local peering)].
The need for donut-peering became acute with the historic evolution of the Internet. The United States was the inventor and first mover of the Internet, which meant that Internet infrastructure developed earlier in the United States. It was not uncommon that Internet traffic on one side of the street in, for example, Europe, destined to the other side of the street, would cross the Atlantic and be exchanged in Northern Virginia. This led to the ICAIS debate over the cost of international transport. The solution is regional or donut peering. Michael Kende in an excellent Internet Society paper reviews the benefits of regional peering: traffic can be kept in-country and out of the reach of surveillance efforts; the building of local exchanges and promote local infrastructure and drive develop local IT businesses, and regional peering bypasses transit expenses.[Kende Assessment of the Impact of IXPs (discussing the benefits of local peering)].
(1) Financial Terms: Settlement Free
(2)(a) Route Announced: On-Net
(2)(b) Long-Haul Transport: Yes (CDN)
Content delivery networks can agree to a peering interconnection relationship with an access network. The two networks only exchange routes between each other. The content delivery network agrees to be responsible for the long haul transport, carrying the traffic as close to the requesting eyeballs of the access network as possible, exchanging the traffic at the gateway of the access network. This is known as "cold potato routing." At the interconnection between the CDN and the access network, the interconnecting partners only exchange on-net traffic.
Traditionally, this arrangement is a win-win for all partners. The content source gains the efficient delivery of the CDN, which reduces transit costs while improving quality of service. The access network likewise avoids transit costs by receiving traffic directly from the CDN instead of over transit links, and the access network customers receive an improved quality of service of the content.
See also Paid Peering (arrangement between CDNs and large access networks)
Slightly Imbalanced Value
In between balanced and imbalanced relationships are arrangements where the value exchanged is somewhat out of balanced. In these situations, whether an arrangement is settlement-free or for a fee may be determined by whether the transaction costs for charging settlement fees exceeds the imbalance of value. Transaction costs include negotiating, administering, measuring and accounting for the transaction. In situations where transaction costs exceed an imbalance of value, both networks would gain more by interconnecting on a settlement-free basis. [Comments of AT&T, Dkt 10-90, 2012 at 13 (“To avoid administrative overhead, parties to these bilateral peering agreements typically forgo the mutual exchange of compensation and peer on a settlement-free basis.”)] [Verizon Email Nov. 6, 2015 (“For example, where two networks exchange roughly equal amounts of traffic and are otherwise similarly situated, they have been more likely to interconnect without bothering to exchange money.”)] [Brock, Economics of Interconnection at iv ("'Sender keeps all' is an administratively simple mutual compensation scheme with zero prices for terminating service.")].
Internet networks have repeatedly stated that not insignificant imbalances of traffic do not disturb their willingness to enter into settlement-free peering arrangements (the network effect benefits being sufficient to drive interconnection). [Norton, Internet Service Providers and Peering (“peering is free until traffic asymmetry reaches a certain ratio (4:1 is common). At this point, the net source of traffic will pay the net sink of traffic a fee based upon traffic flow above this ratio.”)] [Faratin, The Growing Complexity of Internet Interconnection (“For very large networks, the traffic ratio requirement is usually 2:1, and sometimes 1.5:1.”)] [TeliaSonera International Carrier Global Peering Policy, TeliaSonera (2013) ("The ratio of the aggregate amount of traffic exchanged between the prospective peer and TeliaSonera shall be roughly balanced and the ratio of inbound and outbound traffic shall not exceed a ratio of 3:1")] [AT&T Global IP Network Peering Policy, AT&T (Mar. 1, 2016) ("Peer must maintain a balanced traffic ratio between its network and AT&T. In particular, a new peer must have: a. No more than a 2.00:1 ratio of traffic into AT&T.")].
In order for settlement fees to result in a gain, the imbalance of value of interconnection must be sufficiently greater than the transaction costs.
Imbalanced Value :: Interconnection for a Fee
Where the perceived value of interconnection between networks is imbalanced, interconnection may be for a fee (in the early days of the Internet, academic networks provided settlement free transit). Both networks benefit, but not equally. One network needs the other network more. A small access network needs a large backbone network in order to sell access to its customers to the full Internet; the large backbone provider also benefits from a somewhat greater reach to end-users and destinations, but the additional value may be slight. Acquiring backbone service is indispensable to the small access network, the small network cannot built its own backbone service (at a reasonable cost), and therefore the small network is willing to pay the larger network (also known as a "build or buy" decision). The larger network, which will receive only a slight gain from interconnection, has invested in building out its large network and developed a business plan of selling network services, is willing to provide service only if paid. [ECONOMIDES at 51 (discussing refusal of AT&T to interconnect with independent telephone companies at the turn of the 20th century, “The benefit to an independent telephone company of access to the AT&T long-distance network was much larger than the benefit to AT&T of adding to its network the mostly residential customers of an independent… the incentives of firms of different sizes to interconnect differ depending on the value and size of the new demand that is created by interconnection. Typically, a large and high-value network has a significantly smaller incentive to interconnect with a smaller, low-value network than the smaller one has to interconnect with the larger one. This can easily lead to a refusal by the larger, high-value network to interconnect.”)].
Interconnection-for-a-fee is a vendor-customer relationship. The vendor is selling a service to the customer. “Interconnection” is the point of sale and the measurement of the price, but it is not an accurate reflection of the service being purchased; in other words, not all interconnection arrangements are the same. [Houston INET'99 Sec. 3.1 (asking "What exactly is being exchanged between two ISPs that want to interconnect?")].
Transit is an interconnection arrangement where a customer pays a vendor (a) to exchange traffic (On-Net and Off-Net) and (b) for transit through the vendor's network to other Off-Net networks, providing long haul transport. [BEREC Draft Report p. 18 2012] (“Transit is a wholesale product against a payment.”)] [Freedman Slides 4-5] [GAO p. 11(“Transit and peering are distinctive in two key respects. First, while peering generally entails traffic exchange between two providers without payment, transit entails payment by one provider to another for carrying traffic. Transit agreements thus constitute a supplier-customer relationship between some backbone providers, much like the relationship between a backbone provider and a nonbackbone customer (such as an ISP). Second, when a backbone provider buys transit from another provider, it obtains not only access to the “supplier’s backbone network, but also access to any other backbone network with which its supplier peers.”)] [Nuechterlein p. 133] [ENISA p. 75, 77 (“In any form of transit arrangement one of the parties, the transit provider, will carry traffic to and from other parts of the Internet – not just to and from its own users and customers. The transit provider will charge for this service.”)] [BEREC p. 20 Dec. 6, 2012] [Norton, ISPs and Peering ("Internet Transit is the business relationship whereby one ISP provides (usually sells) access to all destinations in its routing table .")] [AT&T Slide 4] [ISOC p. 3 2012 ("Transit is typically a bilateral agreement where an ISP provides full connectivity to the Internet for upstream and downstream transmission of traffic on behalf of another ISP or end-user including an obligation to carry traffic to third parties.")] [NRIC Sec. 2.5 ("Transit is an agreement where an ISP agrees to carry traffic on behalf of another ISP or end user. In most cases transit will include an obligation to carry traffic to third parties. Transit is usually a bilateral business and technical arrangement, where one provider (the transit provider) agrees to carry traffic to third parties on behalf of another provider or an end user (the customer). In most cases, the transit provider carries traffic to and from its other customers, and to and from every destination on the Internet, as part of the transit arrangement. In a transit agreement, the ISP often also provides ancillary services, such as Service Level Agreements, installation support, local telecom provisioning, and Network Operations Center (NOC) support.")] [SBC / AT&T Merger Order ¶ 110, 2005 ("IBPs may exchange traffic either through “peering” or “transit” arrangements. Under a peering arrangement each IBP “peer” will accept and deliver, without charge, traffic destined either for its own network or for one of its own backbone customers. Transit arrangements, by contrast, permit an ISP, small or regional IBP, or other corporate business, to reach the entire Internet using dedicated access lines linking it directly to the transit provider’s Internet backbone network. An IBP providing transit service enables the customer to send and receive traffic through the purchaser’s IBP to any other network or destination on the Internet. Frequently, IBP customers obtain transit packaged with a dedicated highspeed facility as part of a DIA service, with the transit customers paying fees for both the connection and the transit service").] [Nuechterlein p. 132] [AT&T Mair Declaration ¶ 14] MCI WCOM / Sprint Merger DOJ Complaint ¶ 23; AT&T / Bell South Merger Order 2007, para 123. [SBC / AT&T Merger Order ¶ 110 2005] [WCOM MCI Merger Order para 145]
Backbone Vendor to Access Customer:
(1) Financial Terms: For a Fee
(2)(a) Route Announced:
- Backbone: On-Net and Off-Net
- Access: On-Net Only
(2)(b) Long-Haul Transport: Yes (backbone)
In a typical transit relationship, a backbone service provider announces to the access network that it is a route to the full Internet (the customer access network announces that it is a route only to itself - and thus the relationship is asymmetrical). The backbone service also provides or arranges for long haul transport to the full Internet. This created an administrative efficiency. Through one interconnection arrangement, the access network customer can achieve full connectivity. The access network does not have to negotiate arrangements with other ends (i.e., content sources, application services) on the Internet. The access network negotiates one arrangement with an upstream provider, who in turn negotiates interconnection arrangements with other upstream providers, until they reached Tier 1 and full Internet accessibility (at which point the process is repeated down the other side of the topology). See also multihoming.
There can be full or partial transit. Full transit provides access to the full routing table; partial transit provides access only to select destinations (to a portion of the routing table). [ENISA p. 78]
Access service sold to an end user is fundamentally transit service.
Transit is sold in two ways: (1) per port or (2) peak utilization, a.k.a. 95th Percentile.
- Ports: Networks can sell connectivity on a per port basis. Sold this way, the customer can take full advantage of the capacity of the port.
- 95 Percentile: The 95P method measures almost peak utilization. Implementing this methodology, a network measures 5 minute samples of utilization; the highest 5% utilization samples are thrown away; the billing is set pursuant to the 95th percent highest utilization during the month. [Norton ISP and Peering] [TWC Peering Policy ("“95th Percentile” usage shall be measured monthly on a port-by-port basis and on a 95th percentile peak usage basis. SNMP bandwidth monitoring will sample both directions of traffic at each connection point (port) (i.e., record a data point reflecting how much bandwidth is utilized at that particular instance) every five (5) minutes and store such samples for a period of one (1) month. At the end of each calendar month, the data samples are collected and sorted from highest to lowest usage. The highest fifth percentile (5%) for each direction of usage are discarded, and the greater direction of traffic shall be the “95th Percentile peak usage” number.")] [Amie Elcan, Centurylink, 95th Percentile Billing, NANOG 53 Oct. 10, 2011]
It is common for a transit contract to have a
- Committed Data Rate:- a set amount of minimum utilization that a customer buys every month, regardless of whether in fact that utilization is achieved.
"Paid Peering" is peering (exchange of traffic between an interconnection partner and its customers, but not the partner's peers or the partner's transit vendors) that is for a fee (the fee structure is similar to transit). [Norton ISPs and Peering ("Paid Peering is the business relationship whereby companies (Internet Service Providers (ISPs), Content Distribution Networks (CDNs), Large Scale Network Savvy Content Providers) reciprocally provide access to each others’ customers, but with some form of compensation or settlement fee.")] [Yoo at 95-96 (“Paid peering involves all of the same aspects as conventional peering relationships. Peers announce to the rest of the Internet the addresses that their peering partners control, maintain a sufficient number of interconnection points across the country, and maintain the requisite total volume and traffic ratios. The key difference is that one peering partner pays the other partner for its services.”)] [NRIC Sec. 1.2.2 ("A form of peering in which one party pays the other, in order to offset perceived differences in cost or value received. "]
There are two types of Paid Peering: Backbone Paid Peering and Access Paid Peering.
Backbone Paid Peering
Between two almost similar networks. For example, there could be paid peering between two backbones that are not quite equal, and the payment makes up the differential in value. These types of arrangements were considered rare. [Levy Slide 4 OECD 2011] [Marcus Slide 4 OECD 2011] [What is Paid-Peering, ATDN (2011)] [ENISA 2011 p 77 ("Paid Peering is unusual")] [Norton Peering Playbook Chapt 11 ("Internet peering is typically settlement free")] [Norton Paid Peering] [Golding Slide 19] [ENISA at 77, 98 (“Paid peering is unusual.”)] [BEREC Draft Report at 23 (“as of 2011 only 0,27% of peering contracts are paid for”)] [BEREC Report at 51 (Dec. 6, 2012] (“such paid arrangements exist in practice either not at all or only in rare cases.”) Paid Peering was seen as far back as 1998 with the GTEi versus Exodus dispute. Gordon Cook, A Content Versus Carrier Peering Battle in High Stakes BBN vs. Exodus Dispute, The Cook Report on the Internet October 1998 (John Curran, GTEi, " To be fair, what we would want to do is to first say that, if the traffic is in balance, no one pays anyone else. With this model in looking for payment we would only look at traffic above the two for one ratio acceptable for no cost peering.")
Backbone Vendor to Access Customer:
(1) Financial Terms: For a Fee
(2)(a) Route Announced: On-Net
(2)(b) Long-Haul Transport: No
Access Paid Peering
Between a access provider and a content network (generally a CDN). This is a relatively new phenomena, which first made its appearance in the late 2000s. The access provider has reversed the flow of interconnection fees, going from traditionally paying transit in order to exchange traffic with the full Internet, to being paid for content services to access the BIAS provider's eyeballs. [AT&T Reply Dkt 14-28 at 96 (“For more than two decades, such interconnection has taken the form of ‘transit’ and ‘peering’ agreements, and in recent years, ‘on-net-only’ agreements have arisen in response to growing demands for video and other forms of media-rich content.”)] [Labovitz Slide 7 (describing "paid transit" as an experiment of the evolving Internet core)] Bob Fletcher, IP Backbone: Hard sell, not so much, Renesys (Nov. 20, 2009) (Large eyeball networks (5 million+ subscribers) are selling paid peering to the largest content providers.)
See also ITU WCIT: ETNO Sending Party Pays Proposal
"Access Paid Peering" is also known as:
- "Non Transit Connection" [Wholesale Dedicated IP Transit, Comcast (listing under "our services" the offering of "Non-Transit," which provides access to Comcast customer routes only.).]
- "On-Net Only" interconnection. [AT&T Internet Interconnection Ecosystem, Slides 2, 5, 11 ("On-Net Only arrangements are typically provided to content-rich entities)] [AT&T Letter Oct. 30, 2015 at 3 (“AT&T’s MIS service allows customers to choose and pay for the capacity of their connection and to deliver as much traffic to AT&T’s network as those connections will permit. AT&T’s MIS service is used by large content providers REDACTED content delivery networks REDACTED enterprises, and large and small businesses. AT&T MIS service can be “on-net” services or transit service. An on-net service providers access only to AT&T’s customers. Transit services are Internet Access Services in which AT&T will deliver traffic to virtually any point on the Internet (directly or through its peering arrangements with other ISPs or Internet backbone providers).”)] [AT&T Info Request Response at 13 ("AT&T MIS services allows customers to choose the capacity of their connections and to deliver as much traffic to AT&T's network as those connections will permit. AT&T's MIS service is used by large content providers REDACTED, content delivery networks REDACTED, enterprises, and large and small businesses. AT&T's MIS service can be 'on-net- services or transit services. An on-net service provides access only to AT&T customers. Transit services are Internet Access Services in which AT&T will deliver traffic to virtually any point on the Internet (directly or through its peering arrangements with other ISPs). AT&T recently developed CIP service allows customers to collocate servers in AT&T's network at locations closer to the AT&T end users who will be accessing the content on those servers. CIP customers purchase the space, power, cooling, transport, and other capabilities needed to operate their servers in AT&T's network.")] [AT&T Mair Declaration ¶ 15(A company seeking to deliver traffic to AT&T’s network can also purchase direct connections from AT&T. AT&T sells an Internet access service called “Managed Internet Service” (“MIS”) that allows customers to choose the capacity of their connections and to deliver as much traffic to AT&T’s network as those connections will permit. AT&T’s MIS service is used by large content providers REDACTED CDNs REDACTED enterprises, and large and small businesses.)] [TWC Peering Policy ("”On-Net” means that all traffic must originate from direct bona fide paying customers of Candidate who purchase and use access to all of Candidate’s Routes.")]
- "Paid Interconnect" [Michelle Clancy, Time Warner Cable inks paid interconnect deal with Netflix, RapidTV (Aug. 21, 2014)]
Access networks that are able to charge Access Paid Peering exist in a two-sided market. They charge end-users for access to the full Internet; they also charge content providers for interconnection and access to end-users / audience.
Five U.S. BIAS providers are known to charge paid peering. There are several European BIAS providers who also charge paid peering. [Netflix Petition to Deny, MB Dkt 14-57 at 59 (“Comcast was the first large terminating access network to successfully implement a ‘congest transit pipes’ peering strategy to extract direct payment from Netflix, but it is not the only one to do so. Since agreeing to pay Comcast, Netflix also has agreed to pay TWC, AT&T and Verizon for interconnection.”)][Fitzgerald ("The pendulum has been swinging toward the carriers in such disputes. In recent years several big Web companies, including Google Inc., Microsoft Corp. and Facebook Inc., have begun paying major U.S. broadband providers for direct connections that bring faster and smoother access into their networks.")] [Fletcher (Large eyeball networks (5 million+ subscribers) are selling paid peering to the largest content providers.)] [Cogent Comments MB 14-90 at 12, 14 (stating that the four BIAS providers who have successfully imposed paid peering are Comcast, TWC, Verizon and AT&T)] [Cogent Reply Dkt 14-90 at 15 n. 44 (citing M-Lab Report at 13-14, explaining that "there was congestion at Cogent's interconnection points with AT&T, Centurylink, Comcast, Time Warner Cable, and Verizon at measurement points in Dallas and Los Angeles")] [Korea's ?~$3/month border charge on Youtube, Netflix, etc., Net Policy News]
- Comcast: [Comcast Letter Mar. 12, 2015 at 21 (“Paid peering is not a new development – it has been a long-standing means of interconnecting on the Internet and typically results in lower costs and shorter network paths”)] [Netflix Florance Declaration Dkt 14-57 at 9 ("Comcast fundamentally changed the terms of its relationships with CDNs, transit providers, and content providers from 2008 to 2014. During this period, Comcast succeeded in departing from the previous business norm under which the terminating access network paid for the delivery of traffic to its network, or received such traffic without payment. Instead, Comcast sought to impose, and has succeeded in imposing, a new fee on transit providers, CDNs, and content providers. This fee is imposed in exchange for Comcast's agreement to accept the traffic delivered to its network.")] [Comcast Letter Dkt 14-28 Jan. 23, 2015 (defending practice of paid peering, arguing that it should not fall under FCC regulatory oversight) ] [Comcast Letter Dkt 14-57 June 12, 2014 (describing paid peering as one of the offerings of Comcast)] [Comcast Letter Dkt 07-52 2008 at 2 (noting that “Comcast and Vonage announced a collaborative effort to ensure that any network management technique Comcast chooses to deploy effectively balances the need to avoid network congestion with the need to ensure that VoIP services like Vonage work well for consumers”)] [BEREC Draft Report at 23 (Akamai and Limelight may also have similar arrangements with Comcast)] [Norton Peering Playbook Chapt 10 (noting Limelight agreeing to paid peering with Comcast after Akamai had already entered into such an agreement)]
- AT&T: [AT&T Letter Oct. 30, 2015 at 3 (“AT&T’s MIS service allows customers to choose and pay for the capacity of their connection and to deliver as much traffic to AT&T’s network as those connections will permit. AT&T’s MIS service is used by large content providers REDACTED content delivery networks REDACTED enterprises, and large and small businesses. AT&T MIS service can be “on-net” services or transit service. An on-net service providers access only to AT&T’s customers. Transit services are Internet Access Services in which AT&T will deliver traffic to virtually any point on the Internet (directly or through its peering arrangements with other ISPs or Internet backbone providers).”)] [Netflix Comments MB 14-90 at 14, 24 (“AT&T’s substantial broadband footprint (approximately 10 million subscribers) and its status as a Tier I network operator give it the ability to demand terminating access fees from edge providers such as Netflix.”)] [AT&T Exhibit 30.b.1, AT&T Paid Peering Relationship 2006] [AT&T / DirecTV Order para 214]
- Centurylink: [Centurylink Letter Oct. 30, 2015 at 4 (“Centurylink offers a variety of peering/interconnection arrangements with other networks including settlement-free peering, on-net Internet transit (content delivered to Centurylink end-user only, and off-net Internet transit (content is delivered to both Centurylink end-users and the full Internet.”)] [Centurylink Oct. 31, 2014 Response (providing table detailing sale of paid peering, referring to paid peering as an "industry standard")]
- TWC: [OToole] [Bode] [Spangler] [Clancy]
- Verizon: [Verizon Press Release 2009 (“The new pricing initiative - the Verizon Partner Port Program - gives content owners and CDNs the benefit of a direct connection from their content storage devices to the Verizon Internet backbone network. This allows content owners and CDNs to bypass the traditional backbone peering process, which involves multicarrier delivery systems, when delivering content to the broadband end users on Verizon's network. The result is a faster, more responsive connection to Verizon's networks, priced to be competitive within the content delivery market.”)] [Verizon Letter Dkt 14-57 2015 at 1 (providing worksheet showing traffic from Verizon's largest paid-peering customers - noting that for purposes of this response, Verizon considers "our Partner Port and Cache Port customers to be 'paid peering' customers")] [Verizon Letter Dkt 09-191 2011(defending the practice of paid peering)] [Rayburn Jan 8, 2009 (“Yesterday, Verizon announced a new program dubbed the "Verizon Partner Port Program" which gives content owners and CDNs the benefit of a direct connection from their content storage devices to the Verizon Internet backbone network.”)] [Snider]
Generally, the larger the customer base of the access ISP, the more value that access ISP controls and the greater leverage it has in negotiations. Access ISPs with larger customer bases have been able to negotiate higher access paid peering fees. [Bender slide 11 "there is a strong correlation between a transit provider's fitness and the size of its customer base (need 'eyeballs' to peer)"][Charter / TWC Merger]
See also Terminating Monopoly
Settlement Free Interconnection
Access Provider with Backbone Provider
(1) Financial Terms: Hybrid
(2)(a) Route Announced: Can be Hybrid
(2)(b) Long-Haul Transport:Yes (backbone)
There is a hybrid arrangement that mixes settlement-free peering and paid interconnection. Like SFP, where traffic ratios are relatively balanced, traffic is exchanged on a settlement free basis. However, in those cases where the traffic ratio is out of balance, the sending party pays for the excess traffic delivered to the interconnection partner. These type of arrangements are found between large access networks and backbone providers (note that traditionally traffic has always been imbalanced between backbone providers and access networks in the download direction, with access provider's end-users requesting content (lighter traffic) and backbone providers delivering the requested traffic (heavier traffic).[AT&T Mair Declaration, ¶ 12 ("AT&T’s policy allows a peer to transmit up to two times more traffic to AT&T than it receives from AT&T. It thus allows peering even where there is a substantial traffic imbalance in favor of AT&T’s peer. For existing peers, even if the imbalance modestly exceeds 2:1, AT&T’s peering policy provides that it will work with the peer to find other ways to make the settlement-free peering arrangement equitable and sustainable, such as implementing routing arrangements that reduce AT&T’s costs of carrying the additional traffic. But where the imbalance of traffic substantially exceeds 2:1, an Internet access service or other type of paid arrangement is more appropriate")]
Value in Dispute :: Peering Disputes
Where the imbalance of value is not clear, these are the moments where peering disputes have traditionally occurred. While many pages of creative fiction has been written about who should pay whom, in a competitive market the disputes have generally been resolved by who has to pay whom. Resolution of interconnection disputes has generally been swift as network effect drove resolution. [ FCC TAC QoS and VoIP Interconnection WG Dec. 2012 Final Report at 62 ("FCC should state an expectation that interconnection will not be a source of impairment or blocking.")]
Between BIAS and Content Networks
- Netflix / TWC
- Netflix / Verizon
- Netflix / Comcast
- Netflix / AT&T
- Cogent (Netflix) / Verizon
- Cogent / Orange
Hurricane Electric Bakes Cogent a Cake
Between Backbone Networks
- Cogent / The Department of Energy Sciences Network (ESNet) 2009
- Hurricane Electric / Cogent 2009
- Mike Leber, IPv6 Internet Broken, Cogent/Telia/Hurricane Not Peering, Email to NANOG Discussion Group, Oct. 12, 2009
- Rich Miller, Peering Disputes Migrate to IPv6, Data Knowledge Center (Oct. 22, 2009).
- 1998 GTEinternetworking / Exodus
- Exodus and GTE Internetworking Agree to Increase Internet Traffic Exchanges; Agreement Adds Interconnections and Increases Throughput Capacity by 50%, Business Wire (Sept. 15, 1998)
- Matt Hamblen, GTE, Exodus to speed up ‘net access,’ Computer World (Sept. 15, 1998), available at Archive.org
- Gordon Cook, A Content Versus Carrier Peering Battle with High Stakes BBN v Exodus Dispute, The Cook Report (October 1998)
- Randy Barrett, GTEI, Exodus Make Peace On Peering, Inter@ctive Week, September 16, 1998.
- William Norton, The Art of Peering: The Peering Playbook, p. 7
- 1997 Depeering
- UUNet, Sprint, BBN leave CIX
- Large Networks Depeering [Noam 66]
- Sprint DePeers with Smalls ISPs
- UUNET Depeering
- UUNet Details Peering Strategy; Changing Internet Economics Prompts New Policy, Business Wire (May 12, 1997).
- Jeff Pelline, UUNet, Whole Earth Make Up, CNET April 28, 1997
- Jonathan Marshall, Jon Swartz, Net Service Providers Facing Fees From UUNet, SFGate (Apr. 25, 1997).
- Joan Engebretson, Level 3: Whiner or Visionary, TELEPHONY, May 25, 1998
- Jeff Pelline, Whole Earth Dumps President, CNET (April 30, 1997) ('A memo sent to another ISP from UUNet last month read: "As a result of a recent operational audit, UUNet has determined to reduce the number of its peering relationships. Accordingly, UUNet intends to stop peering with you at MAE-West, an Internet hub. We realize this may cause some modification to your network plans and would like to cease peering 90 days from now, or June 3, 1997. We'd like to work with you to make this transition as smooth as possible." ')
- UUNET Press Release, UUNET Details Peering Strategy: Changing Internet Economics Prompt New Policy (May 12, 1997) (Stating that the economics of the Internet have changed radically in the past few years, UUNET Technologies Inc., the world's largest Internet service provider and a subsidiary of WorldCom Inc. (NASDAQ:WCOM), today detailed its policy regarding peering with other ISP's. The company said it will continue to peer with ISP's that can route traffic on a bilateral and equitable basis. However, UUNET will no longer accept peering requests from ISP's whose infrastructures do not allow for the exchange of similar traffic levels. ) Archive copy at MERIT
- Internet Traffic Exchange: Developments and Policy, OECD DSTI/ICCP/TISP(98)1/Final, p. 15-16 (1998)
- 1992 ANS / NSFNET / CIX
Other Interconnection Information
Transit Cost Avoidance
The Internet is a distributed network that routes around "failures." High interconnection fees are a "failure" that can be routed around. Internet access networks and content networks had an incentive to avoid the high-costs of transit, and began to re-engineer their networks. They would avoid transit fees by
- growing their own backbones which they could use instead of using third party backbones;
- "donut peer," establishing interconnection directly with other networks, bypassing transit fees; and
- directly interconnect with CDNs.
All of these strategies were facilitated by the rise of IXPs; if a provider builds out its network and reached an IXP, it can employ all of those strategies. The economics of the Internet ecosystem would drive evolution and make the network of the 2010s very different from the network of the 1990s.
Public versus private peering
At an IXP, networks can enter into public or private peering arrangements:
- "Public Peering" is achieved utilizing the switches provided by the host IXP. Multiple networks may exchange traffic through the IXP switch, and arrangements can be bilateral or multilateral. ('Public' is a bit of a misnomer as everything involved in this arrangement may be achieved through private contract). [BEREC, p. 26 (Dec. 6, 2012) (Public peering is defined as "several operators peering across the shared peering fabric (Ethernet switch). This fabric interconnects the respective edge routers of the ISPs which peer. This form is most common at an IXP.")] [Mwangi (“Any ISP that is connected to the IXP Switch can exchange traffic with any of the other ISPs connected to the IXP, using a single physical connection. Such peering practice is called "public peering””)] [NRIC Sec. 2.4] [ISOC 2014 p. 15 ("The older, now deprecated model is that the IXP exchanges all traffic between participating networks inside a single router. This is usually called a Layer 3 IXP... Layer 3 model may be less costly and simpler to establish initially, but it is less scalable and limits the autonomy of its members who have less control over with whom they can peer and who are dependent on a third party to configure all routes correctly and keep routes up-to-date.")]
- "Private Peering" is a bilateral arrangement directly between network services, generally achieved by ordering from the IXP a cross connect from one network to the other. [BEREC Report p. 27 (Dec. 6, 2012)] [BEREC Draft Report, p. 24 May 29, 2012] [Kende p. 6] [Mwangi (“"private peering", where two ISPs have a direct physical interconnection”)] [ISOC IXP Guide p. 15]
Peering Policies & PeeringDB
A "Peering Policy" is "[t]he decision criteria that a provider applies in deciding with whom they will peer. " [NRIC Sec. 1.2.2] In the words of Bill Norton, a "Peering Policy" is "an articulation of peering inclination." [Norton, A Guide to Peering Contracts] [Norton Open Peering Policy] Originally, it was an articulation by a backbone network (the networks in the 1990s that peered) of whom it would perceive to be a "peer" or equal. It is like an amusement park sign that says, "you must be this tall to ride." Generally, to be a "peer," a network had to satisfy the settlement free peering proxy and be roughly equal in size and exchange roughly balanced traffic. A peering policy delineated how that would be measured.[Golding][BEREC p. 21 Dec. 6, 2012][NRIC FG 4]
A peering policy is not a peering contract; peering contracts are far more elaborate. It is not a legal "offer;" networks reserve the right to negotiate and to enter into a peering arrangement or not. [NRIC Sec. 1.2.2] [Norton, Peering Policies]
In the 1990s, as the nascent commercial Internet matured, Tier 1 backbones looked at smaller networks, concluded that they were not "peers," and migrated smaller networks to transit customer arrangements. [1997 Depeering] Smaller networks that found themselves with large transit bills complained. [Digital Handshake 2000] [First 706 Report, para. 105 (at the time, commenters unanimously opposed FCC intervention into peering and interconnection disputes with one exception; Bell Atlantic, now Verizon, recommended possible action by the FCC to lower barriers of entry to new entrants).] If they were not large enough to qualify for peering, then they wanted to know how large they had to be. They wanted to know what benchmarks they had to meet to avoid transit fees.
In 2001, the FCC's Network Reliability and Interoperability Council (NRIC, predecessor of CSRIC), led by Jim Crowe of Level 3, recommended that networks post peering policies on their websites. [NRIC Sec. 4.1] [NRIC Internet Peering Statement ("NRIC V encourages other Internet providers, and especially the large "backbone" Internet providers that comprise the core of the modern Internet, to consider, consistent with their business practices, publication of their criteria for peering.") Available on the Web Archive.] [GAO ("We were also told that peering policies should be made public.")] [FCC NRIC Encourages Publication of Peering Criteria to Promote Transparency (Oct. 30, 2001).] In the 2005 mergers (Verizon/WCOM, AT&T/SBC) and the 2007 merger (AT&T/Bell South), the parties agreed to post peering policies as a merger condition. [SBC / AT&T ¶ 133] [Verizon / MCI ¶ 134] [AT&T / Bell South, Appendix F]
Peering policies consist of two different types of clauses.[Compare Norton Survey (dividing the clauses into three groups: (1) operations-related Internet peering policy clauses; (2) Technical / Routing / Interconnection clauses; and (3) General Clauses).] First, provisions that determine whether a potential partner is a "peer" (roughly equal size with roughly balanced traffic), and, second, operational conditions regarding what is necessary to successfully interconnect. These provisions fall out as follows:
Peer Criteria (roughly equal size / traffic) Operational Conditions
- Geographic reach
- Redundant network
- Presence at specified IXPs
- Minimum number of points of interconnection
- Minimum traffic capacity / utilization
- Balanced traffic ratio
- No customers as peers
[Norton, Study of 28 Peering Policies] [Golding] [BEREC p. 21 2012] [NRIC Sec. 4.3 (examples include geographic coverage, proximity to exchange points, minimum capacity, symmetry of traffic exchange, minimum traffic loads, reliable network support, and reasonable address aggregation)] [Verizon (“The key common feature, however, is that these voluntary arrangements involve a mutual exchange of value of one form or another.”)] [Aemen Lodhi, Natalie Larson, Amogh Dhamdhere, Constantine Dovrolis, kc claffy, Using PeeringDB to Understand the Peering Ecosystem, ACM SIGCOMM Computer Communication Review, 44(2), 20-27, 21 (2014), ("Finally, we explore what historical snapshots of the PeeringDB database can tell us about the evolution of the Internet peering ecosystem.")]
[William Norton, Don’t Abuse Peering, DrPeering] [Understanding Default Routes, IS-IS Feature Guide, Juniper Networks (May 20, 2010), ("A default route is the route that takes effect when no other route is available for an IP destination address.")] [Configuring Static Routes, CISCO Nexus 7000 Series NX-OS Unicast Routing Configuration Guide (June 14, 2017)] [Luc De Ghein, Specify a Next Hop Address for Static Routes, CISCO Technical Support, Sept. 2, 2014] [Configuring a Gateway of Last Resort Using IP Commands, CISCO Technical Support, Aug. 10, 2005 ("Default routes are used to direct packets addressed to networks not explicitly listed in the routing table. Default routes are invaluable in topologies where learning all the more specific networks is not desirable, as in case of stub networks, or not feasible due to limited system resources such as memory and processing power.")]
In 2004, PeeringDB came on the scene. PeeringDB is a database created "by and for peering coordinators" that provides
- a link to peering policy,
- peering inclination,
- whether interconnection at multiple locations is required,
- whether there is a balanced ratio requirement,
- whether a contract is required,
- peering contact information, and
- the peering facilities where the network is available for interconnection.
According to PeeringDB, "The purpose of this project is to facilitate the exchange of information related to peering. Specifically, what networks are peering, where they are peering, and if they are likely to peer with you." [Martin Levy, PeeringDB and why everyone should use it, presentation at African Peering and Interconnection Forum 2011, slide 7] In 2016, PeeringDB 2.0 was launched and PeeringDB was established as a non-profit organization. By the end of 2016, it listed 8194 peering networks, 2302 interconnection facilities, and 566 IXPs. [Arnold Nipper, PeeringDB, presentation at CEE Peering Days 2017] [Terry Rodery, PeeringDB, presentation at NANOG 40 (2007)] [Aemen Lodhi, Natalie Larson, Amogh Dhamdhere, Constantine Dovrolis, kc claffy, Using PeeringDB to Understand the Peering Ecosystem, ACM SIGCOMM Computer Communication Review, 44(2), 20-27 (2014)]
Generally, networks inclination to peer is divided into four: open, selective, restrictive, and no peering. "Open" is the most liberal inclination where a network is generally willing to peer with anyone who can meet basic requirements (such as meeting the network at an interconnection point). "Selective" is the similar to open, but the selective networks is looking for some basic requirements before they enter into such relationships (for example, minimum traffic utilization). "Restrictive" is the more traditional inclination where a provider is only willing to entering into peering with "peers," networks that meet a relatively high bar of criteria. These inclinations can be regionally specific. A network may have an open policy in one geographic regional and a selective policy in another.
Providers can have multiple peering policies. Some providers have distinct business units that have distinct inclinations. For example, Verizon's CDN Edgecast has a different peering policy than Verizon broadband Internet. Other companies have multiple policies due to mergers and acquisitions. For example, Charter Communications acquired Time Warner Cable; both networks continue to have posted peering policies. Verizon also has distinct posted peering policies for its acquired assets AOL, Yahoo!, and XO.
Who Interconnects with Whom?
See also WHOIS (database of IP address block allocations, revealing what networks hold what address blocks)
See also Statistics | Assessment | Forensics
Augmenting Interconnection Capacity
Networks grow. The Internet grew phenomenally. If settlement-free interconnecting networks grow at roughly the same rate, they continue to be like-networks with roughly balanced traffic ratios, but their interconnection capacity must be augmented. An industry rule of thumb traditionally has been that when interconnection capacity reached 70 percent utilization, the providers augment capacity. [Estes at 155 ("To compensate for the delay in adding capacity, we were having to order new circuits at about a 65% fill point due to the long lead times that some interexchange carriers had in their order fulfillment processes.")] [AT&T Ex Parte May 26, 2015 at 4 (“Historically, 70 percent utilization was often set as the “trigger” point for augments, to account for the cycle times needed to add capacity before the link became congested.”)] [Amendment No. 3 to the Interconnection Agreement between Verizon New York Inc. and PAETEC Communications, Inc. at 3 (2001) ("The Parties will review all Meet Point B Two-Way Interconnection Trunk groups that reach a utilization level of seventy (70%) percent of the engineered capacity or greater to determine whether those groups should be augmented.") ] [Taylor Level 3 ("Each party pays to augment its own network to allow for more traffic exchange (the expense to augment capacity is not significant for either party). And since we often choose to interconnect in a third party data center, the networks usually agree to share the cost of the cross connects by paying for them on an alternating basis.")] [Netflix Petition to Deny, Dkt 14-57 Att A at 10 (capacity augmented when interconnections reach 70% utilization)] [Cogent Ex Parte MB Docket 14-57 (Nov. 18, 2014)] ("Informal, ad hoc discussions concerning augmentations would occur around the time the links between the network reach 70% utilization, the point at which ISPs though out the industry typically upgrade their interconnections with other networks to avoid service problems.")] [Declaration of Henry (Hank) Kilmer, Cogent, Dkt 14-57 para 16 (Aug. 25, 2014)] [Declaration of Dovrolis, Comcast Sec. 3.2 (Sept. 23, 2014) ("Typically, if the utilization of a link during peak-usage time periods is more than 70%, the link can experience congestion episodes in which traffic is delayed or even dropped.)] In response to congestion, some content networks have included provisions in their peering policies that capacity will be augmented when utilization reaches 50%. [Google Peering Policy ("To ensure sufficient peering capacity in the event of an outage we recommend that peers connect with us at more than one location and augment their peering connections when peak traffic exceeds 50%")] [Microsoft Peering ("Peers are expected to upgrade their ports when peak utilization exceeds 50%")]
Typically, the process of augmenting capacity is short, involving ordering a new cross connect from the IXP. The partners generally take turns covering the costs of installation.
In the case of a customer / vendor relationship, augmentation of capacity is determined by the customer when the customer wants to order additional capacity from the network supplier. For example, an access network customer would ask for additional transit capacity from a backbone service provider.
In order to achieve redundancy and resiliency, transit customers will multihome, purchasing transit service from multiple transit providers. If one transit provider service is disrupted (fiber cuts, peering disputes, infrastructure destruction), the redundant transit service will ensure that the customer remains online. [See Baran Distributed Design] Research has concluded that there is not significant additional benefit to multihoming beyond four upstream providers. [Freedman Slide 4]
Multihoming can be used by transit customers to game the 95th Percentile billing measurement methodology of transit.
The advantage of multihoming can be lost when the strategy provides logically diverse paths (over different Internet service providers) but not physically diverse paths (over the same telecommunications infrastructure service provider) or over the same physical path (through the same conduit).
See 2001 Baltimore Train Tunnel Fire. See also Reliability.
- Effects of Catastrophic Events on Transportation System Management and Operations: Howard Street Tunnel Fire, Baltimore City, Maryland, July 18, 2001, Findings of SAIC, Prepared for the US Department of Transportation, ITS Joint Program Office (July 2002)
- Fire's effects ripple onto the Net, CNET 7/19/0
- Train Wreck Derails WorldCom Service, Newsbytes 7/19/01
- Tunnel Burns, Internet Melts, The Industry Standard ( Jul 20, 2001 )
- Keynote, Baltimore Train Fire Memo (July 19, 2001) [Word]
- GAO 06-672 Internet Infrastructure: DHS Faces Challenges in Developing a Joint Public/Private Recovery Plan, GAO Report, p. 24 (June 2006). "On July 18, 2001, a 60-car freight train derailed in a Baltimore tunnel, causing a fire that interrupted Internet and data services between Washington and New York. The tunnel housed fiber-optic cables that served seven of the biggest U.S. Internet service providers. The fire burned and severed fiber-optic cables, causing backbone slowdowns for at least three major Internet service providers. There were sporadic reports from across the Northeast corridor about service disruptions and delays. For example, users in Baltimore did not suffer disrupted service, while users in Washington D.C. did suffer disruptions. In addition, there were selected impacts far outside the disaster zone. For example, the U.S. embassy in Lusaka, Zambia, experienced problems with e-mail. Two of the service providers had service restored within 2 days. Despite the outages caused by the fire, the Internet continued to operate. Efforts to recover Internet service were handled by the affected Internet service providers. City officials also worked with telecommunications and networking companies to reroute cables. Other federal and local government efforts to resolve the disruption consisted of responding to the immediate physical issues of extinguishing the fire, maintaining safety in the surrounding area, and rerouting traffic.
Interconnection Monitoring and Measurement
See also Assessment.
Interconnection can be monitored with protocols like Netflow.
Wikipedia: Netflow is a feature that was introduced on Cisco routers that gives the ability to collect IP network traffic as it enters or exits an interface. By analyzing the data that is provided by Netflow a network administrator can determine things such as the source and destination of the traffic, class of service, and the cause of congestion. Netflow consists of three components: flow caching, Flow Collector, and Data Analyzer.
Netflow reads the routing information and can identify the source ASN (in other words, if the traffic is coming from the Netflix ASN, then the network knows the source is Netflix). Netflow is also used to validate Level 3's "Bit Mile Routing." An alternative solution is Peakflow by Arbor Networks.
- B. Claise, B. Trammell, and P. Aitken. Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information. Internet Engineering Task Force, Sept. 2013. “This document specifies the IP Flow Information Export (IPFIX) protocol, which serves as a means for transmitting Traffic Flow information over the network. In order to transmit Traffic Flow information from an Exporting Process to a Collecting Process, a common representation of flow data and a standard means of communicating them are required. This document describes how the IPFIX Data and Template Records are carried over a number of transport protocols from an IPFIX Exporting Process to an IPFIX Collecting Process.”
- Nick Feamster, Revealing Utilization at Internet Interconnection Points, March 11, 2016 “Large transit provider networks commonly deploy IPFIX [aka Netflow] across all of the routers in their networks to determine whether certain links are overutilized.”
- K. Hedayat, R. Krzanowski, A. Morton, K. Yum, J. Babiarz, A Two-Way Active Measurement Protocol (TWAMP), IETF RFC 5357 (October 2008) ("The One-way Active Measurement Protocol (OWAMP), specified in RFC 4656, provides a common protocol for measuring one-way metrics between network devices. OWAMP can be used bidirectionally to measure one-way metrics in both directions between two network elements. However, it does not accommodate round-trip or two-way measurements. This memo specifies a Two-Way Active Measurement Protocol (TWAMP), based on the OWAMP, that adds two-way or round-trip measurement capabilities. The TWAMP measurement architecture is usually comprised of two hosts with specific roles, and this allows for some protocol simplifications, making it an attractive alternative in some circumstances.")
- Configuring IP SLA, Catalyst 4500 Series Switch Software Configuration Guide, 12.2(44)SG, Cisco ("This chapter describes how to use Cisco IOS IP Service Level Agreements (SLAs) on the switch. Cisco IP SLAs is a part of Cisco IOS software that allows Cisco customers to analyze IP service levels for IP applications and services by using active traffic monitoring—the generation of traffic in a continuous, reliable, and predictable manner—for measuring network performance. With Cisco IOS IP SLAs, service provider customers can measure and provide service level agreements, and enterprise customers can verify service levels, verify outsourced service level agreements, and understand network performance. Cisco IOS IP SLAs can perform network assessments, verify quality of service (QoS), ease the deployment of new services, and assist with network troubleshooting.")
- Report of the AT&T Independent Measurement Expert Analysis of Reporting Requirements and Proposals for measurement models, Dec 29, 2015