Cybertelecom
Cybertelecom
Federal Internet Law & Policy
An Educational Project
Internet Interconnection Dont be a FOOL; The Law is Not DIY

Introduction

When two networks interconnect, both networks benefit. The reachability to and potential communication opportunities with end-users, services, applications, and content has grown by merit of the interconnection, and thus the value of both networks to their end-users has grown. This is known as “network effect.” The benefit of interconnection to the respective networks may be balanced or imbalanced.

Whether one network "should" pay another network for interconnection, and who pays whom can be difficult to resolve. Geoff Huston wrote

given two ISPs that interconnect, the decision as to which party should assume the upstream provider role and which party should assume the downstream customer role is not always immediately obvious to either party, or even to an outside observer. …In many ways, the outcome of these discussions can be likened to two animals meeting in the jungle at night. Each animal sees only the eyes of the other, and from this limited input, the two animals must determine which animal should attempt to eat the other!

[Houston INET'99 Sec. 4.]

In a functioning market, who pays whom is determined by who has to pay whom (not by subjective determinations of who should pay whom, or what is fair). A consumer pays a baker for bread because the consumer has to pay the baker in order to induce the baker to sell the bread; the baker has no other incentive to give the bread to the consumer and the consumer needs to eat. Where the value of interconnection is relatively balanced, interconnection advances both providers' business plans relatively equally. Where the value of interconnection is imbalanced, one provider needs the other more than the latter needs the former. One provider cannot pursue its business plan without purchasing services from the other, while the other may be able to get along just fine without the later. Who has paid whom has been determined fundamentally by who has had to pay whom in order to receive services and pursue their business plan.

Arrangement provides access to partner's Peering Transit
Network
(On Net)
Y
Y
Transit Customers
(On Net)
Y
Y
Peers
(Off Net)
N
Y
Transit Vendors
(Off Net)
N
Y

Internet interconnection arrangements involves the following characteristics: [Charter Order para. 98] [AT&T Comments Docket 10-90 at 12 (“This peering and transit regime encompasses interconnection agreements among all of the Internet’s constituent IP networks, not just those between “backbone” networks”)] [PCH Survey 2016 at 1 ("Each of the interconnecting links takes one of two forms: transit or peering")]

  1. Financial terms (settlement-free or for a fee)
  2. Traffic exchanged
    • (a) Routes announced (See BGP)
      • On-Net only in both directions ~ Peering
      • On-Net and Off-Net ~ Transit
    • (b) Long-haul transport (yes or no)

A network announces On-Net routes when it announces to its interconnection partner that it is a route to itself and to its customers. A network announces Off-Net routes when it announces that it is a route to its other peering partners and to its transit vendors.

Interconnection arrangements involves backbone service providers, internet access service providers, content delivery networks, and end users (edge providers, individual users). Interconnection between networks commonly takes place at an Internet eXchange Points. [BEREC p. 10 Dec. 6 2012] [OECD p. 7 2014 (providing players in the "content distribution value chain")] [Verizon v. FCC, No. 11-1355, Slip at  5 (D.C. Cir. 2014) ("Four major participants in the Internet marketplace are relevant to the issues before us: backbone networks, broadband providers, edge providers, and end users").]

Balanced Value :: Settlement-Free Peering

Financial Terms: Where the perceived value of interconnection between networks is balanced, network providers may enter into settlement-free peering arrangements. Both providers perceive that the benefit gained from interconnection exceeds potential revenue gained from attempting to charge for interconnection, or that they would be unable to impose an interconnection fee on their interconnection partner. [ENISA Report at 93 (“In practice ASes will peer if they both believe that the benefit to themselves outweighs any perceived (or, indeed, actual) inequality in costs or benefits.”)] [NRIC Report Sec. 2.6 ("ISPs tend to peer with ISPs of a similar scale (as this often allows for a perceived rough equality of value).")] [Brock, Price Structure Issues in Interconnection Fees at 3 ("We should expect to see 'sender keep all' arrangements develop in a competitive communications market if either of two conditions are met: Traffic flows are very roughly balanced among the companies so that each sees a clear benefit for its customers in both sending and receiving traffic from other companies; OR The cost to a company of terminating traffic is low in relationship to the transaction costs of measuring and charging for traffic so that even with unbalanced traffic companies find the simple 'sender keep all' approach superior to efforts to develop appropriate cost-based terminating charges.")]. [AT&T, What Does it Mean to be a “Peer”? (“Peering works between networks that are of similar size and scope and have similar traffic characteristics.  Such networks are able to assess whether in exchanging traffic on behalf of their users, they obtain equivalent value to what they provide”)] [ENISA Report at 75 (“This is a generally symmetrical and mutually beneficial arrangement.”)].

Both providers cover their own costs and neither charges the other based on traffic exchanged. Both providers benefit from the interconnection and the value of their service grows. [BEREC p. 22 Dec. 6, 2012 (noting costs associated with "settlement free" peering)]

Settlement-free peering avoids transaction costs, improves QoS and resiliency, and focuses on selling Internet access to customers as the primary source of revenue. [Brock, Economics of Interconnection at iv ("'Sender keeps all' is an administratively simple mutual compensation scheme with zero prices for terminating service.")] [Norton Internet Service Providers and Peering] [BEREC Draft Report 21] [Moyle-Croft] [Sutrisno] [Norton Remote Peering] Settlement-free peering is also attractive because it creates an incentive for innovation and efficiency; a provider that brings traffic to the exchange point efficiently gets to keep the savings. [Brock, Interconnection and Mutual Compensation with Partial Competition at 13 (Under sender keep all, each company has an incentive to increase the efficiency of its operations in order to reduce its costs and to maximize its outgoing traffic relative to its incoming traffic because outgoing traffic is the most profitable.).]

Traffic Exchanged: Peering is an interconnection arrangement where two parties (generally networks) exchange traffic between (1) themselves and (2) their transit customers (a.k.a. On-Net traffic), but do not provide transit through their networks to their other peering partners or transit vendors. [Golding] [BEREC Draft p. 19 2012] [BEREC p. 21 Dec. 6, 2012] [Norton ISPs and Peering ("Internet Peering is the business relationship whereby companies reciprocally provide access to each others’ customers.")] [AT&T Slide 2] [NRIC Sec. 2.5 ("Peering is an agreement between ISPs to carry traffic for each other and for their respective customers. Peering does not include the obligation to carry traffic to third parties. Peering is usually a bilateral business and technical arrangement, where two providers agree to accept traffic from one another, and from one another’s customers (and thus from their customers’ customers)."] [ISOC p. 3 ("Peering agreements are where two network providers agree to exchange traffic between each other's customers (but not use each other's transit connections). Peering is the act of exchanging traffic with a peer.")] [AT&T Mair Declaration para 11 ("Large ISPs often interconnect with AT&T through peering. Peering is a private commercial arrangement under which two “peer” ISPs connect and exchange traffic. Each peer provides the other with access only to its own customers – not to the entire Internet.") MCI WCOM / Sprint Merger DOJ Complaint ¶ 24; AT&T / Bell South Merger Order 2007, para 123. [SBC / AT&T Merger Order ¶ 110 2005] [WCOM MCI Merger Order para 145][TWC Peering Policy ("“Peering” is the interaction between two distinct ASN’s using BGP to exchange TCP/IP routing information between service providers operating Internet Networks")] [First Sec. 706 Report, 1999, ¶ 105 & n 240 ("an arrangement in which two Internet backbone providers exchange traffic that originates from an end user connected to one of the providers and terminates with an end user connected to the other provider.")] [WCOM / MCI, 1998, ¶ 145 ("In a peering arrangement, two IBPs agree to exchange traffic that originates from an end user connected to one IBP and terminates with an end user connected to another IBP.")]

Settlement-free peering has also been referred to as

Networks have different approaches to settlement free peering.  These approaches include

[Norton, Open Peering Policy]

See also History of Settlement Free Peering.

Backbone providers

Backbone SFP:

(1) Financial Terms: Settlement Free
(2)(a) Route Announced: On-Net
(2)(b) Long-Haul Transport: Yes (shared)

Two backbone service providers may "peer." They announce to each other that they are routes to their own networks and to their customers (On-Net traffic). Because backbones are large, these peering arrangements involve the exchange of a large amount of routes and traffic.

Traditionally, backbone providers interconnect at multiple IXPs throughout the nation. The backbone provider of the originating traffic will deliver the traffic to the partner backbone at the closest point of interconnection to the source of traffic (a.k.a. "Hot Potato Routing). Thus the terminating network will provide long haul transport to the destination. Since traffic is relatively balanced and multidirectional, the partners will share responsibilities for the long haul transport.

Donut Peering

Donut Peering SFP:

(1) Financial Terms: Settlement Free
(2)(a) Route Announced: On-Net
(2)(b) Long-Haul Transport: No

Networks that are not backbones may peer. Two local networks can directly peer, exchanging traffic between themselves, avoiding sending traffic to a far off interconnection point to be exchanged (known as "tromboning") and avoiding buying transit from a backbone provider for that traffic. This is known a “donut-peering” because the networks are interconnecting around the core backbone providers. This peering arrangement is limited and controlled. The routes exchanged are only those between the two partners and generally neither party is agreeing to long haul transport for the other's traffic. [Levy Slide 12 OECD 2011 (noting that it can be quite expensive for a network to build its own peered network)] [Marcus Slide 17 OECD 2011 ("Any self-supply is presumably cheaper than buying transit")] [Krogfoss Slide 5 OECD 2011 ("50 global traceroute servers showed 60% of routes to large ASPs avoided T1 ISPs" (backbones)]] [Daniel Golding, AOL, Peering Evolution, Presentation at NANOG 27, Slide 14 (2003), ("The current "hot idea" in peering is the donut - networks with open peering going 'around the core' to peer.")] [Houston INET99 Sec. Sec. 5.2 ("Local ISPs participate at exchanges and announce local routes at the exchange on an SKA basis of interconnection with peer ISPs. Such ISPs are strongly motivated to prefer to use all routes presented at the exchange within such peering sessions, as the ISP is not charged any transit cost for the traffic under an SKA settlement structure.")] [BEREC Report 2012 at 24 (discussing regional or donut peering)] [Kende Assessment of the Impact of IXPs (discussing the benefits of local peering)].

The need for donut-peering became acute with the historic evolution of the Internet. The United States was the inventor and first mover of the Internet, which meant that Internet infrastructure developed earlier in the United States. It was not uncommon that Internet traffic on one side of the street in, for example, Europe, destined to the other side of the street, would cross the Atlantic and be exchanged in Northern Virginia. This led to the ICAIS debate over the cost of international transport. The solution is regional or donut peering. Michael Kende in an excellent Internet Society paper reviews the benefits of regional peering: traffic can be kept in-country and out of the reach of surveillance efforts; the building of local exchanges and promote local infrastructure and drive develop local IT businesses, and regional peering bypasses transit expenses.[Kende Assessment of the Impact of IXPs (discussing the benefits of local peering)].

CDNs

CDN SFP:

(1) Financial Terms: Settlement Free
(2)(a) Route Announced: On-Net
(2)(b) Long-Haul Transport: Yes (CDN)

Content delivery networks can agree to a peering interconnection relationship with an access network. The two networks only exchange routes between each other. The content delivery network agrees to be responsible for the long haul transport, carrying the traffic as close to the requesting eyeballs of the access network as possible, exchanging the traffic at the gateway of the access network. This is known as "cold potato routing." At the interconnection between the CDN and the access network, the interconnecting partners only exchange on-net traffic.

Traditionally, this arrangement is a win-win for all partners. The content source gains the efficient delivery of the CDN, which reduces transit costs while improving quality of service. The access network likewise avoids transit costs by receiving traffic directly from the CDN instead of over transit links, and the access network customers receive an improved quality of service of the content.

See also Paid Peering (arrangement between CDNs and large access networks)

Slightly Imbalanced Value

In between balanced and imbalanced relationships are arrangements where the value exchanged is somewhat out of balanced. In these situations, whether an arrangement is settlement-free or for a fee may be determined by whether the transaction costs for charging settlement fees exceeds the imbalance of value. Transaction costs include negotiating, administering, measuring and accounting for the transaction. In situations where transaction costs exceed an imbalance of value, both networks would gain more by interconnecting on a settlement-free basis. [Comments of AT&T, Dkt 10-90, 2012 at 13 (“To avoid administrative overhead, parties to these bilateral peering agreements typically forgo the mutual exchange of compensation and peer on a settlement-free basis.”)] [Verizon Email Nov. 6, 2015 (“For example, where two networks exchange roughly equal amounts of traffic and are otherwise similarly situated, they have been more likely to interconnect without bothering to exchange money.”)] [Brock, Economics of Interconnection at iv ("'Sender keeps all' is an administratively simple mutual compensation scheme with zero prices for terminating service.")].

Internet networks have repeatedly stated that not insignificant imbalances of traffic do not disturb their willingness to enter into settlement-free peering arrangements (the network effect benefits being sufficient to drive interconnection). [Norton, Internet Service Providers and Peering (“peering is free until traffic asymmetry reaches a certain ratio (4:1 is common). At this point, the net source of traffic will pay the net sink of traffic a fee based upon traffic flow above this ratio.”)] [Faratin, The Growing Complexity of Internet Interconnection (“For very large networks, the traffic ratio requirement is usually 2:1, and sometimes 1.5:1.”)] [TeliaSonera International Carrier Global Peering Policy, TeliaSonera (2013) ("The ratio of the aggregate amount of traffic exchanged between the prospective peer and TeliaSonera shall be roughly balanced and the ratio of inbound and outbound traffic shall not exceed a ratio of 3:1")] [AT&T Global IP Network Peering Policy, AT&T (Mar. 1, 2016)  ("Peer must maintain a balanced traffic ratio between its network and AT&T. In particular, a new peer must have: a. No more than a 2.00:1 ratio of traffic into AT&T.")].

In order for settlement fees to result in a gain, the imbalance of value of interconnection must be sufficiently greater than the transaction costs.

Imbalanced Value :: Interconnection for a Fee

Where the perceived value of interconnection between networks is imbalanced, interconnection may be for a fee (in the early days of the Internet, academic networks provided settlement free transit). Both networks benefit, but not equally. One network needs the other network more. A small access network needs a large backbone network in order to sell access to its customers to the full Internet; the large backbone provider also benefits from a somewhat greater reach to end-users and destinations, but the additional value may be slight. Acquiring backbone service is indispensable to the small access network, the small network cannot built its own backbone service (at a reasonable cost), and therefore the small network is willing to pay the larger network (also known as a "build or buy" decision). The larger network, which will receive only a slight gain from interconnection, has invested in building out its large network and developed a business plan of selling network services, is willing to provide service only if paid. [ECONOMIDES at 51 (discussing refusal of AT&T to interconnect with independent telephone companies at the turn of the 20th century, “The benefit to an independent telephone company of access to the AT&T long-distance network was much larger than the benefit to AT&T of adding to its network the mostly residential customers of an independent… the incentives of firms of different sizes to interconnect differ depending on the value and size of the new demand that is created by interconnection. Typically, a large and high-value network has a significantly smaller incentive to interconnect with a smaller, low-value network than the smaller one has to interconnect with the larger one. This can easily lead to a refusal by the larger, high-value network to interconnect.”)].

Interconnection-for-a-fee is a vendor-customer relationship. The vendor is selling a service to the customer. “Interconnection” is the point of sale and the measurement of the price, but it is not an accurate reflection of the service being purchased; in other words, not all interconnection arrangements are the same. [Houston INET'99 Sec. 3.1 (asking "What exactly is being exchanged between two ISPs that want to interconnect?")].

Transit

Transit is an interconnection arrangement where a customer pays a vendor (a) to exchange traffic (On-Net and Off-Net) and (b) for transit through the vendor's network to other Off-Net networks, providing long haul transport. [BEREC Draft Report p. 18 2012] (“Transit is a wholesale product against a payment.”)] [Freedman Slides 4-5] [GAO p. 11(“Transit and peering are distinctive in two key respects. First, while peering generally entails traffic exchange between two providers without payment, transit entails payment by one provider to another for carrying traffic. Transit agreements thus constitute a supplier-customer relationship between some backbone providers, much like the relationship between a backbone provider and a nonbackbone customer (such as an ISP). Second, when a backbone provider buys transit from another provider, it obtains not only access to the “supplier’s backbone network, but also access to any other backbone network with which its supplier peers.”)] [Nuechterlein p. 133] [ENISA p. 75, 77 (“In any form of transit arrangement one of the parties, the transit provider, will carry traffic to and from other parts of the Internet – not just to and from its own users and customers. The transit provider will charge for this service.”)] [BEREC p. 20 Dec. 6, 2012] [Norton, ISPs and Peering ("Internet Transit is the business relationship whereby one ISP provides (usually sells) access to all destinations in its routing table .")] [AT&T Slide 4] [ISOC p. 3 2012 ("Transit is typically a bilateral agreement where an ISP provides full connectivity to the Internet for upstream and downstream transmission of traffic on behalf of another ISP or end-user including an obligation to carry traffic to third parties.")] [NRIC Sec. 2.5 ("Transit is an agreement where an ISP agrees to carry traffic on behalf of another ISP or end user. In most cases transit will include an obligation to carry traffic to third parties. Transit is usually a bilateral business and technical arrangement, where one provider (the transit provider) agrees to carry traffic to third parties on behalf of another provider or an end user (the customer).  In most cases, the transit provider carries traffic to and from its other customers, and to and from every destination on the Internet, as part of the transit arrangement.  In a transit agreement, the ISP often also provides ancillary services, such as Service Level Agreements, installation support, local telecom provisioning, and Network Operations Center (NOC) support.")] [SBC / AT&T Merger Order ¶ 110, 2005 ("IBPs may exchange traffic either through “peering” or “transit” arrangements. Under a peering arrangement each IBP “peer” will accept and deliver, without charge, traffic destined either for its own network or for one of its own backbone customers. Transit arrangements, by contrast, permit an ISP, small or regional IBP, or other corporate business, to reach the entire Internet using dedicated access lines linking it directly to the transit provider’s Internet backbone network. An IBP providing transit service enables the customer to send and receive traffic through the purchaser’s IBP to any other network or destination on the Internet. Frequently, IBP customers obtain transit packaged with a dedicated highspeed facility as part of a DIA service, with the transit customers paying fees for both the connection and the transit service").] [Nuechterlein p. 132] [AT&T Mair Declaration ¶ 14] MCI WCOM / Sprint Merger DOJ Complaint ¶ 23; AT&T / Bell South Merger Order 2007, para 123. [SBC / AT&T Merger Order ¶ 110 2005] [WCOM MCI Merger Order para 145]

Backbone Vendor to Access Customer:

(1) Financial Terms: For a Fee
(2)(a) Route Announced:
- Backbone: On-Net and Off-Net
- Access: On-Net Only
(2)(b) Long-Haul Transport: Yes (backbone)

In a typical transit relationship, a backbone service provider announces to the access network that it is a route to the full Internet (the customer access network announces that it is a route only to itself - and thus the relationship is asymmetrical). The backbone service also provides or arranges for long haul transport to the full Internet. This created an administrative efficiency. Through one interconnection arrangement, the access network customer can achieve full connectivity. The access network does not have to negotiate arrangements with other ends (i.e., content sources, application services) on the Internet. The access network negotiates one arrangement with an upstream provider, who in turn negotiates interconnection arrangements with other upstream providers, until they reached Tier 1 and full Internet accessibility (at which point the process is repeated down the other side of the topology). See also multihoming.

There can be full or partial transit.  Full transit provides access to the full routing table; partial transit provides access only to select destinations (to a portion of the routing table). [ENISA p. 78]

Access service sold to an end user is fundamentally transit service.

Transit Contracts

Transit prices

Transit is sold in two ways: (1) per port or (2) peak utilization, a.k.a. 95th Percentile.

  • Ports: Networks can sell connectivity on a per port basis.  Sold this way, the customer can take full advantage of the capacity of the port.
  • 95 Percentile: The 95P method measures almost peak utilization. Implementing this methodology, a network measures 5 minute samples of utilization; the highest 5% utilization samples are thrown away; the billing is set pursuant to the 95th percent highest utilization during the month. [Norton ISP and Peering] [TWC Peering Policy ("“95th Percentile” usage shall be measured monthly on a port-by-port basis and on a 95th percentile peak usage basis. SNMP bandwidth monitoring will sample both directions of traffic at each connection point (port) (i.e., record a data point reflecting how much bandwidth is utilized at that particular instance) every five (5) minutes and store such samples for a period of one (1) month. At the end of each calendar month, the data samples are collected and sorted from highest to lowest usage. The highest fifth percentile (5%) for each direction of usage are discarded, and the greater direction of traffic shall be the “95th Percentile peak usage” number.")] [Amie Elcan, Centurylink, 95th Percentile Billing, NANOG 53 Oct. 10, 2011]

It is common for a transit contract to have a

  • Committed Data Rate:- a set amount of minimum utilization that a customer buys every month, regardless of whether in fact that utilization is achieved. 
  • SLA

Paid Peering

"Paid Peering" is peering (exchange of traffic between an interconnection partner and its customers, but not the partner's peers or the partner's transit vendors) that is for a fee (the fee structure is similar to transit). [Norton ISPs and Peering ("Paid Peering is the business relationship whereby companies (Internet Service Providers (ISPs), Content Distribution Networks (CDNs), Large Scale Network Savvy Content Providers) reciprocally provide access to each others’ customers, but with some form of compensation or settlement fee.")] [Yoo at 95-96 (“Paid peering involves all of the same aspects as conventional peering relationships. Peers announce to the rest of the Internet the addresses that their peering partners control, maintain a sufficient number of interconnection points across the country, and maintain the requisite total volume and traffic ratios. The key difference is that one peering partner pays the other partner for its services.”)] [NRIC Sec. 1.2.2 ("A form of peering in which one party pays the other, in order to offset perceived differences in cost or value received. "]

There are two types of Paid Peering: Backbone Paid Peering and Access Paid Peering.

Backbone Paid Peering

Between two almost similar networks. For example, there could be paid peering between two backbones that are not quite equal, and the payment makes up the differential in value. These types of arrangements were considered rare. [Levy Slide 4 OECD 2011] [Marcus Slide 4 OECD 2011] [What is Paid-Peering, ATDN (2011)] [ENISA 2011 p 77 ("Paid Peering is unusual")] [Norton Peering Playbook Chapt 11 ("Internet peering is typically settlement free")] [Norton Paid Peering] [Golding Slide 19] [ENISA at 77, 98 (“Paid peering is unusual.”)] [BEREC Draft Report at 23 (“as of 2011 only 0,27% of peering contracts are paid for”)] [BEREC Report at 51 (Dec. 6, 2012] (“such paid arrangements exist in practice either not at all or only in rare cases.”) Paid Peering was seen as far back as 1998 with the GTEi versus Exodus dispute. Gordon Cook, A Content Versus Carrier Peering Battle in High Stakes BBN vs. Exodus Dispute, The Cook Report on the Internet October 1998 (John Curran, GTEi, " To be fair, what we would want to do is to first say that, if the traffic is in balance, no one pays anyone else. With this model in looking for payment we would only look at traffic above the two for one ratio acceptable for no cost peering.")

Backbone Vendor to Access Customer:

(1) Financial Terms: For a Fee
(2)(a) Route Announced: On-Net
(2)(b) Long-Haul Transport: No

Access Paid Peering

Between a access provider and a content network (generally a CDN). This is a relatively new phenomena, which first made its appearance in the late 2000s. The access provider has reversed the flow of interconnection fees, going from traditionally paying transit in order to exchange traffic with the full Internet, to being paid for content services to access the BIAS provider's eyeballs. [AT&T Reply Dkt 14-28 at 96 (“For more than two decades, such interconnection has taken the form of ‘transit’ and ‘peering’ agreements, and in recent years, ‘on-net-only’ agreements have arisen in response to growing demands for video and other forms of media-rich content.”)] [Labovitz Slide 7 (describing "paid transit" as an experiment of the evolving Internet core)] Bob Fletcher, IP Backbone: Hard sell, not so much, Renesys (Nov. 20, 2009) (Large eyeball networks (5 million+ subscribers) are selling paid peering to the largest content providers.)

See also ITU WCIT: ETNO Sending Party Pays Proposal

"Access Paid Peering" is also known as:

  • "Non Transit Connection" [Wholesale Dedicated IP Transit, Comcast (listing under "our services" the offering of "Non-Transit," which provides access to Comcast customer routes only.).]
  • "On-Net Only" interconnection. [AT&T Internet Interconnection Ecosystem, Slides 2, 5, 11 ("On-Net Only arrangements are typically provided to content-rich entities)] [AT&T Letter Oct. 30, 2015 at 3 (“AT&T’s MIS service allows customers to choose and pay for the capacity of their connection and to deliver as much traffic to AT&T’s network as those connections will permit. AT&T’s MIS service is used by large content providers  REDACTED content delivery networks REDACTED enterprises, and large and small businesses.  AT&T MIS service can be “on-net” services or transit service.  An on-net service providers access only to AT&T’s customers. Transit services are Internet Access Services in which AT&T will deliver traffic to virtually any point on the Internet (directly or through its peering arrangements with other ISPs or Internet backbone providers).”)] [AT&T Info Request Response at 13 ("AT&T MIS services allows customers to choose the capacity of their connections and to deliver as much traffic to AT&T's network as those connections will permit. AT&T's MIS service is used by large content providers REDACTED, content delivery networks REDACTED, enterprises, and large and small businesses.  AT&T's MIS service can be 'on-net- services or transit services. An on-net service provides access only to AT&T customers. Transit services are Internet Access Services in which AT&T will deliver traffic to virtually any point on the Internet (directly or through its peering arrangements with other ISPs). AT&T recently developed CIP service allows customers to collocate servers in AT&T's network at locations closer to the AT&T end users who will be accessing the content on those servers. CIP customers purchase the space, power, cooling, transport, and other capabilities needed to operate their servers in AT&T's network.")] [AT&T Mair Declaration ¶ 15(A company seeking to deliver traffic to AT&T’s network can also purchase direct connections from AT&T. AT&T sells an Internet access service called “Managed Internet Service” (“MIS”) that allows customers to choose the capacity of their connections and to deliver as much traffic to AT&T’s network as those connections will permit. AT&T’s MIS service is used by large content providers REDACTED CDNs REDACTED enterprises, and large and small businesses.)] [TWC Peering Policy ("”On-Net” means that all traffic must originate from direct bona fide paying customers of Candidate who purchase and use access to all of Candidate’s Routes.")]
  • "Paid Interconnect" [Michelle Clancy, Time Warner Cable inks paid interconnect deal with Netflix, RapidTV (Aug. 21, 2014)]

Access networks that are able to charge Access Paid Peering exist in a two-sided market. They charge end-users for access to the full Internet; they also charge content providers for interconnection and access to end-users / audience.

Five U.S. BIAS providers are known to charge paid peering. There are several European BIAS providers who also charge paid peering. [Netflix Petition to Deny, MB Dkt 14-57 at 59 (“Comcast was the first large terminating access network to successfully implement a ‘congest transit pipes’ peering strategy to extract direct payment from Netflix, but it is not the only one to do so. Since agreeing to pay Comcast, Netflix also has agreed to pay TWC, AT&T and Verizon for interconnection.”)][Fitzgerald ("The pendulum has been swinging toward the carriers in such disputes. In recent years several big Web companies, including Google Inc., Microsoft Corp. and Facebook Inc., have begun paying major U.S. broadband providers for direct connections that bring faster and smoother access into their networks.")] [Fletcher (Large eyeball networks (5 million+ subscribers) are selling paid peering to the largest content providers.)] [Cogent Comments MB 14-90 at 12, 14 (stating that the four BIAS providers who have successfully imposed paid peering are Comcast, TWC, Verizon and AT&T)] [Cogent Reply Dkt 14-90 at 15 n. 44 (citing M-Lab Report at 13-14, explaining that "there was congestion at Cogent's interconnection points with AT&T, Centurylink, Comcast, Time Warner Cable, and Verizon at measurement points in Dallas and Los Angeles")] [Korea's ?~$3/month border charge on Youtube, Netflix, etc., Net Policy News]

  • Comcast: [Comcast Letter Mar. 12, 2015 at 21 (“Paid peering is not a new development – it has been a long-standing means of interconnecting on the Internet and typically results in lower costs and shorter network paths”)] [Netflix Florance Declaration Dkt 14-57 at 9 ("Comcast fundamentally changed the terms of its relationships with CDNs, transit providers, and content providers from 2008 to 2014. During this period, Comcast succeeded in departing from the previous business norm under which the terminating access network paid for the delivery of traffic to its network, or received such traffic without payment. Instead, Comcast sought to impose, and has succeeded in imposing, a new fee on transit providers, CDNs, and content providers. This fee is imposed in exchange for Comcast's agreement to accept the traffic delivered to its network.")] [Comcast Letter Dkt 14-28 Jan. 23, 2015 (defending practice of paid peering, arguing that it should not fall under FCC regulatory oversight) ] [Comcast Letter Dkt 14-57 June 12, 2014 (describing paid peering as one of the offerings of Comcast)] [Comcast Letter Dkt 07-52 2008 at 2 (noting that “Comcast and Vonage announced a collaborative effort to ensure that any network management technique Comcast chooses to deploy effectively balances the need to avoid network congestion with the need to ensure that VoIP services like Vonage work well for consumers”)] [BEREC Draft Report at 23 (Akamai and Limelight may also have similar arrangements with Comcast)] [Norton Peering Playbook Chapt 10 (noting Limelight agreeing to paid peering with Comcast after Akamai had already entered into such an agreement)]
  • AT&T: [AT&T Letter Oct. 30, 2015 at 3 (“AT&T’s MIS service allows customers to choose and pay for the capacity of their connection and to deliver as much traffic to AT&T’s network as those connections will permit. AT&T’s MIS service is used by large content providers REDACTED content delivery networks REDACTED enterprises, and large and small businesses.  AT&T MIS service can be “on-net” services or transit service.  An on-net service providers access only to AT&T’s customers. Transit services are Internet Access Services in which AT&T will deliver traffic to virtually any point on the Internet (directly or through its peering arrangements with other ISPs or Internet backbone providers).”)] [Netflix Comments MB 14-90 at 14, 24 (“AT&T’s substantial broadband footprint (approximately 10 million subscribers) and its status as a Tier I network operator give it the ability to demand terminating access fees from edge providers such as Netflix.”)] [AT&T Exhibit 30.b.1, AT&T Paid Peering Relationship 2006] [AT&T / DirecTV Order para 214]
  • Centurylink: [Centurylink Letter Oct. 30, 2015 at 4 (“Centurylink offers a variety of peering/interconnection arrangements with other networks including settlement-free peering, on-net Internet transit (content delivered to Centurylink end-user only, and off-net Internet transit (content is delivered to both Centurylink end-users and the full Internet.”)] [Centurylink Oct. 31, 2014 Response (providing table detailing sale of paid peering, referring to paid peering as an "industry standard")]
  • TWC: [OToole] [Bode] [Spangler] [Clancy]
  • Verizon: [Verizon Press Release 2009 (“The new pricing initiative - the Verizon Partner Port Program - gives content owners and CDNs the benefit of a direct connection from their content storage devices to the Verizon Internet backbone network.  This allows content owners and CDNs to bypass the traditional backbone peering process, which involves multicarrier delivery systems, when delivering content to the broadband end users on Verizon's network.  The result is a faster, more responsive connection to Verizon's networks, priced to be competitive within the content delivery market.”)] [Verizon Letter Dkt 14-57 2015 at 1 (providing worksheet showing traffic from Verizon's largest paid-peering customers - noting that for purposes of this response, Verizon considers "our Partner Port and Cache Port customers to be 'paid peering' customers")] [Verizon Letter Dkt 09-191 2011(defending the practice of paid peering)] [Rayburn Jan 8, 2009 (“Yesterday, Verizon announced a new program dubbed the "Verizon Partner Port Program" which gives content owners and CDNs the benefit of a direct connection from their content storage devices to the Verizon Internet backbone network.”)] [Snider]

Generally, the larger the customer base of the access ISP, the more value that access ISP controls and the greater leverage it has in negotiations. Access ISPs with larger customer bases have been able to negotiate higher access paid peering fees. [Bender slide 11 "there is a strong correlation between a transit provider's fitness and the size of its customer base (need 'eyeballs' to peer)"][Charter / TWC Merger]

See also Terminating Monopoly

Settlement Free Interconnection

Access Provider with Backbone Provider

(1) Financial Terms: Hybrid
(2)(a) Route Announced: Can be Hybrid
(2)(b) Long-Haul Transport:Yes (backbone)

There is a hybrid arrangement that mixes settlement-free peering and paid interconnection. Like SFP, where traffic ratios are relatively balanced, traffic is exchanged on a settlement free basis. However, in those cases where the traffic ratio is out of balance, the sending party pays for the excess traffic delivered to the interconnection partner. These type of arrangements are found between large access networks and backbone providers (note that traditionally traffic has always been imbalanced between backbone providers and access networks in the download direction, with access provider's end-users requesting content (lighter traffic) and backbone providers delivering the requested traffic (heavier traffic).[AT&T Mair Declaration, ¶ 12 ("AT&T’s policy allows a peer to transmit up to two times more traffic to AT&T than it receives from AT&T. It thus allows peering even where there is a substantial traffic imbalance in favor of AT&T’s peer. For existing peers, even if the imbalance modestly exceeds 2:1, AT&T’s peering policy provides that it will work with the peer to find other ways to make the settlement-free peering arrangement equitable and sustainable, such as implementing routing arrangements that reduce AT&T’s costs of carrying the additional traffic. But where the imbalance of traffic substantially exceeds 2:1, an Internet access service or other type of paid arrangement is more appropriate")]

Value in Dispute :: Peering Disputes

Where the imbalance of value is not clear, these are the moments where peering disputes have traditionally occurred. While many pages of creative fiction has been written about who should pay whom, in a competitive market the disputes have generally been resolved by who has to pay whom. Resolution of interconnection disputes has generally been swift as network effect drove resolution. [ FCC TAC QoS and VoIP Interconnection WG Dec. 2012 Final Report at 62 ("FCC should state an expectation that interconnection will not be a source of impairment or blocking.")]

Between BIAS and Content Networks

Between Backbone Networks

Other Interconnection Information

Transit Cost Avoidance

The Internet is a distributed network that routes around "failures." High interconnection fees are a "failure" that can be routed around. Internet access networks and content networks had an incentive to avoid the high-costs of transit, and began to re-engineer their networks. They would avoid transit fees by

All of these strategies were facilitated by the rise of IXPs; if a provider builds out its network and reached an IXP, it can employ all of those strategies. The economics of the Internet ecosystem would drive evolution and make the network of the 2010s very different from the network of the 1990s.

Public versus private peering

At an IXP, networks can enter into public or private peering arrangements:

Peering Policies & PeeringDB

A "Peering Policy" is "[t]he decision criteria that a provider applies in deciding with whom they will peer. " [NRIC Sec. 1.2.2] In the words of Bill Norton, a "Peering Policy" is "an articulation of peering inclination." [Norton, A Guide to Peering Contracts] [Norton Open Peering Policy] Originally, it was an articulation by a backbone network (the networks in the 1990s that peered) of whom it would perceive to be a "peer" or equal. It is like an amusement park sign that says, "you must be this tall to ride." Generally, to be a "peer," a network had to satisfy the settlement free peering proxy and be roughly equal in size and exchange roughly balanced traffic. A peering policy delineated how that would be measured.[Golding][BEREC p. 21 Dec. 6, 2012][NRIC FG 4]

A peering policy is not a peering contract; peering contracts are far more elaborate. It is not a legal "offer;" networks reserve the right to negotiate and to enter into a peering arrangement or not. [NRIC Sec. 1.2.2] [Norton, Peering Policies]

In the 1990s, as the nascent commercial Internet matured, Tier 1 backbones looked at smaller networks, concluded that they were not "peers," and migrated smaller networks to transit customer arrangements. [1997 Depeering] Smaller networks that found themselves with large transit bills complained. [Digital Handshake 2000] [First 706 Report, para. 105 (at the time, commenters unanimously opposed FCC intervention into peering and interconnection disputes with one exception; Bell Atlantic, now Verizon, recommended possible action by the FCC to lower barriers of entry to new entrants).] If they were not large enough to qualify for peering, then they wanted to know how large they had to be. They wanted to know what benchmarks they had to meet to avoid transit fees.

In 2001, the FCC's Network Reliability and Interoperability Council (NRIC, predecessor of CSRIC), led by Jim Crowe of Level 3, recommended that networks post peering policies on their websites. [NRIC Sec. 4.1] [NRIC Internet Peering Statement ("NRIC V encourages other Internet providers, and especially the large "backbone" Internet providers that comprise the core of the modern Internet, to consider, consistent with their business practices, publication of their criteria for peering.") Available on the Web Archive.] [GAO ("We were also told that peering policies should be made public.")] [FCC NRIC Encourages Publication of Peering Criteria to Promote Transparency (Oct. 30, 2001).] In the 2005 mergers (Verizon/WCOM, AT&T/SBC) and the 2007 merger (AT&T/Bell South), the parties agreed to post peering policies as a merger condition. [SBC / AT&T ¶ 133] [Verizon / MCI ¶ 134] [AT&T / Bell South, Appendix F]

Peering policies consist of two different types of clauses.[Compare Norton Survey (dividing the clauses into three groups: (1) operations-related Internet peering policy clauses; (2) Technical / Routing / Interconnection clauses; and (3) General Clauses).] First, provisions that determine whether a potential partner is a "peer" (roughly equal size with roughly balanced traffic), and, second, operational conditions regarding what is necessary to successfully interconnect. These provisions fall out as follows: 

Peer Criteria (roughly equal size / traffic) 
Operational Conditions 
  • Geographic reach
  • Redundant network
  • Presence at specified IXPs
  • Minimum number of points of interconnection
  • Minimum traffic capacity / utilization
  • Balanced traffic ratio
  • No customers as peers

[Norton, Study of 28 Peering Policies] [Golding] [BEREC p. 21 2012] [NRIC Sec. 4.3 (examples include geographic coverage, proximity to exchange points, minimum capacity, symmetry of traffic exchange, minimum traffic loads, reliable network support, and reasonable address aggregation)] [Verizon (“The key common feature, however, is that these voluntary arrangements involve a mutual exchange of value of one form or another.”)] [Aemen Lodhi, Natalie Larson, Amogh Dhamdhere, Constantine Dovrolis, kc claffy, Using PeeringDB to Understand the Peering Ecosystem, ACM SIGCOMM Computer Communication Review, 44(2), 20-27, 21 (2014), ("Finally, we explore what historical snapshots of the PeeringDB database can tell us about the evolution of the Internet peering ecosystem.")]

[William Norton, Don’t Abuse Peering, DrPeering] [Understanding Default Routes, IS-IS Feature Guide, Juniper Networks (May 20, 2010), ("A default route is the route that takes effect when no other route is available for an IP destination address.")] [Configuring Static Routes, CISCO Nexus 7000 Series NX-OS Unicast Routing Configuration Guide (June 14, 2017)] [Luc De Ghein, Specify a Next Hop Address for Static Routes, CISCO Technical Support, Sept. 2, 2014] [Configuring a Gateway of Last Resort Using IP Commands, CISCO Technical Support, Aug. 10, 2005 ("Default routes are used to direct packets addressed to networks not explicitly listed in the routing table. Default routes are invaluable in topologies where learning all the more specific networks is not desirable, as in case of stub networks, or not feasible due to limited system resources such as memory and processing power.")]

In 2004, PeeringDB came on the scene. PeeringDB is a database created "by and for peering coordinators" that provides 

According to PeeringDB, "The purpose of this project is to facilitate the exchange of information related to peering. Specifically, what networks are peering, where they are peering, and if they are likely to peer with you." [Martin Levy, PeeringDB and why everyone should use it, presentation at African Peering and Interconnection Forum 2011, slide 7] In 2016, PeeringDB 2.0 was launched and PeeringDB was established as a non-profit organization. By the end of 2016, it listed 8194 peering networks, 2302 interconnection facilities, and 566 IXPs.  [Arnold Nipper, PeeringDB, presentation at CEE Peering Days 2017] [Terry Rodery, PeeringDB, presentation at NANOG 40 (2007)] [Aemen Lodhi, Natalie Larson, Amogh Dhamdhere, Constantine Dovrolis, kc claffy, Using PeeringDB to Understand the Peering Ecosystem, ACM SIGCOMM Computer Communication Review, 44(2), 20-27 (2014)]

Generally, networks inclination to peer is divided into four: open, selective, restrictive, and no peering. "Open" is the most liberal inclination where a network is generally willing to peer with anyone who can meet basic requirements (such as meeting the network at an interconnection point). "Selective" is the similar to open, but the selective networks is looking for some basic requirements before they enter into such relationships (for example, minimum traffic utilization). "Restrictive" is the more traditional inclination where a provider is only willing to entering into peering with "peers," networks that meet a relatively high bar of criteria. These inclinations can be regionally specific. A network may have an open policy in one geographic regional and a selective policy in another. 

Providers can have multiple peering policies.  Some providers have distinct business units that have distinct inclinations.  For example, Verizon's CDN Edgecast has a different peering policy than Verizon broadband Internet.  Other companies have multiple policies due to mergers and acquisitions.  For example, Charter Communications acquired Time Warner Cable; both networks continue to have posted peering policies.  Verizon also has distinct posted peering policies for its acquired assets AOL, Yahoo!, and XO.

Who Interconnects with Whom?

See also WHOIS (database of IP address block allocations, revealing what networks hold what address blocks)

See also Statistics | Assessment | Forensics

Augmenting Interconnection Capacity

Networks grow. The Internet grew phenomenally. [See Maj. Joseph Haughney, DCA, ARPANET NEWS from DCA, ANEWS-1 (1 July 1980) (asking ARPANET networks to forecast their growing need for 50/56 kbps circuits from AT&T)]

If settlement-free interconnecting networks grow at roughly the same rate, they continue to be like-networks with roughly balanced traffic ratios, but their interconnection capacity must be augmented.

An industry rule of thumb traditionally has been that when interconnection capacity reached 70 percent utilization, the providers augment capacity. [Estes at 155 ("To compensate for the delay in adding capacity, we were having to order new circuits at about a 65% fill point due to the long lead times that some interexchange carriers had in their order fulfillment processes.")] [AT&T Ex Parte May 26, 2015 at 4 (“Historically, 70 percent utilization was often set as the “trigger” point for augments, to account for the cycle times needed to add capacity before the link became congested.”)] [Amendment No. 3 to the Interconnection Agreement between Verizon New York Inc. and PAETEC Communications, Inc. at 3 (2001) ("The Parties will review all Meet Point B Two-Way Interconnection Trunk groups that reach a utilization level of seventy (70%) percent of the engineered capacity or greater to determine whether those groups should be augmented.") ] [Taylor Level 3 ("Each party pays to augment its own network to allow for more traffic exchange (the expense to augment capacity is not significant for either party). And since we often choose to interconnect in a third party data center, the networks usually agree to share the cost of the cross connects by paying for them on an alternating basis.")] [Netflix Petition to Deny, Dkt 14-57 Att A at 10 (capacity augmented when interconnections reach 70% utilization)] [Cogent Ex Parte MB Docket 14-57 (Nov. 18, 2014)] ("Informal, ad hoc discussions concerning augmentations would occur around the time the links between the network reach 70% utilization, the point at which ISPs though out the industry typically upgrade their interconnections with other networks to avoid service problems.")] [Declaration of Henry (Hank) Kilmer, Cogent, Dkt 14-57 para 16 (Aug. 25, 2014)] [Declaration of Dovrolis, Comcast Sec. 3.2 (Sept. 23, 2014) ("Typically, if the utilization of a link during peak-usage time periods is more than 70%, the link can experience congestion episodes in which traffic is delayed or even dropped.)] In response to congestion, some content networks have included provisions in their peering policies that capacity will be augmented when utilization reaches 50%. [Google Peering Policy ("To ensure sufficient peering capacity in the event of an outage we recommend that peers connect with us at more than one location and augment their peering connections when peak traffic exceeds 50%")] [Microsoft Peering ("Peers are expected to upgrade their ports when peak utilization exceeds 50%")]

Typically, the process of augmenting capacity is short, involving ordering a new cross connect from the IXP. The partners generally take turns covering the costs of installation.

In the case of a customer / vendor relationship, augmentation of capacity is determined by the customer when the customer wants to order additional capacity from the network supplier. For example, an access network customer would ask for additional transit capacity from a backbone service provider.

Multihoming

In order to achieve redundancy and resiliency, transit customers will multihome, purchasing transit service from multiple transit providers. If one transit provider service is disrupted (fiber cuts, peering disputes, infrastructure destruction), the redundant transit service will ensure that the customer remains online. [See Baran Distributed Design] Research has concluded that there is not significant additional benefit to multihoming beyond four upstream providers. [Freedman Slide 4]

Multihoming can be used by transit customers to game the 95th Percentile billing measurement methodology of transit.

The advantage of multihoming can be lost when the strategy provides logically diverse paths (over different Internet service providers) but not physically diverse paths (over the same telecommunications infrastructure service provider) or over the same physical path (through the same conduit).

See 2001 Baltimore Train Tunnel Fire. See also Reliability.

Interconnection Monitoring and Measurement

See also Assessment.

Interconnection can be monitored with protocols like Netflow.

Wikipedia: Netflow is a feature that was introduced on Cisco routers that gives the ability to collect IP network traffic as it enters or exits an interface. By analyzing the data that is provided by Netflow a network administrator can determine things such as the source and destination of the traffic, class of service, and the cause of congestion. Netflow consists of three components: flow caching, Flow Collector, and Data Analyzer.

Netflow reads the routing information and can identify the source ASN (in other words, if the traffic is coming from the Netflix ASN, then the network knows the source is Netflix).  Netflow is also used to validate Level 3's "Bit Mile Routing." An alternative solution is Peakflow by Arbor Networks.

© Cybertelecom ::