- Measuring Broadband America
- Other Governments
- Network Performance Measurement Projects
- See AT&T / DirecTV Merger (measurement interconnection congestion)
Measuring Broadband America MBA
Derived From: FCC MBA: "The Measuring Broadband America (MBA) program is an ongoing nationwide performance study of broadband service in the United States that developed out of a recommendation by the National Broadband Plan to improve the availability of information for consumers about their broadband service." [FCC MBA]
Measuring Fixed Broadband
"Reports offer results of rigorous broadband performance testing for 13 of the largest wireline broadband providers [AT&T; CenturyLink; Charter Communications; Cincinnati Bell; Comcast; Cox Communications; Frontier Communications Company; Hawaiian Telecom; Hughes Network Systems; Mediacom Communications Corporation; Optimum; Verizon; Windstream Communications] that serve well over 80 percent of the U.S. consumer market. Tests conducted used automated, direct measurements of service delivered to the homes of thousands of volunteers across the United States. The Measuring Fixed Broadband studies began in 2011 with the release of annual reports based on data typically collected during a single month with few large-scale traffic events, such as major holidays, sports events or other elections. The data analyzed in the Reports thus reflect stable network conditions that provide the most accurate view of a provider's performance under controlled conditions. The FCC works in collaboration with SamKnows, an international statistics and analytics firm supporting similar projects in other countries around the world. The Measuring Broadband America program incorporates the latest engineering best practices from these diverse stakeholders to collect and report the most accurate data for consumer broadband performance in the Unites States." [FCC MBA]
"The methodologies and assumptions underlying the measurements described in this Report are reviewed at meetings that are open to all interested parties and documented in public ex parte letters filed in the GN Docket No. 12-264. Policy decisions regarding the MBA program were discussed at these meetings prior to adoption, and involved issues such as inclusion of tiers, test periods, mitigation of operational issues affecting the measurement infrastructure, and terms-of-use notifications to panelists. Participation in the MBA program is open and voluntary. Participants include members of academia, consumer equipment vendors, telecommunications vendors, network service providers, consumer policy groups as well as our contractor for this project, SamKnows." [Eight MBA Report at 20]
Measuring Mobile Broadband
"In September 2012, the FCC announced it was expanding the program to include information on mobile broadband service performance in the United States. The program uses the FCC Speed Test app for Android and iPhone devices to test the performance of volunteers' smartphone mobile broadband services. Broadband performance data is being collected that includes upload and download speed, latency and packet loss, as well as the wireless performance characteristics of the broadband connection and the kind of handsets and versions of operating systems tested." [FCC Measuring Mobile Broadband]
Eighth MBA Report: "The measurements that provided the underlying data for this report relied both on measurement clients and measurement servers. The measurement clients (i.e., whiteboxes) resided in the homes of 6,034 panelists... The measurement servers were hosted by M-Lab and Level 3 Communications, and were located in ten cities across the United States near a point of interconnection between the ISP's network and the network on which the measurement server resided. (See Technical Appendix)
"The measurement clients collected data throughout the year. However, only data collected from September 1 through 6 and September 28 through October 21, 2017, referred to throughout this report as the "September 2017" reporting period, were used to generate the charts in this Report. [This data is uploaded from the end user white boxes to samKnows, who then provides the data to the FCC]
"Broadband performance varies with the time of day. At peak hours, more people are attempting to use their broadband Internet connections, giving rise to a greater potential for network congestion and degraded user performance. Unless otherwise stated, this Report focuses on performance during peak usage period, which is defined as weeknights between 7:00 p.m. to 11:00 p.m. local time at the subscriber's location. Focusing on peak usage period provides the most useful information because it demonstrates the performance users can expect when the Internet in their local area is experiencing the highest demand from users.
"Our methodology focuses on the network performance of each of the participating ISPs. The metrics discussed in this Report are derived from traffic flowing between a measurement client, located within the modem or router within a panelist's home, and a measurement server, located outside the ISP's network. For each panelist, the tests automatically choose the measurement server that has the lowest latency to the measurement client. Thus, the metrics measure performance along a path within each ISP's network, through a point of interconnection between the ISP's network and the network on which the chosen measurement server resides.
"However, the service performance that a consumer experiences could differ from our measured values for several reasons:
- MBA measures performance only to a single measurement server rather than to multiple servers across a topological and geographical diversity of interconnections. Some paths or interconnection [links with other networks] may be more degraded than others and this can affect a specific consumer's service.
- Performance beyond an ISP's network (backbone networks, routing, CDNs, host server performance) is not measured in MBA.
- A consumer's home network may be a source of degraded performance. The home network may be shared simultaneously among multiple users and applications within the home. In-home networks, which may includes Wi-Fi, may not have sufficient capacities to support peak loads.[Sundaresan]
- Consumers typically experience performance through applications that they utilize. The overall performance of an application depends on network performance, the application's architecture and implementation, and the operating system and hardware on which it runs.
"This Report is based on the following measurement tests:
- Speed: This test measures the speed (download and upload) of each whitebox over a 10-second period, once per hour during peak hours (7 p.m. to 11 p.m.) and once during each of the following periods: midnight to 6 a.m., 6 a.m. to noon, and noon to 6 p.m. The measurement results from each whitebox are then averaged across the measurement month; and the median value for these average speeds across the entire set of whiteboxes is used to determine the median speed for a service tier. The overall ISP speed is computed as the weighted median for each service tier, using the subscriber counts for the tiers as weights.
- Latency and packet loss: These tests measure the round-trip times for approximately 2,000 packets per hour sent at randomly distributed intervals. Response times less than three seconds are used to determine the mean latency. If the whitebox does not receive a response within three seconds, the packet is counted as lost.
- Web browsing: The web browsing test measures the total time it takes to request and receive webpages, including the text and images, from nine popular websites and is performed once every hour. The measurement includes the time required to translate the web server name (URL) into the webserver's network (IP) address.
[Eighth MBA Report]
Broadband measurement methodology can distort results. Aggregating can mask degraded performance underneath. This problem can emerge topologically (aggregating different networks' traffic together masking issues occurring as a result of negotiations between two networks), chronologically (aggregating different times' traffic together masking peak hour issues with non-peak hour traffic), or geographically (aggregating traffic from diverse locations on the network masking issues at a specific POP).
- FCC Measuring Broadband America webpage [FCC MBA]
- Eight MBA Report, FCC (Dec. 14, 2018)
- Technical Appendix - Eighth Report (PDF)
- Marguerite Reardon, Are you getting the broadband speeds you're paying for?, CNET Nov. 21, 2018 ("News of the report's release comes amid scathing criticism from former FCC officials and public interest groups that the commission didn't publish its 2017 report because it was unhappy with the findings")
- Karl Bode, Former Staffers Say FCC May Be Hiding Data Showing Broadband Industry Problems VICE Motherboard Nov. 20, 2018 ("Since Ajit Pai was appointed boss of the FCC in early 2017, the FCC has yet to release any data from the program. Last year's report wasn't released at all, and the FCC has remained mum on whether it will release any of the program's data this year, either.")
- Jon Brodkin, Ajit Pai isn't saying whether ISPs deliver the broadband speeds you pay for, Ars Technica Nov. 19, 2018 ("the FCC hasn't released any new Measuring Broadband America reports since Republican Ajit Pai became the commission chairman in January 2017. Pai's first year as chair was the first time the FCC failed to issue a new Measuring Broadband America report since the program started")
- Delivering on High Speed Broadband for New York, Charter Policy Blog Dec. 19, 2018 ("Charter, under the brand name Spectrum, delivers over 111% of advertised download speeds, and we deliver those speeds more consistently than any other ISP in the country at 104% of advertised download speed, according to the latest FCC Report")
- Seventh MBA Report, Dec. 14, 2018
- Sixth MBA Report. Dec. 1, 2016
- Measuring Broadband America's 2016 report, Sam Knows 12-07-2016 ("We are extremely proud to announce that the Federal Communications Commission (FCC) has published its Measuring Broadband America (MBA) report, which is entirely based on SamKnows data.")
- John Eggerton, Cable Shines in FCC Broadband Speed Report, MultiChannel News (Dec. 1, 2016)
- Fifth MBA Report. 2015
- 2014 MBA Report, 2013
- Measuring Broadband America 2014 (PPT presentation; no identified author)
- Lily Hay Newman, An FCC Report Says U.S. Internet Should Be Awesome and Reliable, but Don't Get Excited. SLATE June 19, 2014
- James K. Glassman, If it ain't broke… FCC's 'Measuring Broadband America' report shows a healthy Internet sector, AEIdeas June 25, 2014 ("That consumers are trading up to higher speeds is further evidence that they are enthusiastic about having high-quality Internet service – and trust the providers to give them exactly that. That is hardly a marketplace that's crying out for a regulatory fix.")
- Walden, Latta Respond to FCC Report Showing Broadband Speeds are Higher than Ever, House Energy and Commerce Committee June 18, 2014 Archive.org ("This year's report again shows that broadband Internet access is flourishing across the country and consumers continue to reap the benefits of American innovation and investment," said Walden and Latta. "The evidence continues to mount that the administration's call for 'net neutrality' regulations is grounded in finding solutions for problems that don't exist.")
- Ever wonder why your home Internet isn't always as fast as it should be?, Consumer Reports June 19, 2014 ("What the report doesn't address is that these "advertised speeds" are a pretty low bar to hit. U.S. consumers are paying more money for less speed than much of the connected world.")
- 2013 MBA Report, 2013
- 2012 MBA Report, 2012
- 2011 MBA Report, 2011
- FCC Unveils New Research That Measured Broadband Performance. Fact Sheet on first nationwide test of residential wireline broadband service. FACT SHEET CGB TXT
- FCC Chairman Julius Genachowski Remarks On "Measuring Broadband America" Report Best Buy Washington, Dc August 2, 2011, available at Archive.org ("First, we found that most major ISPs are providing service close to what they're advertising. This represents a significant improvement over the findings from two years ago, when we first shone a light on this issue. We also found that, while there are some differences between technologies, DSL, cable, and fiber-to-the-home are all delivering quality service generally consistent with what they advertise. Another finding was that during peak hours--7 to 11 pm--broadband performance generally decreases somewhat. But most services still provide actual speeds that are 80% to 90% of advertised speeds, or better.")
- Sevcik and Wetzel, Assessment of FCC's "Measuring Broadband America" Test Methodology, Network World Nov. 14, 2011 ("we note insufficient transparency in explaining the data analysis methodology and we note inconsistencies between the stated data analysis plan and its execution. More descriptive information is needed to determine and validate how data calculations behind the FCC results were made")
- Concluded "Improving Federal data on the deployment and adoption of broadband service will assist in the development of broadband technology across all regions of the Nation." Sec. 102.
- Amended Sec. 706 authority to be an annual report. Sec. 103(a).
- Improved international broadband reporting. Sec. 103(b).
- Added obligations for the FCC to gather data on the use of broadband service capability. Sec. 103(c)
- Directed the U.S. Census to include questions about broadband. Sec. 103(d). [Subjects Planned for the 2020 Census and American Community Survey, Census p. 25 March 2017 ]
- Called for a GAO Report to conduct a study to consider and evaluate additional broadband metrics or standards. Sec. 104.
- Directed the Small Business Administration to evaluate "the impact of broadband speed and price on small businesses" Sec. 105. [Impact of Broadband Speed and Price on Small Businesses, Columbia Telecommunications Corporation, for the SBA (November 2010)]
- Authorized a new round of BTOP grants. Sec. 106
- Enacted the "Protecting Children in the 21st Century Act". Sec. 201.
Four months later BDIA would be followed by - and perhaps overshadowed by - the American Recovery and Reinvestment Act calling for the National Broadband Plan.
- Broadband Data Improvement Act of 2008, Public Law 110-385, 122 Stat. 4096, Stat 4096 § 103(c)(1), codified at 47 USC 1301-03 (amending Sec. 706 of the Telecommunications Act)
Sec. 102(3): "Improving Federal data on the deployment and adoption of broadband service will assist in the development of broadband technology across all regions of the Nation."
Sec. 103(c) Consumer survey of broadband service capability
(1) In general For the purpose of evaluating, on a statistically significant basis, the national characteristics of the use of broadband service capability, the Commission shall conduct and make public periodic surveys of consumers in urban, suburban, and rural areas in the large business, small business, and residential consumer markets to determine—
(A) the types of technology used to provide the broadband service capability to which consumers subscribe;
(B) the amounts consumers pay per month for such capability;
(C) the actual data transmission speeds of such capability;
(D) the types of applications and services consumers most frequently use in conjunction with such capability;
(E) for consumers who have declined to subscribe to broadband service capability, the reasons given by such consumers for declining such capability;
(F) other sources of broadband service capability which consumers regularly use or on which they rely; and
(G) any other information the Commission deems appropriate for such purpose.
(2) Public availability: The Commission shall make publicly available the results of surveys conducted under this subsection at least once per year.
The data gatherings required by BDIA Sec. 103(c) have been implemented as part of and incorporated into the FCC's Sec. 706 proceedings.
- Report of the Committee on Commerce, Science, and Transportation, on S. 1492 S. Rept. 110-204 Oct. 24, 2007
- "The purpose of S. 1492 is to improve the quality of data collected at State and Federal levels regarding the availability and robustness of broadband services and to promote the deployment of affordable broadband services to all parts of the Nation. " p. 1
- "The lack of comprehensive data regarding the availability and penetration of broadband in the United States has hampered the development of effective policies to promote widespread access to affordable broadband service. "
- Efforts by the Federal Communications Commission (FCC) to measure the speed and quality of broadband deployment across the United States originated after Congress enacted of the Telecommunications Act of 1996. Under section 706 of that Act, the FCC is required to conduct regular inquiries concerning the availability of advanced telecommunications capability and to determine whether advanced telecommunications capability is being deployed to all Americans in a reasonable and timely fashion."" p. 2 "[BDIA] would update section 706 of the Telecommunications Act of 1996..." p. 6
- S. 1492, the Broadband Data Improvement Act, will improve the quality of Federal and State data regarding the availability and deployment of broadband services in order to promote the deployment of affordable broadband services to all parts of the Nation. p. 6
- Announcement of Open Docket For Measuring Broadband America Program, GN Docket No. 12-264 (Sept. 26, 2012) ("the Commission has decided that it is appropriate to make a separate docket for the Program at this time. Previously, Program filings were predominantly submitted to CG Docket No. 09-158, CC Docket No. 98-170, and WC Docket No. 04-36")
- MBA Inquiries CG Docket No. 09-158, CC Docket No. 98-170, WC Docket No. 04-36
- Comment Sought on Residential Fixed Broadband Services Testing and Measurement Solution, Pleading Cycle Established, Public Notice, 25 FCC Rcd 3836 (2010)
- Comment Sought on Measurement of Mobile Broadband Network Performance and Coverage, Public Notice, 25 FCC Rcd 7069 (2010) Fed Reg
- Released: 07/01/2010. FCC Staff To Hold Meeting To Discuss Residential Fixed Broadband Services Testing And Measurement Solution. (DA No. 10-1249). (Dkt No 04-36 09-158 98-179 ). OET .
- FCC Request for Quotation awarded to Sam Knows
- Note that the Consumer Bureau initiated implementation of the NBP MBA recommendation; however, administration of MBA was given to the Office of Engineering and Technology.
- National Broadband Plan
- OBI Technical Paper No. 4: Broadband Performance, Aug 13, 2010 ("The data and analyses in this paper were used to create the National Broadband Availability Target described in the National Broadband Plan (NBP), and therefore represent key inputs into the calculation of the broadband availability gap. The paper also explains how networks have evolved and could evolve over time and what drives network usage and performance demands")
- NBP, p. 64, n. 54 "The FCC, as part of the National Broadband Plan, will issue an RFP to potentially contract with a third party and conduct a six-month consumer panel to gather more detail on actual connection speeds and performance of U.S. broadband services. The results of this panel would ideally have been used as part of the formal report to Congress, as they are critical data points in recommendations, but will now be concluded after the formal report is delivered (for a number of reasons, it was not possible to conduct this panel earlier). Panel results will therefore be finalized after the formal report is delivered, and the FCC will submit results of this panel publicly and to Congress during Fiscal Year 2010 as a supplement to the Plan."
- National Broadband Plan (March 2010)
- RECOMMENDATION 4.3: The FCC, in coordination with the National Institute of Standards and Technology (NIST), should establish technical broadband measurement standards and methodology and a process for updating them. The FCC should also encourage the formation of a partnership of industry and consumer groups to provide input on these standards and this methodology.
- Recommendation 4.4: "The FCC should continue its efforts to measure and publish data on actual performance of fixed broadband services. The FCC should publish a formal report and make the data available online." Discussion: "The FCC should continue its efforts to measure and report on fixed broadband connections and, similar to the approach taken by the United Kingdom regulator (the Office of Communications, or Ofcom), the FCC should explore contracts with third parties as a means of doing so. These measurement efforts would make data on actual performance easily accessible to all interested parties, especially consumers, and create a mechanism for checking service provider broadband performance claims. The FCC should also use these efforts to conduct pilot projects on different measurement and reporting approaches." P. 45
- "The FCC "should collect data that enable more detailed analyses of the market and competition and make that data more publicly available to ensure visibility into competitive behavior of firms." NBP p. 42-43.")
- NBP made its initial MBA findings based on non public data from comScore. NBP p. 64, n. 34.
- Comment Sought on Broadband Measurement and Consumer Transparency of Fixed Residential and Small Business Services in the United States – NBP Public Notice #24, GN Docket No. 09-51, Public Notice, 24 FCC Rcd 14120 (WCB 2009)
- NBP NOI 2009, paras 3, 34, 61: requesting comment on data collection to measure "our nation's progress towards the goal of ensuring that all Americans have access to broadband."
- Note that the NBP contained a recommendation to do MBA; it was not itself authority to do MBA. NBP. Sec. 17.3
- Inquiry Concerning the Deployment of Advanced Telecommunications Capability to All Americans in a Reasonable and Timely Fashion, and Possible Steps to Accelerate Such Deployment Pursuant to Section 706 of the Telecommunications Act of 1996, as Amended by the Broadband Data Improvement Act, Notice of Inquiry, 24 FCC Rcd 10505 (2009). Comment Sought On International Comparison And Consumer Survey Requirements In The Broadband Data Improvement Act, Pleading Cycle Established, GN Docket No. 09-47 (March 31, 2009)
Academic Research using MBA Data / Infrastructure
- MBA Assisted Research Studies
- Srikanth Sundaresan, Nick Feamster, and Renata Teixeira, Home Network or Access Link? Locating Last-mile Downstream Throughput Bottlenecks In T. Karagiannis, & X. Dimitropoulos (Eds.), Passive and Active Measurement: 17th International Conference PAM 2016-Proceedings, Cham et al (pp. 111–123). Springer ("As home networks see increasingly faster downstream throughput speeds, a natural question is whether users are benefiting from these faster speeds or simply facing performance bottlenecks in their own home networks. In this paper, we ask whether downstream throughput bottlenecks occur more frequently in their home networks or in their access ISPs. We identify lightweight metrics that can accurately identify whether a throughput bottleneck lies inside or outside a user’s home network and develop a detection algorithm that locates these bottlenecks. We validate this algorithm in controlled settings and report on two deployments, one of which included 2,652 homes across the United States. We find that wireless bottlenecks are more common than access-link bottlenecks—particularly for home networks with downstream throughput greater than 20 Mbps, where access-link bottlenecks are relatively rare.")
- A. Mandalari, A. Lutu, A. Dhamdhere, M. Bagnulo, and k. claffy, "Tracking the Big NAT across Europe and the U.S.", Tech. rep., Center for Applied Internet Data Analysis (CAIDA), Apr 2017. ("Carrier Grade NAT (CGN) mechanisms enable ISPs to share a single IPv4 address across multiple customers, thus offering an immediate solution to the IPv4 address scarcity problem. In this paper, we perform a large scale active measurement campaign to detect CGNs in fixed broadband networks using NAT Revelio – a tool we have developed and validated. Revelio enables us to actively determine from within residential networks the type of upstream network address translation, namely NAT at the home gateway (customer-grade NAT) or NAT in the ISP (Carrier Grade NAT). We demonstrate the generality of the methodology by deploying Revelio in the FCC Measuring Broadband America testbed operated by SamKnows and also in the RIPE Atlas testbed.")
- Lutu A., Bagnulo M., Dhamdhere A., Claffy K.C. (2016) NAT Revelio: Detecting NAT444 in the ISP. In: Karagiannis T., Dimitropoulos X. (eds) Passive and Active Measurement. PAM 2016. Lecture Notes in Computer Science, vol 9631. Springer, Cham (future work: "we will deploy NAT Revelio in the SamKnows FCC Measuring Broadband America testbed in the US")
- Connect America Fund - Performance Measurements, Appendix A
- Connect America Fund, Order July 6, 2018 Appendix A ("we establish a uniform framework for measuring the speed and latency performance for recipients of high-cost universal service support to serve fixed locations.")
- Connect America Fund reporting requirements, Sam Knows Aug. 28, 2019
- Walter Johnson, Measuring Broadband America NANOG 2014 Slides
- See also Consumer Information and Disclosure NOI 2009
- Bauer, Steven & Lehr, William, 2018. "Measuring Mobile Broadband Performance," 29th European Regional ITS Conference, Trento 2018 184930, International Telecommunications Society (ITS).
- Analyzing data from the FCC’s Measuring Broadband America Report Steven Bauer MIT ITIF 2011 ("Better policy, better news analysis, better broadband")
- USA, actively participate in FCC policy-making!, SamKnows Aug. 5, 2019 ("This year, the FCC and SamKnows are continuing their long-standing partnership with the launch of a new Measuring Broadband America (MBA) volunteer intake - giving you the chance to be at the heart of U.S. policy-making.... Tests include download, upload, latency, packet loss, jitter, and web-browsing.")
- Jon Brodkin, "Lipstick on a pig": Time Warner Cable "deceived the FCC" in speed tests, 2/2/2017, 3:26 PM Ars Technical ("Schneiderman accused TWC of providing customers with older-generation cable modems that couldn't produce the speeds customers paid for. This was a problem in the FCC's testing, so in the summer of 2013, TWC assured the FCC that it would replace the deficient modems for all of its subscribers and would start by replacing the modems of those subscribers involved in the FCC panel. Based on this assurance, the FCC excluded poor results from its annual report.")
- Stacey Higginbotham, There's something rotten in the state of online video streaming, and the data is starting to emerge Gigaom FEB 6, 2014 ("While the initial report published in 2011 showed that some ISPs were delivering sub par speeds versus their advertised speeds, ISPs have since improved their delivery and FCC rankings. ") [Higginbotham 020614]
- Let's Be Clear: ISPs Don't Want Accurate Public Broadband Data - They want public data that paints a rosy picture & keeps government at bay..., dslreports 2/12/2009
- Government Activity
- Measuring Broadband America
- Next-Generation Measurement Architecture Standardization and Outreach Group (NMASOG) Architecture Standards and Specifications, FCC (not officially published / announced) 2012? No Author ("Standardization of this work effort is another step in this policy, ensuring that characterizations of broadband performance are openly specified and widely supported. We expect that such performance data will be used increasingly both to gauge overall progress of broadband evolution in the United States and to support other interests as well. We believe a standardization of both the metrics and collection infrastructure would benefit all in terms both of affording an efficient mechanism for the collection of such information and ensuring that the metrics employed are well founded, providing comparable results across networks and service providers. We therefore ask that the Broadband Forum undertake this proposal.")
- Henning Schulzrinne, FCC CTO and James Miller, FCC Attorney, led an IETF BOF Q4 2012 in Atlanta. [Weil]
- Broadband Data Improvement Act :: The BDIA directed the GAO to "conduct a study to consider and evaluate additional broadband metrics or standards that may be used by industry and the Federal Government to provide users with more accurate information about the cost and capability of their broadband connection, and to better compare the deployment and penetration of broadband in the United States with other countries. " BDIA, Sec. 104. GAO produced a report that largely simply surveyed the field, except for criticizing NTIA's BTOP program for failing to provide guidance on required broadband performance data requirements.
- Current Broadband Measures Have Limitations, and New Measures Are Promising but Need Improvement , GAO-10-49 2009 ("Similarly, FCC does not require broadband providers to include information on actual broadband connection speeds experienced by consumers, although the data from the revised broadband reporting form will provide information on the number of connections by advertised speed. As previously mentioned, actual delivered speed can determine the applications that can be run by consumers and could be useful in comparing broadband service across various segments of the country. The Commission also has open proceedings concerning how the Commission might require broadband service providers to report actual broadband connection speeds, and any alternative means, in addition to or other than requiring such service provider reporting." citing BDIA NOI 2009; NBP.)
- See also NIST Information Security Measurement Standards
- Broadband Forum, TR-304 Broadband Access Service Attributes and Performance Metrics Issue: 1 Issue Date: February 2015 ("there is currently no agreed, widely used test methodology or terminology; this makes meaningful comparison of test data difficult. The goals of this document are therefore to: a Define a standard set of Access Service Attributes that Service Providers may use to characterize their service offerings. These may be used in their own right and/or to determine the impact on customer experience.")
- IETF Large-Scale Measurement of Broadband Performance LMAP "The Large-Scale Measurement of Broadband Performance (LMAP) working group standardizes the LMAP measurement system for performance measurements of broadband access devices such as home and enterprise edge routers, personal computers, mobile devices, set top box, whether wired or wireless."
- Henning Schulzrinne, Re: [lmap] New Non-WG Mailing List: lmap -- Large Scale Measurement of Access network Performance, IETF Mail Archive April 13, 2012 ("this is motivated by the FCC "Measuring Broadband America" effort. The goal is to provide consumers and service providers with better information about broadband (access) performance. Such measurements can inform public policy, facilitate informed consumer choice of broadband providers, and allow debugging of performance issues.")
- RFC 8193 Information Model for Large-Scale Measurement Platforms 2017 ("This Information Model applies to the Measurement Agent within an LMAP framework. As such, it outlines the information that is configured or preconfigured on the Measurement Agent or exists in communications with a Controller or Collector within an LMAP framework. The purpose of such an Information Model is to provide a protocol- and device-independent view of the Measurement Agent that can be implemented via one or more Control and Report Protocols.")
- RFC 7594 A Framework for Large-Scale Measurement of Broadband Performance (LMAP) 2015 ("Measuring broadband service on a large scale requires a description of the logical architecture and standardisation of the key protocols that coordinate interactions between the components. This document presents an overall framework for large-scale measurements.")
- RFC 7536 Large-Scale Measurement of Broadband Performance 2015 ("Measuring broadband performance on a large scale is important for network diagnostics by providers and users, as well as for public policy. Understanding the various scenarios and users of measuring broadband performance is essential to development of the Large-scale Measurement of Broadband Performance (LMAP) framework, information model, and protocol. ")
- Jason Weil, IETF Measurement Standardization, CAIDA WIE 2015
- B. Constantine, R. Krishnan, Traffic Management Benchmarking, IETF RFC 7640 (Sept. 2015)
- P. Chimento, J. Ishac, Defining Network Capacity, IETF RFC 5136 (Feb. 2008)
- IETF RFC 4689, Terminology for Benchmarking Network-layer Traffic Control Mechanisms, Oct. 2006
- Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, "Framework for IP Performance Metrics", RFC 2330, May 1998.
- S Bradner, Editor, Benchmarking Terminology for Network Interconnection Devices, IETF RFC 1242, Sec. 3.17 (July 1991)
- L. Ciavattone ; A. Morton ; G. Ramachandran, Standardized active measurements on a tier 1 IP backbone, IEEE Communications Magazine, Vol. 41, Issue 6 2009
Other Government Measurements
- Measuring Broadband Canada
- Ofcom Research and Data (The Communications Market; Broadband Speeds; Pricing Trends)
- New York Attorney General lawsuit against Time Warner Cable alleging failure to deliver advertised speeds, alleging "Spectrum-TWC Manipulated The FCC's Speed Tests". Complaint p. 45. "A 2013 Spectrum-TWC engineering presentation, which predated the decision to overprovision speeds by 20%, bluntly characterized the over-provisioning maneuver as putting “lipstick on a pig.”" Complaint Para. 218.
Network Performance Measurement Projects
- AppDynamics: real-time monitoring of app performance
- Broadband Internet Service Benchmark BISMark: Broadband Internet Service Benchmark. “a research project and a collaboration effort between Georgia Tech, Princeton University and M-Lab, created to develop an open platform for home broadband internet research. BISmark research is centered but not limited to home network performance measurement (benchmarking) -- appears to have been replaced by Network Microscope
- Catchpoint "world's largest, most reliable monitoring network" 270 nodes in US & Canada. 55 Cities. 36 providers.Backbone. Last Mile. Wireless. Cloud.
- Cloudcheck: "Cloudcheck analyzes the connections within your Wi-Fi and broadband service provider network, using techniques that the world’s largest service providers trust"
- Cyclops UCLA "a network audit tool for service providers and enterprise networks, providing a mechanism to compare the observed behavior of the network and its intended behavior. "
- DSLReports Speed Test
- dynatrace: Cloud Monitoring ( Internet Health Report (Keynote) now leads to dynatrace, as does Internet Weather Report Matrix.net)
- European Internet Exchange Association (showing traffic charts at IXPs)
- GlobalNOC Realtime Atlas (Internet2, NLR, academic)
- Google Analytics: Web analytics that analyzes traffic to a website.
- Google Video Quality Report
- Google Transparency Report
- UCLA Internet Weather Report
- LogicMonitor: infrastructure monitoring and analytics
- MLAB (New America Foundation, Plantlab Consortium, and Google) “M-Lab is a consortium of research, industry and public-interest partners dedicated to: Providing an open, verifiable measurement platform for global network performance Hosting the largest open Internet performance dataset on the planet Creating visualizations and tools to help people make sense of Internet performance
- Netflix Fast.com
- Netflix Speed Index
- Network Diagnostic Tool R. Carlson. (throughput tests) Internet2
- Ookla Speedtest
- Fixed Broadband Speedtest Data, Q2-Q3 2018, Speedtest by Ookla (Dec. 12, 2018),
- perfSONAR ("Performance focused Service Oriented Network monitoring ARchitecture") is open source toolkit for running network tests across multiple domains, and being used extensively by the network research and education community, with over 2,000 deployments as of March, 2018. perfSONAR development and support is led by an international collaboration that includes Internet2, ESnet, Indiana University, the University of Michigan, and the EU funded GN2 JRA1 project.
- RIPE Atlas :: "RIPE Atlas employs a global network of probes that measure Internet connectivity and reachability, providing an unprecedented understanding of the state of the Internet in real time." (ping, traceroute, DNS, SSL Certificates, NTP, HTTP, WiFi)
- Sam Knows
- Thousand Eyes
- Network Monitoring Tools Stanford Linear Acceleator Lab
- Traffic Measurement Tools :: by Henning Schulzrinne Columbia Uni
- Multi Router Traffic Grapher ("The Multi Router Traffic Grapher, or just simply MRTG, is free software for monitoring and measuring the traffic load on network links. It allows the user to see traffic load on a network over time in graphical form")
- Netflow: What traffic is coming from where? ("NetFlow is a feature that was introduced on Cisco routers that provides the ability to collect IP network traffic as it enters or exits an interface. ") See RFC3954
- IPFIX See RFC5101 based on netflow
- JFLOW 'netflow' for Junipers
- Andy Davidson, BGP Traffic Engineering, AFPIC 2013, NANOG
- Causes Degraded Network performance
- Insufficient transmission capacity (edge home, access ISP, core backbone); insufficient interconnection capacity; lack of interconnection; server capacity; client computer capacity; misconfigurations (DNS, BGP); DOS
- Increase capacity, CDN, caching, time-shifting traffic, compression, traffic shaping, traffic engineering, intserv, diffserv, prioritization, resource reservation, adaptive bit rate streaming
- Application tolerance: (delay intolerant voice - jitter intolerant streaming video - loss intolerant web browsing - tolerant email)
- Measurement agent: hardware; downloaded software (app, or webbased)
- Location of agent: on consumer CPE; within home network; Boundary of home/ISP; within carriers network, boundary of ISP and 3rd party network hosting agent
- Hosts: Individual's computer (smartphone, PC, tablet, TV); home AP router; network router, server
- Timing of measurement
- Scheduled or on demand
- Peak and off peak; 24/7
- While edge network is in use by end user?
- Frequency / number of samples
- Number of agents / beacons
- Active or passive
- Passive: collects measurement data from host but does not require special test
- Statistical validity
- Network Anomalies
- Lack of network uniformity (merger of different networks)
- Transparency (methodology, data)
- Looking Glass,
- Trace Route
Bandwidth See Capacity
Buffer / Router Cue:
Capacity: See Traffic Capacity
bandwidth of a link for packets of size k "The capacity, in bits/second, where only those bits of the IP packet are counted, for packets of size k bytes." [RFC 2330 Sec. 7]
IP-layer link capacity, "C(L,T,I), to be the maximum number of IP-layer bits that can be transmitted from the source S and correctly received by the destination D over the link L during the interval [T, T+I], divided by I." [RFC 5136 Sec. 2.3.2]
IP-layer path capacity "simply becomes that of the link with the smallest capacity along that path." [RFC 5136 Sec. 2.3.3]
Nominal Physical Link Capacity, "NomCap(L), is the theoretical maximum amount of data that the link L can support. For example, an OC-3 link would be capable of 155.520 Mbit/s. We stress that this is a measurement at the physical layer and not the network IP layer, which we will define separately."" [RFC 5136 Sec. 2.1]
Throughput (see also Traffic Capacity) "Throughput is the rate at which the traffic can be transported by a specific protocol." [TR-304 Sec. 8.1]. :: "The number of bits, characters, or blocks passing through a data communication system, or portion of that system" [ATIS Telecom Glossary] :: "The maximum rate at which none of the offered frames are dropped by the device." [RFC 1242 Sec. 3.17]
Delay " One-way packet (or frame) delay for a service is defined as the interval between the time that the first bit of a packet ingresses the service at the sending interface and the time that the last bit of the corresponding packet egresses the service at the receiving interface" [TR-304 Sec. 8.2] [ATIS Telecom Glossary] :: "Delay is the time for packets to traverse the network from source to destination. " [Ciavattone]
Frame Loss Rate " Percentage of frames that should have been forwarded by a network device under steady state (constant) load that were not forwarded due to lack of resources." [RFC 1242 Sec. 3.6]
Host "A computer capable of communicating using the Internet protocols; includes "routers"." [RFC 2330 Sec. 5] :: "Almost any kind of computer, including a centralized mainframe that is a host to its terminals, a server that is host to its clients, or a desktop personal computer (PC) that is host to its peripherals. In network architectures, a client station (users machine) is also considered a host because it is a source of information to the network, in contrast to a device, such as a router or switch, that directs traffic. " - Miles Tracy, Wayne Jansen, Karen Scarfone, Theodore Winograd, Guidelines on Securing Public Web Servers, NIST Special Publication 800-44 ver. 2 at B-1 (Sept. 2007)
- NIST, Security Considerations for VoIP Systems 800-58 p. 17 (April 2004)
Jitter refers to non-uniform packet delays. It is often caused by low bandwidth situations in VOIP and can be exceptionally detrimental to the overall QoS. Variations in delays can be more detrimental to QoS than the actual delays themselves . Jitter can cause packets to arrive and be processed out of sequence. RTP, the protocol used to transport voice media, is based on UDP so packets out of order cannot be reassembled at the protocol level. However, RTP allows applications to do the reordering using the sequence number and timestamp fields. The overhead in reassembling these packets is non-trivial, especially when dealing with the tight time constraints of VOIP.
When jitter is high, packets arrive at their destination in spurts. This situation is analogous to uniform road traffic coming to a stoplight. As soon as the stoplight turns green (bandwidth opens up), traffic races through in a clump. The general prescription to control jitter at VOIP endpoints is the use of a buffer, but such a buffer has to release its voice packets at least every 150 ms (usually a lot sooner given the transport delay) so the variations in delay must be bounded. The buffer implementation issue is compounded by the uncertainty of whether a missing packet is simply delayed an anomalously long amount of time, or is actually lost. If jitter is particularly erratic, then the system cannot use past delay times as an indicator for the status of a missing packet. This leaves the system open to implementation specific behavior regarding such a packet.
Jitter can also be controlled at the nexuses of the VOIP network by using routers, firewalls, and other network elements that support QoS. These elements process and pass along time urgent traffic like VOIP packets sooner than less urgent data packets. Unfortunately, not all network components were designed with QoS in mind. An example of a network element that does not implement this QoS demand is a crypto-engine, which ignores Type of Service (ToS) bits in an IP header and other indicators of packet urgency (see 8.7). Another method for reducing delay variation is to pattern network traffic to diminish jitter by making as efficient use of the bandwidth as possible. Unfortunately, this constraint is at odds with some security measures in VOIP. Chief among these is IPsec, whose processing requirements may increase latency, thus limiting effective bandwidth and contributing to jitter. Effective bandwidth is compromised when packets are expanded with new headers. In normal IP traffic, this problem is negligible since the change in the size of the packet is very small compared with the packet size. Because VOIP uses very small packets, even a minimal increase is important because the increase accrues across all the packets, and VOIP sends a very high volume of these small packets.
The window of delivery for a VOIP packet is very small, so it follows that the acceptable variation in packet delay is even smaller. Thus, although we are concerned with security, the utmost care must be given to assuring that delays in packet deliveries caused by security devices are kept uniform throughout the traffic stream. Implementing devices that support QoS and improving the efficiency of bandwidth with header compression allows for more uniform packet delay in a secured VOIP network.
- "The absolute value of the difference between the Forwarding Delay of two consecutive received packets belonging to the same stream." [RFC 4689]
- See ITU-T G. 114 General Recommendations on the transmission quality for an entire international telephone connection (05/2003)
- NIST, Security Considerations for VoIP Systems 800-58 p. 16 (April 2004) "Latency in VOIP refers to the time it takes for a voice transmission to go from its source to its destination. Ideally, we would like to keep latency as low as possible but there are practical lower bounds on the delay of VOIP. The ITU-T Recommendation G.114  set forth a number of time constraints on one-way latency. The upper bound is150 ms. for one-way traffic. This corresponds to the current latency bound experienced in domestic calls across PSTN lines in the continental United States . For international calls, a delay of up to 400 ms. was deemed tolerable , but since most of the added time is spent routing and moving the data over long distances, we consider here only the domestic case and assume our solutions are upwards compatible in the international realm."
- Latency " Forstore and forward devices: The time interval starting when the last bit of the input frame reaches the input port and ending when the first bit of the output frame is seen on the output port. For bit forwarding devices: The time interval starting when the end of the first bit of the input frame reaches the input port and ending when the start of the first bit of the output frame is seen on the output port." [RFC 1242 Sec. 3.8]
Link "a connection between two of these network devices or nodes." [RFC 5136 Sec. 2.1] :: "A single link-level connection between two (or more) hosts; includes leased lines, ethernets, frame relay clouds, etc." [RFC 2330 Sec. 5]
Loss "The packet (or frame) loss ratio is the ratio of total lost packets (or frames) to total transmitted packets (or frames) in a population of interest" [TR-304 Sec. 8.4]
Non Peak Hours: Traffic measurement period outside peak utilization.
- NIST, Security Considerations for VoIP Systems 800-58 p. 18 (April 2004)VOIP is exceptionally intolerant of packet loss. Packet loss can result from excess latency, where a group of packets arrives late and must be discarded in favor of newer ones. It can also be the result of jitter, that is, when a packet arrives after its surrounding packets have been flushed from the buffer, it is useless. VOIP-specific packet loss issues exist in addition to the packet loss issues already associated with data networks; these are the cases where a packet is not delivered at all. Compounding the packet loss problem is VOIP’s reliance on RTP, which is based on the unreliable UDP, and thus does not guarantee packet delivery. Unfortunately, the time constraints do not allow for a reliable protocol such as TCP to be used to deliver media. By the time a packet could be reported missing, retransmitted, and received, the time constraints for QoS would be well exceeded. The good news is that VOIP packets are very small, containing a payload of only 10-50 bytes , which is approximately 12.5-62.5 ms, with most implementations tending toward the shorter range. The loss of such a minuscule amount of speech is not discernable or at least not worthy of complaint for a human VOIP user. The bad news is these packets are usually not lost in isolation. Bandwidth congestion and other such causes of packet loss tend to affect all the packets being delivered around the same time. So although the loss of one packet is fairly inconsequential, probabilistically the loss of one packet means the loss of several packets, which severely degrades the quality of service in a VOIP network.
Path "a path P of length n as a series of links (L1, L2, ..., Ln) connecting a sequence of nodes (N1, N2, ..., Nn+1)." [RFC 5136 Sec. 2.1] :: "A sequence of the form < h0, l1, h1, ..., ln, hn >, where n >= 0, each hi is a host, each li is a link between hi-1 and hi, each h1...hn-1 is a router. A pair
- is termed a 'hop'." [RFC 2330 Sec. 5]
propagation time of a link "The time, in seconds, required by a single bit to travel from the output port on one Internet host across a single link to another Internet host." [RFC 2330 Sec. 7]
Router "A host which facilitates network-level communication between hosts by forwarding IP packets." [RFC 2330 Sec. 5] :: "The path, as defined in Section 5, from A to B at a given time." [RFC 2330 Sec. 7]
Speed: See Traffic Capacity
Utilization: Amount of traffic on a network. May be shown as a percentage of capacity or as Mbps. Measured in samples and may be averaged and shown as peak or non-peak utilization, or utilization over certain amounts of time (i.e., hour, day, week, month). :: "The average usage of a link L, Used(L,T,I), is the actual number of IP-layer bits from any source, correctly received over link L during the interval [T, T+I], divided by I." [RFC 5136 Sec. 2.3.4] :: "We express usage as a fraction of the overall IP-layer link capacity. Util(L,T,I) = ( Used(L,T,I) / C(L,T,I) ) Thus, the utilization now represents the fraction of the capacity that is being used and is a value between zero (meaning nothing is used) and one (meaning the link is fully saturated). Multiplying the utilization by 100 yields the percent utilization of the link." [RFC 5136 Sec. 2.3.5]
- Özgü Alay, et. al., Measuring and assessing mobile broadband networks with MONROE, 2016 IEEE 17th International Symposium on A World of Wireless, Mobile and Multimedia Networks (WoWMoM) 28 July 2016
- BITAG, Report on Real-Time Network Management of Internet Congestion (January 2014)
- Vaibhav Bajpai, et. al., Global Measurements: Practice and Experience (Report on Dagstuhl Seminar #16012), ACM SigComm CCR Online Vo. 46, No. 2 April 2016
- Bauer, Steven and Lehr, William and Mou, Merry, Improving the Measurement and Analysis of Gigabit Broadband Networks (March 31, 2016). Available at SSRN: https://ssrn.com/abstract=2757050 or http://dx.doi.org/10.2139/ssrn.2757050
- Steven Bauer, Massachusetts Institute of Technology, David Clark, Massachusetts Institute of Technology, Bill Lehr - Massachusetts Institute of Technology Understanding Broadband Speed Measurements TPRC 2010
- Dovrolis, C., Ramanathan, P., and D. Moore, "Packet Dispersion Techniques and a Capacity Estimation Methodology", IEEE/ACM Transactions on Networking 12(6): 963-977, December 2004
- William Lehr, Steven Bauer, & David D. Clark. (2013). Measuring Performance when Broadband is the New PSTN. Journal of Information Policy, 3, 411-441. doi:10.5325/jinfopoli.3.2013.0411 ("As broadband becomes the new PSTN, how do we ensure that services are delivered appropriately, and how can this be measured? Doctors Lehr, Bauer, and Clark propose an updated set of measurements and address the policy considerations they raise, in order to ensure that broadband services indeed serve the public interest.") [Lehr 2013]
- William Lehr, Tony Smith-Grievo "Measurement and Assessment of Broadband Availability," report prepared for John Adams Innovation Institute of the Massachusetts Technology Collaborative, January 2009
- "Measuring Broadband: Improving Communications Policymaking through Better Data Collection," with Kenneth Flamm, John Horrigan, and Amy Friedlander, Pew Internet & American Life Project, Washington, DC, November 2007. ( pdf )
- R.S.Prasad, M.Murray, C.Dovrolis, and K.C.Claffy. Bandwidth Estimation: Metrics, Measurement Techniques, and Tools. IEEE Network, 2003 which notes that "Several factors may influence TCP throughput, including transfer size, type of cross traffic (UDP or TCP), number of competing TCP connections, TCP socket buffer sizes at both sender and receiver sides, congestion along the reverse path, as well as the size of router buffers and capacity and load of each link in the network path. Variations in the specification and implementation of TCP, such as NewReno, Reno, or Tahoe, use of selective ACKs (SACKs) vs. cumulative ACKs, selection of the initial window size, and several other parameters also affect TCP throughput.
- S. Sundaresan, W. de Donato, N. Feamster, R. Teixeira, S. Crawford, and A. Pescape. Broadband internet performance: A view from the gateway. Toronto, Ontario, Canada, Aug. 2011.