Pinterest Stumbleupon Whatsapp
Ads by Google

What happened when the Internet gets too big for the Internet? The 12th of August saw widespread disruption to Internet users worldwide, as multiple Internet routers fell victim to the a serious problem with how Internet traffic is managed, on a day which has became known as ‘512K Day’.

Affected users saw drastically increased ping times, with many websites failing to load altogether.

The issue – which had been predicted for a long time – was due to the table used for managing how to reach certain IPv4 addresses exceeding their limit of 512,000 routes. This resulted in the older routers that are still used by major ISPs to experience memory overflows and crashes, with users subsequently facing downtime and performance issues as a result.

Affected ISPs – which include BT, Comcast, AT&T, Sprint and Verizon – all reported serious performance issues for some part of Tuesday, with some Web hosting companies being knocked offline altogether.

Curious about how finer details of what went down on ‘512K Day’? Read on for more information.

Border Gateway Protocol and You

When you visit any website, you tend to type in a domain name. These are human-understandable addresses that allow you to access a website without having to manually type an IP address into your Web browser. From there, your computer transforms it into a numeric – or alpha-numeric, in the case of the latest generation of IP addressing IPv6 & The Coming ARPAgeddon [Technology Explained] IPv6 & The Coming ARPAgeddon [Technology Explained] Read More – IP address, which is almost like the phone number of the website you want to visit.

Ads by Google

From there, your computer has to work out how it accesses that website. If you’ve ever used the Traceroute application before, you’ll know that when you visit a website, your computer doesn’t directly access that server. Rather, the packets your computer sends out embark upon an unusual voyage through multiple servers and multiple countries before eventually reaching its destination.

bgp-traceroute

Fortunately, a lot of this is planned in advance. Routes to blocks of IP addresses are stored in their entirety by ISPs in their high-performance routers. These are phenomenally powerful, phenomenally expensive devices. They contain what can best be described as a map of the Internet on their internal storage, and allow home and business users to access the global Internet. These aren’t your garden variety Linksys boxes.

This map of the Internet is stored in what’s called a Border Gateway Protocol (BGP) table. ISPs have always been able to add new routes to the BGP table, which is then shared globally throughout all ISPs. Whenever a new route is added, the shared routing table is automatically updated to reflect that change. This also means that when one party with access to the BGP table makes a mistake, it affects every user.

Perhaps the most notorious example of this was back in 2008, when Pakistan Telecom blocked YouTube in response to a court order. They made a miscalculation with respect to how they were going to block the site, and ended up making a change to the BGP table which propagated worldwide, and ended up blocking it for everyone.

The routers that are used to host the BGP tables have storage space specifically allocated for this purpose. It is measured in terms of routes, with the default limit being artificially established at 512,000 routes for IPv4 addresses, with an additional 512,000 routes for IPv6 addresses IPv6 & The Coming ARPAgeddon [Technology Explained] IPv6 & The Coming ARPAgeddon [Technology Explained] Read More . Although many have predicted the increase of the BGP table exceeding 512K routes for years, we’ve never quite came close to exceeding this limit. The size allocated was more than sufficient. And then suddenly, it wasn’t.

So, what happened?

A few things, really. The first – and most glaringly obvious problem – was with the ISPs themselves. Years of underinvestment had resulted in many running woefully outdated routers. These machines are supposed to be able to handle the traffic of millions of users, and yet found themselves totally unprepared for a much-predicted milestone in the size of the BGP table.

bgp-datacenter

Another issue was with the type of address we use to uniquely identify servers on the Internet. Until recently, we’ve almost exclusively used IPv4 addresses IPv6 vs. IPv4 : Should You Care (Or Do Anything) As A User? [MakeUseOf Explains] IPv6 vs. IPv4 : Should You Care (Or Do Anything) As A User? [MakeUseOf Explains] More recently, there's been a lot of talk about switching to IPv6 and how it will bring a lot of benefits to the Internet. But, this "news" keeps repeating itself, as there's always an occasional... Read More , of which there is a finite amount available. The exhaustion of this pool has been looming over us for years, and we’ve found a number of graceless responses to this problem.

One of the techniques used to mitigate against a shortfall of these addresses was created by the Internet Engineering Task Force (IETF). They aggressively used a technique called Classless Inter-Domain Routing (CIDR), which effectively ‘subnetted’ the IP addressing system, and more efficiently distributed the number of IP addresses available. This helped mitigate the exhaustion of these IP addresses, but this came with other unintended consequences. Namely, the BGP table fragmented, and swollen into a unmanageable amount, bringing 512K day ever closer.

bgp-server

And then, we have to accept that the Internet has been a victim of its own success. More users, more websites and more ISPs have resulted in more routes to map. More routes to map means a larger BGP table. A larger BGP table means… Well, you get the idea.

What’s been done?

To the credit of the ISPs, they resolved the issue phenomenally quickly. In the interim, some effective (albeit ugly) routes were created to ensure the shortest downtime. Artificial limits on the BGP routing table were swiftly increased, and older hardware that physically cannot handle the increased routing table size will be decommissioned, and replaced with newer hardware.

Fingers crossed, we might not have to face another ‘512K Day’ for a long, long while.

Were you impacted by the disruption?

  1. Paul H
    April 16, 2015 at 1:47 am

    Internet issues! I'll give you internet issues! How about slowdowns to dropouts to high pings to random servers I pick. My internet has been doing this for over a year and here's a the kicker, I had AT&T for a year and tried to get out of there near schadel problems and tried Suddenlink and have them for over a year with the Same near schadel problems. And the ISP's tells you nothing that this is a major problem to me and to a lot of other people.

    Suddenlink ISP is using only IP4 only. AT&T ran at IP6 but that didn't stop the ">modem<" from dropping out. And another kicker is every time I call I get an automated call saying the is an outage somewhere every time and only once I got in my nation. OVER a 100 calls just to Suddenlink alone and 100 more to AT&T.

    Check it out at http://www.downdetector.com for any sit to any ISP's and see what I mean. I've been pissed off for over 2 years.

  2. Howard B
    August 17, 2014 at 3:03 am

    "...their limit of 512,000 routes." 512K is NOT 512,000, it's 524,288 (2 to the 19th power).

    • Matthew H
      August 27, 2014 at 11:05 am

      Quite right. Thanks man.

  3. Paul
    August 16, 2014 at 4:37 am

    P.S.

    I hope that the slowdown pissed the NSA off.

  4. Paul
    August 16, 2014 at 4:36 am

    I noticed several slow downs in accessing some sites, and my computer illiterate friend called me yesterday and asked me if I had noticed a slower Internet. I told him crudely what had happened and lost him after the first sentence.
    He'll ask me again tomorrow.
    I live in BC, Canada, on Telus

    • a
      August 16, 2014 at 5:26 am

      Really? I'm on Telus in BC as well and I didn't notice anything.
      Wonder if I wasn't online at quite the right time.

  5. Brian Wisti
    August 15, 2014 at 7:14 pm

    "In the interim, some effective (albeit ugly) routes were created to ensure the shortest downtime."

    Isn't that pretty much the story of modern development?

    • Matthew H
      August 27, 2014 at 11:04 am

      Pretty much, yeah.

Leave a Reply

Your email address will not be published. Required fields are marked *