GCHQ (Government Communications Headquarters) [Broken URL Removed] is the UK's counterpart of the NSA, responsible for surveillance and intelligence gathering. They operate under the strictest secrecy, with much of their activities kept out of the public domain, and all of their employees having sworn the Official Secrets Act.

But in 2011, Edward Snowden blew the lid on everything. From a hotel room in Hong Kong, and subsequently from a dacha in Moscow's suburbs, he has shone a light on the activities of these intelligence agencies. For the first time ever, we understand how these far-reaching surveillance dragnets (like PRISM) impact us.

Over the past couple of years, Snowden's treasure-trove of secrets have been analyzed and exposed, mostly by First Look Media's The Intercept. They've been drip-feeding us state secrets, in effect, with each revelation as equally earth-shatteringly astonishing as the last.

Last week, Ryan Gallagher writing for The Intercept revealed the existence of a previously-unknown GCHQ program called KARMA POLICE. This initiative records the browsing habits of every user on the Internet, irrespective of whether they live in the US, UK, Europe or elsewhere. This data is then compiled into uniquely-identifying profiles, which can then be queried. The technical details of this are fascinating. The consequences are terrifying.

What Is Karma Police?

The heart of KARMA POLICE (surely named after the song by Radiohead, one of the biggest bands on the planet) is a deeply ambitious plan to capture a record of every single action that ever happens on the Internet, to save it for posterity, and to preserve it in a format that's easily retrievable later on. Unsurprisingly, this produces an eye-watering amount of records. At the time of the Snowden leak, almost 50 billion were being produced per day.

KARMA POLICE collects only metadata, which is essentially "data about data".

It doesn't record, for example, the contents of the email you sent to your grandmother, or your phone call with your doctor. But it does record the related details of each web visit, message sent, or Skype call made. It'll note what webpages you visit, who you call or message, and how long for, all without preserving a record of those events. Although the original messages and webpages aren't being preserved, metadata is still incredibly useful to the intelligence services. A lot can be derived from metadata.

The information caught in KARMA POLICE's dragnet has been used in real-life operations. In 2009, GCHQ undertook operation BLAZING SADDLES. This operation targeted listeners of online radio shows (in particular, Islam-oriented) being broadcast out of Iraq and Egypt), and tried to find trends.

So comprehensive was GCHQ's data gathering infrastructure, they were able to track the browsing habits of one Egyptian listener from their records. They were able to identify that he'd visited the porn site Redtube, as well as Facebook, Yahoo, YouTube, Blogspot, Flickr, a website about Islam, and an Arab advertising site.

So, how was this possible?

Into The Black Hole

The UK's favorable geographic position makes it easy for GCHQ to capture all this information. Great Britain sits between Europe and America, and on the periphery of the continent.

This geography has resulted in 25 percent of all global Internet traffic transiting through the UK, through 1,600 submerged fiber-optic cables that land or connect on the UK's shores. As a result, GCHQ were able to simply "tap" these cables in transit, and take what they need.

To handle the sheer quantities of data, GCHQ created a purpose-built data-storage facility called The Black Hole. According to a report from 2012 that was leaked by Edward Snowden, it contained over 1 trillion records. Since then, it's certainly increased in size exponentially, given the sheer quantities of data that were being collected at the time, and GCHQ expected to eventually collect 100 billion records per day.

When broken down, 41% of the records stored in the Black Hole were simply records of browsing history. The rest were Internet searches, email records, and instant messaging and Voice Over IP (VOIP) conversations. More troublingly, records of anonymized traffic were kept, bringing into question the viability of things like TOR.

A Steaming Bowl of MUTANT BROTH

GCHQ's unholy trinity of surveillance infrastructure consisted of KARMA POLICE for data gathering, The Black Hole for storage, and something called MUTANT BROTH for querying all stored records.

MUTANT BROTH was pretty remarkable, as it wasn't just a "search engine" of people. You could provide it a single point of reference, and it would return a rich library of that person's Internet history. We're not just talking about email addresses and usernames, either. It can even associate an IP address with a human user.

IP addresses are numbers that identify devices on a network, not users. But MUTANT BROTH allows GCHQ spooks to associate IP addresses with records, and then extrapolate who is using that device. But as I alluded to earlier, it wasn't just IP addresses MUTANT BROTH was capable of tracking.

MUTANT BROTH also captured incredible numbers of browser cookies, from sites like Reddit, Hotmail, AOL, as well as broadcasters like Channel 4, BBC and CNN.

Cookies are more than merely used for advertising tracking, or website session management. They frequently contain a broad range of personally identifying information, like login credentials, usernames, and emails. This simply provided an extra point of reference for spooks to search by.

KARMA POLICE, Arrest This Man

Although the NSA bore the brunt of the criticism after Edward Snowden's revelations, the biggest offender was always GCHQ. They were the agency with the most over-reaching programs, and were generally the least-privacy friendly. Even Snowden himself said that GCHQ were worse than the NSA.

Take Tempora, for example. This program saw GCHQ intercept data from submarine fiberoptic cables wholesale, with no distinction between targeted suspects and private individuals. Everything from the contents of telephone calls, to email and Facebook messages and Internet traffic was captured.

We thought that was as bad is it got. But somehow, KARMA POLICE and its associated programs are far worse. They're utterly indiscriminate, and fundamentally undermine any concept of privacy.

Whether the public outcry over it will somehow result in its cancellation remains to be seen, however.