AI监控反乌托邦：我们正梦游般滑入

Illustration by Chuan Ming Ong for Transformer

There are almost too many ways it could start.

Emboldened ICE agents are sent en masse into another blue city. The White House sends federal agents to “observe” polling stations. Federalized national guards are deployed. In the face of what they see as a life-or-death threat to their neighbors — or to American democracy itself — local residents start to mobilize, as they have in so many cities before.

Cut through the noise.

As public social media accounts post details of protests, these are already being cross-referenced against bulk public datasets — bought, entirely legally, by federal agencies — to identify who is behind them. Connecting your TikTok account to a shop for a discount, or using X to log in to a delivery service just once, is enough: entirely legally, your anonymous social media account is tied to your real identity, and perhaps even your home address. From there, contact networks can be unmasked. This time, though, the federal government has made preparations.

Under the guise of monitoring for foreign interference in US politics, the FBI has been operating hundreds of AI-powered personas, or “bot” accounts, based in different US cities. Having built up a posting history over the course of months, these AI agents easily earn the trust required to get inside the various Signal and Telegram groups used to mobilize resistance, gathering information about meet places, timings and unguarded statements. Usernames and other details are easily matched to real-world identities, including their full online histories, their Amazon purchases, their…everything.

By the time anyone even takes to the streets, the government has the contents of their messages and coordination, and has preloaded facial recognition systems and geo-tagging tools to ping when certain individuals cross a police line, or enter somewhere off-limits. It is ready for mass arrests on all manner of charges: conspiracy, impeding the operations of law enforcement, and whatever it goes on to find. As networks of license-plate-tracking cameras coupled with advanced AI systems, such as those operated by Flock, proliferate, such pursuits only get easier. The kind of peaceful protest against ICE seen in Minneapolis just became impossible.

A Flock safety camera in Burbank, California. Credit: Justin Sullivan/Getty Images

For now, this is a near-future hypothetical, another warning of surveillance dystopia, a refreshed version of 1984 or Minority Report. But this time, there’s a crucial difference: all of the technology to deliver this already exists, and is already in widespread use. Existing large language models would be more than capable of delivering such a system, not just pulling together disparate sources of intelligence, but analyzing them in real-time, “advising” police on who to grab, and where to find them. Those systems will only improve in time. An administration could at least argue — and try its luck through the courts —that existing laws would allow all of this to be done.

Whether we have all-pervasive surveillance or not is now a matter of policy and budget priorities alone. It no longer relies upon development timelines. When a little over a decade ago Edward Snowden revealed the scale of the NSA’s mass collection of data — the reporting of which I worked on for the Guardian — the agency struggled to come up with success stories to justify its program to Congress. All it could show for spending billions of dollars was that the system had helped it catch a single $8,500 donation to the Somali terror group al-Shabab. The aim of Keith Alexander, the NSA director at the time, to “collect it all” ran ahead of the government’s capabilities to analyze that much data in real time.

Over the last few months, I have had conversations with current and former law enforcement staff, former intelligence agents from multiple countries, figures inside the large AI companies and broader technology sector, alongside academics and figures in civil society. Out of that, one clear message has emerged. Thanks to AI, the surveillance dystopia science fiction has spent decades warning us about is already here. We just haven’t noticed yet.

“We’ll enjoy this brief period of endurance of individual liberal rights solely on the basis of the good fortune of incompetence,” says Seth Lazar, a philosopher of the School of Government and Policy at Johns Hopkins University. Lazar, not a natural optimist, suggests we may look back on this as the good times. “The language models are, like, total fucking narcs. They could just as easily be snitches as well.”

The truth is that the specter of mass surveillance as a threat to our civil liberties is the dog that never barked. Snowden upended his life — to this day, he lives in exile in Russia — to reveal the extent of US mass surveillance capabilities. After some initial outrage, however, the world largely shrugged. Cameras have become ubiquitous in London, and life continues largely unchanged. We are mostly desensitized to these capabilities.

With the Snowden revelations, it seemed as if, yes, the NSA was collecting everything they could — but they were just drowning themselves in data. In London, the reason is fairly simple: yes, there are cameras everywhere, but no one is watching what they record, even if we’d like them to (when our bike is stolen, for example).

Most criticisms of mass surveillance have been predicated on it not working as intended. The civil liberties version of the argument goes like this: imagine you’re living in a Western country and you text your friend, who happens to be of Pakistani heritage, that the game this weekend is going to be “THE BOMB.” One misfire of a crude surveillance algorithm later, and you could both be on the wrong side of some aggressive police attention.

The public safety argument is a mirror image of this, first set out to me years ago by the independent surveillance law expert Eric Kind. In most Western countries, the number of people who are seriously planning or considering carrying out a major terror attack in the immediate future is far more likely to be in the hundreds at the most, rather than the thousands. Generally, these people can be identified through old-fashioned methods — human and community intelligence. In the UK, the perpetrator of almost every major terror attack in the last decade was already known to authorities.

Using mass surveillance to tackle this problem is essentially making the haystack ever bigger as you look for a fixed number of needles. All of that changes, though, in a world where AI is smart enough to get it right much more often. How do you argue against an always-on surveillance system that actually works?

Take London: all of the technology for collection is already there. Cameras are everywhere, though mostly not centrally networked. Mobile phone tracking is not just possible, but actively used on the underground system. Technology linking people’s payment cards to their movement already exists. Police already have facial recognition technology, though only use it at present on particular deployments. Even the legal framework to use these new technologies is almost entirely in place, if often untested.

A Metropolitan Police facial recognition van deployed in central London. Credit: Leon Neal/Getty Images

Tying these together into an AI-powered system to — at first, at least — help with the detection of serious crime is a matter of joining up existing systems and getting access to either a data center with the processing power to handle that much information, or an API from one of the big AI providers. The tools are now powerful enough to ingest, sort and join-up this data in a way that only a decade ago was out of reach.