Marius cursed and jammed a mic stand between the crash bars of the TV studio door. “If SWAT’s on its way, we don’t have much time,” he said.
“I don’t understand.” Michaela, who up until a couple of minutes ago had been streaming their interview live, still sat on one of the oval chairs under the hot lights. “What are they talking about?”
The cube-shaped television studio had black-painted walls surrounding the bright stage area. Big monitors on the walls were showing the same “live” feed as they had five minutes ago, but now a red banner flashed at the bottom of the screens: ACTIVE SHOOTER AT COMPLETE PICTURES BUILDING.
Michaela pointed at a moving figure on the screen. “That looks like you—but—”
Marius nodded. “Uh-huh. Apparently I like assault rifles.”
Adan, their cameraman, had called up a local news feed after the first shouts of panic and confusion filtered through the studio’s thick doors. What it showed was entirely and completely not what the three of them were seeing. Marius was inside the windowless second-floor studio, empty-handed, yet the monitors showed what looked like a drone feed of him moving into and out of view through the building’s windows on the 10th floor. He was armed, and every now and then he would pause and shoot, calmly and methodically.
Marius shook his head in disgust. “Hey, Adan, could you give me a hand with this?”
The cameraman was hunched over his laptop. “Sorry, gotta figure out who’s hijacked our signal.”
“The same people who own the SWAT team,” said Marius. “But forget what I said. I think you’d better get out of here.”
“Look.” Marius pointed at the monitors. They were showing a jumble of witness cell-phone videos. “There!” A jiggly shot showed a man lying in a corridor, dead eyes staring upward, a dark stain on his chest.
“But ...” Adan gaped. “That’s me.”
“Yes. This scene’s not real yet. Listen, Adan, I mean it: you need to leave. The SWAT team’s not on their way here to save any of us. They’re here to make sure that what’s up there”— he pointed—“matches what’s down here.”
Michaela stood up, staring at him. “So it’s real. The antilibrary is real.”
“And soon, the antinet. Michaela, I’m so sorry. I shouldn’t have said anything. I knew they might be watching. Figured they’d be mad if I revealed it, but I never imagined they’d do this.” With the door at least somewhat secure, he went to join Adan at the mixing console. “Any luck?”
Adan shook his head. “I don’t know whether they’re intercepting our feed in here, at the router, or somewhere outside.”
“All this footage,” said Michaela. “It’s being computer-generated in real time? Like you said—by an antilibrary?”
“Yeah.” He smiled ruefully at her. “You got more than you bargained for, I guess. Honestly, I was only going to talk about Augmented Manners. I guess you pushed my buttons, I—”
“That’s okay.” She glanced at the monitors, a little rueful herself. “Digging for the truth is what I do, or used to. Apparently I’m good at it ... What do we do now?”
The TVs showed a black armored personnel carrier plowing up the avenue, with the Complete Pictures building a few blocks ahead of it.
“I think,” Marius said with a grimace, “we’re about to disappear.”
For Michaela, this morning had promised something other than bitter disappointment, for a change. She’d spent the past two years trying to rebuild her career, after her exposé of campaign irregularities in the last election. She’d been sued for libel, doxed, and made a pariah by the winning side, and ultimately lost her job.
Now, after months of taping flower shows and costume conventions for local TV, she’d finally chased down a friend of a friend who knew someone and managed to net an exclusive interview with a rising IT god. Marius Rivas, already a minor celebrity in the hacker world, was young, charismatic, and engaging, not yet rich but certain to be a billionaire in a few years; and he was notoriously hard to find, much less interview. Michaela used his product, Augmented Manners. Hell, almost everybody she knew did—one in 10 American adults, according to the company’s publicity handout. On the street, people were already calling it “the new iPhone,” the indispensable invention of the decade. But traditional media didn’t know what to make of it. Their hesitation had given her the opening she needed.
“Welcome back to the Complete Pictures News Feed. I’m Michaela Kline—yes, that Michaela Kline—and today I’m talking to Marius Rivas, the founder and CEO of Augmented Manners.” She turned to him with a smile. “Marius, you’ve called it ‘the first true political app.’ It’s just a phone app, but it has this magic ability to get people working together. How does it do this?”
Marius grinned and shrugged, a gee-whiz gesture that was going to do great things for his image. “It’s pretty straightforward, actually. You see, when people talk, they often mean entirely different things by the same words. Say, what I mean by ‘liberal’ may be very different from what you mean. Or, say, ‘family values.’ Or even ‘fairness,’ or ‘truth.’ And that’s the problem: people who think they share the same language are talking past one another, because in fact they don’t share an understanding of the part of the language that matters—the connotations of words.
“Augmented Manners uses simultaneous translation technology to translate within the English language, rather than between it and another one. If the app has access to the social-media profile of the person you’re talking to, it can assess pretty well how they express themselves. Using your own voice, it will change trigger words and phrases into more neutral synonyms. Basically, it acts like a translator, or mediator, that helps the other side hear what you mean rather than just what you say.”
“Why do you think it’s caught on so quickly?”
He started talking happily about adoption rates and structured dialogues, and Michaela began to relax.
And maybe it would have all turned out all right if she hadn’t asked Marius where he’d come from.
“The Navy had its own cyber-warfare unit. I did everything I could to get myself transferred to it. I wanted to serve my country in the best way I knew how—by programming. The first project I got put on was for fighting fake news. We were told to figure out a way to defuse it—because it’s propaganda, right? It’s misinformation, or lies that serve the enemy. And I thought, Great! This is what I was meant to do.
“The cyber-war project focused on how to suppress the fake news and amplify the signal of the truth. There were fake-news factories, so one of my first jobs was to find ways to hack into them. I’m pretty good; it worked. We were able to shut down the big ones. But it made no difference. Fake news had gone viral. There were as many sources of it as there were people who doubted Big Media. A whole segment of the population was spamming the net—how do you fight that?
“I started thinking about it differently, focusing not on what fake news is but what it’s for. What’s its product? The answer is simple: doubt. Fake news is designed to sow doubt. I went to my team lead, a senior officer named Cather, and told him that playing whack-a-mole with the sources wasn’t going to work. If fake news is a technology of doubt, we needed to build a technology of trust.
“Cather had a different plan. Why not beat them at their own game? The factories had minuscule resources compared with ours. We had supercomputers that could manufacture disinformation on the fly. Why not use them to spam the spammers?
“The project team was big. Maybe I should have noticed Cather’s hiring practices sooner. I woke up one day and realized I was surrounded by a—well, a certain type of person ...”
“What type?” she asked.
“Intense, quiet young white men. Humorless. Even as civilians they’d buzz-cut their hair. Dress in black. Their e-mail signatures are quotes from Jordan Peterson.”
“Oh,” she said.
“The project was all about defanging fake news. The problem was, these guys couldn’t actually see what was wrong with it to begin with. To them, there was no such thing as real news—never had been. There was only more or less powerful messaging. I’d gone along with Cather’s plan to fight fire with fire because I still thought we were doing it to clear a path for truth. But inch by inch, almost unnoticeably, Cather and his boys steered the project away from using fake news to fight fake news ... toward just making better versions of it.”
He couldn’t seem to stop himself now; he talked in more and more detail about what he’d done, and as he did he got more and more passionate. Michaela was intensely aware of the cameras, and that everything he said was streaming live. He didn’t seem to care—he’d fallen over some cliff of decision, and there was no going back for him.
“Cather rebranded us as Project Antilibrary. And an antilibrary is exactly what we built. We took the newest game engines, which can produce photorealistic video in real time, and we mated them to photo databases of places and people. At first, we used streets and boardrooms in Russia and China. It turned out to be so easy to build the databases, though, that we eventually said, ‘What the hell—let’s do the whole world!’ We built a system that could deepfake live video from anywhere on the planet, of anybody we had a photo or video record of. We could deepfake voices, too—perfectly, of course.
“There were already systems to write fake scientific papers; we improved those so we could generate blog posts, TV news spots read out by America’s favorite announcers, supporting documentation, you name it. I kind of surfed my way through all this, morally. I rode a wave of denial, the way I guess concentration camp guards did.
“Those quiet young men, though ... They saw something in what we were building that I couldn’t even guess at, at the time. By the time I left, we had a tool that could produce a literal library’s worth of entirely bogus material: videos, articles, subtly nonsensical books cross-referenced and supported by other nonsensical books—a true antilibrary—and do it faster than the news cycle, or even social media, could keep up.”
“So you left the Navy?”
He nodded. “I could see where Cather’s work was leading. And I felt, morally, I had to counter it somehow. But we’d worked together, analyzing how disinformation operates. I knew there was no way to attack it directly, so I designed Augmented Manners.
“Can you tell me more about that? What was Augmented Manners supposed to do that would counter fake news? It seems like it’s solving a different problem, the problem of how people communicate—”
“But that’s exactly it. Fake news is designed to break down our ability to trust one another. So I built something that would help you see the real person you were talking to. It would be like your own personal diplomat. You go around, your diplomat talks to other people’s diplomats. It finds common ground, builds bridges, learns what your values are and how to mesh what you want with what other people want. Yes, it really is the first true political app, in the sense that it does what politics is supposed to do: align everybody with collective decisions. It didn’t try to attack the vector of the fake-news disease; it’s designed to immunize us against its effects.”
“And it works! Augmented Manners is the hottest app on three different platforms!”
Marius sighed and shook his head. To Michaela’s alarm, he looked defeated rather than confident. “I’ve been having nightmares ever since we launched Augmented Manners ... I think they’re my subconscious trying to get me to think it through, to recognize where Cather’s really going with his work.”
“Which is?” She glanced uneasily at Adan, who was frowning back from behind the mixing console.
Marius was sitting forward now, hands clasped between his knees. “The antilibrary runs on a cluster of supercomputers. But you know, computers are getting faster and cheaper all the time. And with neural-net and quantum machines, we knew we could double our speed, triple it. I understand the endgame now. It’s to build a system that can tailor every piece of media a person consumes to whatever message you want to convey, on the fly and at any level of resolution. Not just a library’s worth of bogus truth, but any amount, up to and including an internet’s worth. I’ve started calling it the antinet.
“And here’s the thing: he’s built it.”
That’s when the monitors flickered and then showed Marius Rivas kicking back his chair and screaming as he brandished an assault rifle conjured from nowhere.
Marius watched the monitors. They showed police and SWAT pulling up to the building. He eyed the door he’d just barricaded. “Maybe I’m wrong. I think we need to leave.”
“If you’re right, what good will that do?” Adan slumped in his chair. “Even if we go online and wave our shorts at the world, they’ll claim we’re the fake news.”
Marius stepped back. “That’s true—right now. But you know what I said about not being able fight the antinet directly? I ... might be wrong about that.”
Michaela blinked, then looked down from the monitors to him. “What do you mean?”
“When Cather and I were building the antilibrary, we were fixated on politics and society. We looked at how deepfaking could impact elections, opinions, social movements. If we’re lucky, that’s how Cather and his clients have continued to think.”
“You know what groupthink is? When a whole team shares a worldview so intensely that they literally can’t see things—obvious things to you or me—outside it? If we’re lucky ... Put it this way: I was already mostly through the Manners project before some friends who aren’t interested in politics and society pointed out a ... maybe an Achilles’ heel to both the antilibrary and the antinet. Their solution is already out there. I just don’t know whether it’ll reveal itself in time to save us.”
She blinked at him. “What are you talking about?”
He shrugged. “See, if you’re purely political, you might think that there’d be no consequence to faking everybody’s news and social media. You could just steer them like cattle, and that would be that.
“But the whole modern economy runs on accurate data. Without it, production targets are wrong, planes don’t leave on time, things get shipped to wrong addresses. The margins of error are so fine now that, say, failing to predict the exact weather for New York in four days will have an international cascade effect.”
He dug in his pocket for his phone, looked at it, and grimaced. “Do you, do either of you, have a signal?”
They both brought out their smartphones and held them up. They were deep inside the building. “No bars,” said Adan.
“I’ve got one,” said Michaela.
“Can I borrow that?” Marius reached for her phone. Opening its web browser, he tapped in an address. “It’s a contradiction, right?” he muttered as he worked. “The system that’s emerging, at one and the same time it wants to lie to us about everything, even while it’s demanding perfectly accurate data at all scales for it to run at all. Everything from taxi locations to strain-gauge readings in bridge abutments—it all has to be accurate.”
“They’ll be here any second,” said Michaela. “Are we getting out of this place or not?”
He barely heard her. Moving around to look at the phone screen, she saw that he was filling out a long form on a gray web page that had no graphics or title. “Some coders came to me a while back,” he said. “They explained about the Internet of Things blockchain that all these sensors report to. Every sensor puts an encrypted signature on each packet of data it sends, and these are knitted together on a blockchain at the next level, which reports up the line to the next. It’s a chain of provenance, with auditable, publicly available proof at every step that the data hasn’t been tampered with.”
He laughed. “Imagine me when I realized it: we don’t need to build the anti-antinet! It already exists!”
“I don’t understand,” said Michaela hopelessly. “How does that help us?”
“Because if you can build an incorruptible pyramid of facts out of IoT sensor data,” said Marius, “you can do the same thing with journalistic facts.”
“Oh.” Her eyes went wide. “Oh?”
The door rattled as someone or something tried to force it open. “This is the police! Mr. Rivas, open the door and slide out your weapon. You have one minute to comply.”
“They’re not negotiating?” Michaela heard herself say. “Why aren’t they negotiating?”
“Quick!” Adan came out from behind the console and grabbed Michaela’s arm. He hissed under his breath: “I’ve pulled the shelves out of the storage cupboard under the mixer. Maybe you can cram yourself in there—”
“Good idea,” Marius whispered. Then he put his back against the wall, next to the door. “Hey!” he shouted. “How many people do you think are in here?”
There was silence from outside. “You have video of Adan Stokley being killed, right?” he continued. “Well—Adan, can you come over here?” Adan reluctantly went to join him. “Talk to them,” Marius hissed.
“Uh, hi. This is Adan. I’m not dead.”
“Who is this?” barked the voice from outside. “What are you playing at?”
“I’m not a hostage,” Adan shouted. “We’re fine!”
“These friends of mine, they told me they’ve been building a hybrid,” said Marius, speaking barely above a whisper so only Michaela and Adan could hear. He finished entering his data and clicked the old-style OK button at the bottom of the form. Then he blew out a heavy sigh. “Their system lets anybody ask the IoT blockchain for its testimony as an incorruptible witness to anything, provided the data doesn’t violate somebody’s privacy. So, for instance, there are hundreds of diagnostic systems in this room—in the lights, the walls, even the chairs—that report wear and tear up the line to the manufacturers, building owners ... But that data can be used to testify about other things that’re happening in the room. Or aren’t happening. Like, from pressure-wave readings, whether someone has fired a gun in here.”
“Get under here!” As Adan helped Michaela squeeze under the mixing console, she protested to Marius, “These friends of yours, they can prove what really happened here!”
Adan closed the cupboard. Marius came over to it and leaned in toward its door so Michaela could hear. “Once it’s running, the testimonial service can’t be shut down without crashing the economy. But it’s not ready yet; we hadn’t figured out how to make it accessible to everyone. Listen, just now I sent my friends a message. Told them to tune the system to track one particular set of inputs: yours, Michaela. My friends will be able to find you—and to hide you, I hope. I’ve given them as much help as I can. It’s time to hand it over to someone like you who knows how to pursue the truth and tell the world about it.”
The studio door crashed inward and the lights went out, and then it was all chaos and gunfire.
Michaela could barely breathe in the hot and claustrophobically dark cupboard. She heard somebody shout, “Anybody else?”
“Nothing on infrared,” said another voice. Bodies rushed back and forth, and then suddenly everything was silent.
She was too terrified to move, sure she was about to be found or shot through the cupboard doors. The silence stretched out for long seconds, then a minute.
Light reappeared in the crack between the doors.
Michaela pushed, and tumbled out. She stood to find the studio empty, its door hanging open from one hinge.
The chairs were overturned; she moved past them to the door and looked cautiously through it. The outer office, which was usually bustling, was empty, its lights also off. They came on as she stepped out.
She frowned at the humming fluorescents and then went back for her phone, which was lying by one of the chairs. She grabbed her purse from the mixing console. After standing there for a long moment, she called up a local news feed on the phone. It wasn’t hard to find the live feed she was after: the Complete Pictures building was front and center on the main page.
The SWAT team pushed their way out of the building as police built a cordon around the front entrance. In their wake, ambulance attendants were bringing out stretchers. There were three of them.
The sightless eyes of Marius Rivas panned past the camera’s jittery perspective. His limp arm flopped over the side of the stretcher. His shirt was dark with blood. Behind him came Adan, his eyes closed, arms folded across his chest. Michaela sobbed at the sight.
And then her own eyes stared back at her from the third stretcher. For just a moment, she saw herself as others must see her, an image on a screen, the slightly crooked nose bent in the opposite direction from what she saw in the mirror, her hair and shoulders subtly different. Yet this Michaela Kline was clearly dead.
The stretchers moved on and the camera panned to a breathless commentator. Step by step, eyes not really tracking anything, Michaela drifted through the office to the stairs, and down them. Silence on the floor below. She continued past the main floor, headed for the parking garage.
Her car was one flight further down, but some instinct warned her not to go to it. SWAT clearly weren’t watching the news. They hadn’t yet spotted the mismatch in the number of stretchers coming out of the building, but that couldn’t last long. On the landing where she stood was an emergency exit that led to an alley.
She was dead. Everybody knew it. She thought about her life of frustrations, about the empty apartment that only her landlady would care to clear out, about her dead-end job. Even if she cut some sort of deal with this Cather or his masters and miraculously came back to life, what was the point of trying to rebuild her career? It would be a lie, a life spent feeding the antilibrary.
She started to check the broadcast on her phone again, then remembered what Marius had been doing on it. She dug in her purse for a pen and notebook, and wrote down the numbered web address of the site he’d been on. Then she battered the phone until it broke, and pushed open the door to the alley.
There was nobody outside, just drifting plastic bags and the blue hulk of a dumpster. The murmur of a crowd came from the right; Michaela turned left and started walking.
All she had now was an address, but she had been an investigative reporter. There really were facts, there really was a chain of truths, and she would follow that chain—
Until it led her out of the antilibrary.
Karl Schroeder is a futurist and the author of 11 science fiction books. His latest novel is The Million. His next novel, due in 2019, is Stealing Worlds, about the future of work (and burglary) in a near-future, “post-real” America.
This new data poisoning tool lets artists fight back against generative AI
The tool, called Nightshade, messes up training data in ways that could cause serious damage to image-generating AI models.
The Biggest Questions: What is death?
New neuroscience is challenging our understanding of the dying process—bringing opportunities for the living.
Rogue superintelligence and merging with machines: Inside the mind of OpenAI’s chief scientist
An exclusive conversation with Ilya Sutskever on his fears for the future of AI and why they’ve made him change the focus of his life’s work.
How to fix the internet
If we want online discourse to improve, we need to move beyond the big platforms.
Get the latest updates from
MIT Technology Review
Discover special offers, top stories, upcoming events, and more.