Skip to Content
Artificial intelligence

Bans on deepfakes take us only so far—here’s what we really need

There is a big elephant in the room: outright bans might not even be technically feasible.

February 27, 2024
Taylor Swift and Vladimir Putin in the style of American Gothic
Stephanie Arnett/MITTR | Getty

This story originally appeared in The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here.

There has been some really encouraging news in the fight against deepfakes. A couple of weeks ago the US Federal Trade Commission announced it is finalizing rules banning the use of deepfakes that impersonate people. Leading AI startups and big tech companies also unveiled their voluntary commitments to combatting the deceptive use of AI in 2024 elections. And last Friday, a group of civil society groups, including the Future of Life Institute, SAG-AFTRA, and Encode Justice came out with a new campaign calling for a ban on deepfakes.

These initiatives are a great start and raise public awareness—but the devil will be in the details. Existing rules in the UK and some US states already ban the creation and/or dissemination of deepfakes. The FTC would make it illegal for AI platforms to create content that impersonates people and would allow the agency to force scammers to return the money they made from such scams. 

But there is a big elephant in the room: outright bans might not even be technically feasible. There is no button someone can flick on and off, says Daniel Leufer, a senior policy analyst at the digital rights organization Access Now. 

That is because the genie is out of the bottle. 

Big Tech gets a lot of heat for the harm deepfakes cause, but to their credit, these companies do try to use their content moderation systems to detect and block attempts to generate, say, deepfake porn. (That’s not to say they are perfect. The deepfake porn targeting Taylor Swift reportedly came from a Microsoft system.) 

The bigger problem is that many of the harmful deepfakes come from open-source systems or systems built by state actors, and they are disseminated on end-to-end-encrypted platforms such as Telegram, where they cannot be traced.

Regulation really needs to tackle every actor in the deepfake pipeline, says Leufer. That may mean holding companies big and small accountable for allowing not just the creation of deepfakes but also their spread. So “model marketplaces,” such as Hugging Face or GitHub, may need to be included in talks about regulation to slow the spread of deepfakes. 

These model marketplaces make it easy to access open-source models such as Stable Diffusion, which people can use to build their own deepfake apps. These platforms are already taking action. Hugging Face and GitHub have put into place measures that add friction to the processes people use to access tools and make harmful content. Hugging Face is also a vocal proponent of OpenRAIL licenses, which make users commit to using the models in a certain way. The company also allows people to automatically integrate provenance data that meets high technical standards into their workflow. 

Other popular solutions include better watermarking and content provenance techniques, which would help with detection. But these detection tools are no silver bullet. 

Rules that require all AI-generated content to be watermarked are impossible to enforce, and it’s also highly possible that watermarks could end up doing the opposite of what they’re supposed to do, Leufer says. For one thing, in open-source systems, watermarking and provenance techniques can be removed by bad actors. This is because everyone has access to the model’s source code, so specific users can simply remove any techniques they don’t want.

If only the biggest companies or most popular proprietary platforms offer watermarks on their AI-generated content, then the absence of a watermark could come to signify that content is not AI generated, says Leufer. 

“Enforcing watermarking on all the content that you can enforce it on would actually lend credibility to the most harmful stuff that’s coming from the systems that we can’t intervene in,” he says. 

I asked Leufer if there are any promising approaches he sees out there that give him hope. He paused to think and finally suggested looking at the bigger picture. Deepfakes are just another symptom of the problems we have had with information and disinformation on social media, he said: “This could be the thing that tips the scales to really do something about regulating these platforms and drives a push to really allow for public understanding and transparency.” 

Now read the rest of The Algorithm

Deeper Learning

Watch this robot as it learns to stitch up wounds

An AI-trained surgical robot that can make a few stitches on its own is a small step toward systems that can aid surgeons with such repetitive tasks. A video taken by researchers at the University of California, Berkeley, shows the two-armed robot completing six stitches in a row on a simple wound in imitation skin, passing the needle through the tissue and from one robotic arm to the other while maintaining tension on the thread. 

A helping hand: Though many doctors today get help from robots for procedures ranging from hernia repairs to coronary bypasses, those are used to assist surgeons, not replace them. This new research marks progress toward robots that can operate more autonomously on very intricate, complicated tasks like suturing. The lessons learned in its development could also be useful in other fields of robotics. Read more from James O’Donnell here

Bits and Bytes

Wikimedia’s CTO: In the age of AI, human contributors still matter
Selena Deckelmann argues that in this era of machine-generated content, Wikipedia becomes even more valuable. (MIT Technology Review

Air Canada has to honor a refund policy its chatbot made up
The airline was forced to offer a customer a partial refund after its customer service chatbot inaccurately explained the company’s bereavement travel policy. Expect more cases like this as long as the tech sector sells chatbots that still make things up and have security flaws. (Wired)

Reddit has a new AI training deal to sell user content
The company has struck a $60 million deal to give an unnamed AI company access to the user-created content on its platform. OpenAI and Apple have reportedly been knocking on publishers’ doors trying to strike similar deals. Reddit’s human-written content is a gold mine for AI companies looking for high-quality training data for their language models. (Bloomberg

Google pauses Gemini’s ability to generate AI images of people after diversity errors
It’s no surprise that AI models are biased. I’ve written about how they are outright racist. But  Google’s effort to make its model more inclusive backfired after the model flat-out refused to generate images of white people. (The Verge

ChatGPT goes temporarily “insane” with unexpected outputs, spooking users
Last week, a bug made the popular chatbot produce bizarre and random responses to user queries. (Ars Technica

Deep Dive

Artificial intelligence

How to opt out of Meta’s AI training

Your posts are a gold mine, especially as companies start to run out of AI training data.

Why does AI hallucinate?

The tendency to make things up is holding chatbots back. But that’s just what they do.

Apple is promising personalized AI in a private cloud. Here’s how that will work.

Apple’s first big salvo in the AI wars makes a bet that people will care about data privacy when automating tasks.

This AI-powered “black box” could make surgery safer

A new smart monitoring system could help doctors avoid mistakes—but it’s also alarming some surgeons and leading to sabotage.

Stay connected

Illustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at with a list of newsletters you’d like to receive.