MIT Technology Review Subscribe

Troll farms reached 140 million Americans a month on Facebook before 2020 election, internal report shows

“This is not normal. This is not healthy.”

Conceptual illustration showing facebook likes with a glitch effect

Update: After this story published, Facebook took down two of the five troll-farm pages we identified.

In the run-up to the 2020 election, the most highly contested in US history, Facebook’s most popular pages for Christian and Black American content were being run by Eastern European troll farms. These pages were part of a larger network that collectively reached nearly half of all Americans, according to an internal company report, and achieved that reach not through user choice but primarily as a result of Facebook’s own platform design and engagement-hungry algorithm.

Advertisement

The report, written in October 2019 and obtained by MIT Technology Review from a former Facebook employee not involved in researching it, found that after the 2016 election, Facebook failed to prioritize fundamental changes to how its platform promotes and distributes information. The company instead pursued a whack-a-mole strategy that involved monitoring and quashing the activity of bad actors when they engaged in political discourse, and adding some guardrails that prevented “the worst of the worst.”

This story is only available to subscribers.

Don’t settle for half the story.
Get paywall-free access to technology news for the here and now.

Subscribe now Already a subscriber? Sign in
You’ve read all your free stories.

MIT Technology Review provides an intelligent and independent filter for the flood of information about technology.

Subscribe now Already a subscriber? Sign in

But this approach did little to stem the underlying problem, the report noted. Troll farms—professionalized groups that work in a coordinated fashion to post provocative content, often propaganda, to social networks—were still building massive audiences by running networks of Facebook pages. Their content was reaching 140 million US users per month—75% of whom had never followed any of the pages. They were seeing the content because Facebook’s content-recommendation system had pushed it into their news feeds.

“Instead of users choosing to receive content from these actors, it is our platform that is choosing to give [these troll farms] an enormous reach,” wrote the report’s author, Jeff Allen, a former senior-level data scientist at Facebook.

Joe Osborne, a Facebook spokesperson, said in a statement that the company “had already been investigating these topics” at the time of Allen’s report, adding: “Since that time, we have stood up teams, developed new policies, and collaborated with industry peers to address these networks. We’ve taken aggressive enforcement actions against these kinds of foreign and domestic inauthentic groups and have shared the results publicly on a quarterly basis.”

In the process of fact-checking this story shortly before publication, MIT Technology Review found that five of the troll-farm pages mentioned in the report remained active.

This is the largest troll-farm page targeting African-Americans in October 2019. It still remains active on Facebook.

The report found that troll farms were reaching the same demographic groups singled out by the Kremlin-backed Internet Research Agency (IRA) during the 2016 election, which had targeted Christians, Black Americans, and Native Americans. A 2018 BuzzFeed News investigation found that at least one member of the Russian IRA, indicted for alleged interference in the 2016 US election, had also visited Macedonia around the emergence of its first troll farms, though it didn’t find concrete evidence of a connection. (Facebook said its investigations hadn’t turned up a connection between the IRA and Macedonian troll farms either.)

“This is not normal. This is not healthy,” Allen wrote. “We have empowered inauthentic actors to accumulate huge followings for largely unknown purposes … The fact that actors with possible ties to the IRA have access to huge audience numbers in the same demographic groups targeted by the IRA poses an enormous risk to the US 2020 election.”

As long as troll farms found success in using these tactics, any other bad actor could too, he continued: “If the Troll Farms are reaching 30M US users with content targeted to African Americans, we should not at all be surprised if we discover the IRA also currently has large audiences there.”

Advertisement

Allen wrote the report as the fourth and final installment of a year-and-a-half-long effort to understand troll farms. He left the company that same month, in part because of frustration that leadership had “effectively ignored” his research, according to the former Facebook employee who supplied the report. Allen declined to comment.

The report reveals the alarming state of affairs in which Facebook leadership left the platform for years, despite repeated public promises to aggressively tackle foreign-based election interference. MIT Technology Review is making the full report available, with employee names redacted, because it is in the public interest.

Its revelations include:

How Facebook enables troll farms and grows their audiences

The report looks specifically at troll farms based in Kosovo and Macedonia, which are run by people who don’t necessarily understand American politics. Yet because of the way Facebook’s newsfeed reward systems are designed, they can still have a significant impact on political discourse.

In the report, Allen identifies three reasons why these pages are able to gain such large audiences. First, Facebook doesn’t penalize pages for posting completely unoriginal content. If something has previously gone viral, it will likely go viral again when posted a second time. This makes it really easy for anyone to build a massive following among Black Americans, for example. Bad actors can simply copy viral content from Black Americans’ pages, or even Reddit and Twitter, and paste it onto their own page—or sometimes dozens of pages.

Second, Facebook pushes engaging content on pages to people who don’t follow them. When users’ friends comment on or reshare posts on one of these pages, those users will see it in their newsfeeds too. The more a page’s content is commented on or shared, the more it travels beyond its followers. This means troll farms, whose strategy centers on reposting the most engaging content, have an outsize ability to reach new audiences.

Third, Facebook’s ranking system pushes more engaging content higher up in users’ newsfeeds. For the most part, the people who run troll farms have financial rather than political motives; they post whatever receives the most engagement, with little regard to the actual content. But because misinformation, clickbait, and politically divisive content is more likely to receive high engagement (as Facebook’s own internal analyses acknowledge), troll farms gravitate to posting more of it over time, the report says.

As a result, in October 2019, all 15 of the top pages targeting Christian Americans, 10 of the top 15 Facebook pages targeting Black Americans, and four of the top 12 Facebook pages targeting Native Americans were being run by troll farms.

“Our platform has given the largest voice in the Christian American community to a handful of bad actors, who, based on their media production practices, have never been to church,” Allen wrote. “Our platform has given the largest voice in the African American community to a handful of bad actors, who, based on their media production practices, have never had an interaction with an African American.”

Advertisement

“It will always strike me as profoundly weird ... and genuinely horrifying,” he wrote. “It seems quite clear that until that situation can be fixed, we will always be feeling serious headwinds in trying to accomplish our mission.”

The report also suggested a possible solution. “This is far from the first time humanity has fought bad actors in our media ecosystems,” he wrote, pointing to Google’s use of what’s known as a graph-based authority measure—which assesses the quality of a web page according to how often it cites and is cited by other quality web pages—to demote bad actors in its search rankings.

“We have our own implementation of a graph-based authority measure,” he continued. If the platform gave more consideration to this existing metric in ranking pages, it could help flip the disturbing trend in which pages reach the widest audiences.

When Facebook’s rankings prioritize engagement, troll-farm pages beat out authentic pages, Allen wrote. But “90% of Troll Farm Pages have exactly 0 Graph Authority … [Authentic pages] clearly win.”

Systemic issues

A search of all the troll-farm pages listed in the report reveals that five are still active nearly two years later:

A troll-farm page targeting Christian Americans.

Facebook’s recent controversial “Widely Viewed Content” report suggests that some of the core vulnerabilities the troll farms exploited also remain. Fifteen of the 19 most viewed posts listed in the report were plagiarized from other posts that had previously gone viral on Facebook or another platform, according to an analysis from Casey Newton at The Verge.

Samantha Bradshaw, a postdoctoral research fellow at Stanford University who studies the intersection of disinformation, social media, and democracy, says the report “speaks to a lot of the deeper systemic problems with the platform and their algorithm in the way that they promote certain kinds of content to certain users, all just based on this underlying value of growth.” If those are not fixed, they will continue to create distorted, economic incentives for bad actors, she adds: “That’s the problem.”

Read the full report here:

1 free story remaining
This is your last free story.
Sign in Subscribe now

Your daily newsletter about what’s up in emerging technology from MIT Technology Review.

Please, enter a valid email.
Privacy Policy
Submitting...
There was an error submitting the request.
Thanks for signing up!

Our most popular stories

Advertisement