Researchers call for new way of thinking about content moderation

Washington Post

29-04-2025

Happy Tuesday! I'm Jeremy Merrill, stepping in for my colleague Will Oremus on today's Tech Brief. Send news tips to: [e-mail].
Below: Congress passes a revenge porn law, but some advocates are left frustrated. But first:
Researchers call for new way of thinking about content moderation
Facebook's loosening of its content moderation standards early this year got lots of attention and criticism. But a new study suggests that it might matter less what is taken down than when.
The research finds that Facebook posts removed for violating standards or other reasons have already been seen by at least three-quarters of the people who would be predicted to ever see them.
'Content takedowns on Facebook just don't matter all that much, because of how long they take to happen,' said Laura Edelson, an assistant professor of computer science at Northeastern University and the lead author of the paper in the Journal of Online Trust and Safety.
Social media platforms generally measure how many bad posts they have taken down as an indication of their efforts to suppress harmful or illegal material.
The researchers advocate a new metric: How many people were prevented from seeing a bad post by Facebook taking it down?
To measure the 'prevented dissemination' from Facebook's content moderation, the researchers collected more than 1.7 million posts from U.S. news-focused Facebook pages and identified the approximately 13,000 posts of them that had been taken down.
'Removed content we saw was mostly garden-variety spam — ads for financial scams, [multilevel marketing] schemes, that kind of thing,' Edelson said.
The predominance of spam and scam content puts the heated public dispute over content moderation and online censorship in context.
Civil rights groups have criticized Meta, Facebook's owner, for not taking down enough posts. The Anti-Defamation League last year found that only a tiny fraction of antisemitic posts were removed when reported as a regular Facebook user. Leanna Garfield of the LGBTQ+ advocacy organization GLAAD said: 'Meta takes anywhere from several days to weeks to sometimes even months' to review posts with 'anti-LGBTQ slurs and posts promoting violence.'
Before this year's changes, free-expression advocates, in contrast, criticized the company for taking down too much content. 'Arbitrary and unfair enforcement practices … reduce users' confidence both in platforms and in the state of free expression online,' wrote the Foundation for Individual Rights and Expression.
Meta CEO Mark Zuckerberg agreed, saying that while there is 'a lot of legitimately bad stuff out there,' the platform's previous policy caused 'too many mistakes and too much censorship.'
The new research is a reminder that platforms inadvertently host lots of posts that everyone agrees are bad.
The company has said that content that violates its rules is only a tiny fraction of all content on the platform, and that it blocks a lot of bad content before it's successfully posted. It also says it puts substantial effort into stopping fraud and scams.
Edelson and her colleagues at research group Cybersecurity for Democracy focused not on whether a removal was justified, but on when posts were removed. Her group identified all posts from about 10,000 U.S. news pages in July 2023 and collected data on the number of times the posts were liked, commented on or otherwise engaged with, every six hours for two days.
Popular posts got half of their engagement in less than eight hours, they found. But the typical bad post was only removed after more than 20 hours.
After creating a machine learning model that predicts how many engagements a post would likely have after two days, the researchers concluded that Facebook's removals prevented at most only 24 percent of the removed posts' predicted engagement — probably much less. (The paper uses engagement as a proxy for views, because Meta doesn't publish view counts for Facebook posts. Facebook does not disclose why a post was taken down, so it's possible that some of the removed posts were deleted by the poster.)
Facebook has a hard job in finding bad posts amid the sea of acceptable ones, and it's all the harder to do so quickly.
Edelson, who has collaborated with The Washington Post on a project analyzing TikTok users' feeds, suggests a tighter focus would help. Facebook could better predict which posts its algorithms will show to lots of people and have moderators 'prioritize posts with high predicted future views.' That may not be a silver bullet, but it could help Facebook reduce the amount of bad content it shows to users.
Congress passes a revenge porn law, but some advocates are left frustrated
The internet is about to get a new federal online safety law, as your usual host Will Oremus reported Monday. But not all online safety advocates are overjoyed.
The House of Representatives voted 409-2 on Monday evening to pass the Take It Down Act, which President Donald Trump has already indicated he plans to sign. The act criminalizes the publication of nonconsensual intimate imagery, or NCII, including revenge porn and AI deepfake nudes, and requires online platforms to take it down within 48 hours of a valid report.
The vote followed the bill's unanimous passage in February in the Senate, where it was coauthored by Sens. Ted Cruz (R-Texas) and Amy Klobuchar (D-Minnesota). There was no debate on the House floor, where it passed under an expedited process called suspension of rules that requires a two-thirds majority.
The bill had support from an unusually wide swath of the political spectrum, from First Lady Melania Trump to Columbia Law Professor Tim Wu. And it thrilled advocates who have experienced the nightmare of NCII themselves, among them Elliston Berry, who was 14 when classmates distributed deepfake nudes of her on Snapchat.
'With the passage of the TAKE IT DOWN Act, we can protect future generations from having to experience the pain I went through,' Berry said in a statement.
Still, a contingent of tech policy wonks, including free expression advocates and digital rights groups, was left frustrated. Among them was Mary Anne Franks, president of the Cyber Civil Rights Initiative (CCRI), who has long been a leading advocate of a federal law criminalizing revenge porn.
In a statement Monday, CCRI called the criminalization of revenge porn 'long overdue' but said it has 'serious concerns about the constitutionality, efficacy, and potential misuse' of the provision in the Take It Down Act that requires online platforms to remove reported content within 48 hours. Those provisions, the group argued, are 'likely to be selectively and improperly misused for political or ideological purposes that endanger the very communities most affected by image-based sexual abuse.'
Wikipedia's nonprofit status questioned by D.C. U.S. attorney (Will Oremus and Julian Mark)
Government customer service shake ups have the less tech-savvy on edge (Heather Kelly)
Elon Musk had the government in his grasp. Then it unraveled. (Dan Diamond, Faiz Siddiqui, Trisha Thadani and Jeff Stein)
Tech tips for Defense Secretary Pete Hegseth (and everyone else) (Heather Kelly)
Congress passes Take It Down Act to fight deepfake nudes, revenge porn (Will Oremus)
Critics fear the Trump administration could weaponize the Take It Down Act (The Verge)
Meta's 'Digital Companions' Will Talk Sex With Users—Even Children (Wall Street Journal)
Researchers secretly ran a massive, unauthorized AI persuasion experiment on Reddit users (404 Media)
Wall Street banks sell final slug of Elon Musk's X debt (Wall Street Journal)
China's Huawei develops new AI chip, seeking to match Nvidia (Wall Street Journal)
ChatGPT goes shopping with new product-browsing feature (Ars Technica)
Duolingo will replace contract workers with AI (The Verge)
These autistic people struggled to make sense of others. Then they found AI. (Andrea Jiménez)
The group chats that changed America (Semafor)
That's all for today — thank you so much for joining us! Make sure to tell others to subscribe to the Tech Brief. Get in touch with Will (via email or social media) for tips, feedback or greetings!