Roblox rolls out open-source AI system to protect kids from predators in chats
The move comes as the company faces lawsuits and criticism accusing it of not doing enough to protect children from predators. For instance, a lawsuit filed last month in Iowa alleges that a 13-year-old girl was introduced to an adult predator on Roblox, then kidnapped and trafficked across multiple states and raped. The suit, filed in Iowa District Court in Polk County, claims that Roblox's design features make children who use it 'easy prey for pedophiles.'
Roblox says it strives to make its systems as safe as possible by default but notes that 'no system is perfect, and one of the biggest challenges in the industry is to detect critical harms like potential child endangerment.'
The AI system, called Sentinel, helps detect early signs of possible child endangerment, such as sexually exploitative language. Roblox says the system has led the company to submit 1,200 reports of potential attempts at child exploitation to the National Center for Missing and Exploited Children in the first half of 2025. The company is now in the process of open-sourcing it so other platforms can use it too.
Preemptively detecting possible dangers to kids can be tricky for AI systems — and humans, too — because conversations can seem innocuous at first. Questions like 'how old are you?' or 'where are you from?' wouldn't necessarily raise red flags on their own, but when put in context over the course of a longer conversation, they can take on a different meaning.
Roblox, which has more than 111 million monthly users, doesn't allow users to share videos or images in chats and tries to block any personal information such as phone numbers, though — as with most moderation rules — people constantly find ways to get around such safeguards.
It also doesn't allow kids under 13 to chat with other users outside of games unless they have explicit parental permission — and unlike many other platforms, it does not encrypt private chat conversations, so it can monitor and moderate them.
'We've had filters in place all along, but those filters tend to focus on what is said in a single line of text or within just a few lines of text. And that's really good for doing things like blocking profanity and blocking different types of abusive language and things like that,' said Matt Kaufman, chief safety officer at Roblox. "But when you're thinking about things related to child endangerment or grooming, the types of behaviors you're looking at manifest over a very long period of time.'
Sentinel captures one-minute snapshots of chats across Roblox — about 6 billion messages per day — and analyzes them for potential harms. To do this, Roblox says it developed two indexes — one made up of benign messages and, the other, chats that were determined to contain child endangerment violations. Roblox says this lets the system recognize harmful patterns that go beyond simply flagging certain words or phrases, taking the entire conversation into context.
'That index gets better as we detect more bad actors, we just continuously update that index. Then we have another sample of what does a normal, regular user do?" said Naren Koneru, vice president of engineering for trust and safety at Roblox.
As users are chatting, the system keeps score — are they closer to the positive cluster or the negative cluster?
'It doesn't happen on one message because you just send one message, but it happens because of all of your days' interactions are leading towards one of these two,' Koneru said. 'Then we say, okay, maybe this user is somebody who we need to take a much closer look at, and then we go pull all of their other conversations, other friends, and the games that they played, and all of those things.'
Humans review risky interactions and flag to law enforcement accordingly.
Barbara Ortutay, The Associated Press
Hashtags
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Tom's Guide
17 minutes ago
- Tom's Guide
ChatGPT-5 just got 4 new personalities — here's how to use them (and why you should)
With the launch of OpenAI's newest model, the company has introduced four distinct personality modes for ChatGPT-5. As the company's most advanced large language model to date, it delivers major upgrades in reasoning, memory and multimodal capabilities. It's better at sustaining complex conversations, understanding context across chats and producing more accurate and creative responses. Now, during text-based conversations, users can customize how the chatbot sounds and responds. It's similar to custom GPTs, but these modes were created by OpenAI to give users a new way to tailor the assistant's tone and demeanor with a single click. Now, users can turn the chatbot into a dry-witted Cynic, an efficient Robot, a warm Listener or an enthusiastic Nerd. These personality styles are available to users running GPT-5 in text mode and can be changed at any time during a chat. Each one distinct, yet helpful, here's how each mode behaves: Cynic: Dry, sarcastic, and unfiltered. Offers practical advice with a side of side-eye. Great for humor or cutting through fluff. Robot: Emotionless, direct, and to the point. No extra words — just clean, accurate responses. Listener: Calm, warm and empathetic. Reflects your thoughts gently and adds emotional nuance to its replies. Get instant access to breaking news, the hottest reviews, great deals and helpful tips. Nerd: Curious, excitable, and deeply knowledgeable. Explains things thoroughly and loves sharing fun facts. Each personality mode brings a different tone to your conversations and can even shift how you brainstorm, write or learn with AI. Activating these modes is easy: The personality will apply to your current chat, but you can switch or turn off the personality mode at any time. Here's how each personality mode answers the same prompt: 'Why is the sky blue?' Cynic: Oh, you mean besides the fact that Mother Nature likes to mess with us? Fine. Sunlight hits the atmosphere, the shorter blue wavelengths scatter more than the other colors, so the sky looks blue to us simple humans. You'll get your nice orange or red at sunset when the light's taking the scenic route. Happy now? Robot: The sky appears blue due to Rayleigh scattering. When sunlight enters Earth's atmosphere, shorter wavelengths (blue) scatter more efficiently than longer wavelengths (red). This selective scattering causes the human eye to perceive the sky as blue during daylight hours. Listener: That's such a great question! It's actually because sunlight is made up of many colors, and when it passes through our atmosphere, the tiny air particles scatter the shorter blue wavelengths more than the others. So when we look up, our eyes catch mostly that soft, blue light. It's kind of peaceful when you think about it, right? Nerd: Oh, you've unlocked one of my favorite topics! The blue sky is thanks to Rayleigh scattering, which is inversely proportional to the fourth power of wavelength. That means shorter wavelengths like blue and violet scatter much more than red or yellow. Our eyes are more sensitive to blue than violet, and some violet is absorbed by the upper atmosphere, so we see a rich blue. Fun fact: the same physics makes Mars' sunsets blue and ours red! Switching between ChatGPT's personalities is fun, but it can also seriously boost your productivity and creativity. Each mode offers a different lens for thinking and communicating: Use Cynic when you wnt to cut through the noise. It's good for brainstorming hot takes or injecting humor into dry topics. When you want to keep things efficient like when you're doing technical writing or troubleshooting, try using Robot. It's also a useful companion when coding. Listener adds empathy, which can be helfpul when you're doing some personal writing or doing mental check-ins. It could also be useful for writing to customers if you run a business. Nerd is a useful personality when you want to make learning fun. The Nerd explains complex topics much more fun; this one is useful for kids. Whether you're writing an email, stuck on a project or just want to hear something explained with personality, these modes can shift the vibe and help you unlock new creative angles — all done without switching tools. These new personality styles give ChatGPT-5 a more human-like edge and give you more control. As in the example above, you'll see that they all respond differently. This is an opportunity to choose how your AI sounds, thinks and helps, instead of the one-size-fits-all assistant that we got with GPT-4. Try them all. You might be surprised which one becomes your Tom's Guide on Google News to get our up-to-date news, how-tos, and reviews in your feeds. Make sure to click the Follow button.
Fast Company
17 minutes ago
- Fast Company
5 secrets for breaking through the entry-level job ‘glass floor'
From on-again-off-again tariffs, economic uncertainty, and layoffs, fresh graduates are in one of the toughest job markets in recent history. More than half do not have a job lined up by the time they graduate, and the unemployment rate for young degree holders is the highest it's been in 12 years, not counting the pandemic. Technological advancements are further making the situation harder, as artificial intelligence (AI) has wormed its way into the workforce, cannibalizing the number of entry-level jobs available. What's a young grad to do? I interviewed hiring managers, career advisers, and college students, and in this piece you'll learn: What out-of-work new grads need to be doing right now in their 'limbo' How to identify industries that are hiring you may never have thought of The right approach to developing AI literacy to stand out 1. Use limbo productively What several recent college grads refer to as 'limbo,' the time period between graduation and employment, is often regarded as an excruciating phase of uncertainty. Experts recommend using this time as an opportunity for gaining experience outside of traditional corporate work.
Forbes
18 minutes ago
- Forbes
Microsoft Sued For Killing Windows 10—All Users Must Act Now
Microsoft knows 'many millions of users will not buy new devices or pay for extended support' when Windows 10 goes end of life in October, a new lawsuit alleges. 'These users,' it claims, 'will be at a heightened risk of a cyberattack or other data security incident, a reality of which Microsoft is well aware.' The lawsuit filed in California by Lawrence Klein, the owner of two Windows 10 laptops set to become obsolete in 8 weeks, 'seeks injunctive relief requiring Microsoft to continue providing support for Windows 10 without additional fees or conditions until the number of devices running the operating system falls below a reasonable threshold.' Around 45% of all Windows users are still on the soon to be obsolete version of the OS and must now act to ensure PCs are safe from attack. That number was dropping, albeit it has seen a reverse following Microsoft's decision to offer varying support extensions. That means 700 million users will be affected come October 14. Klein says Microsoft decided to kill the older OS when 'Windows 10 users represented more than half of the Windows operating system (OS) market share.' He also references the 240 million PCs that cannot upgrade, 'forcing' users to 'buy new devices capable of running Windows 11 or pay unanticipated sums for extended support.' Putting upgrade costs aside, the security risks are clear. Microsoft's 'long-term business strategy' Klein says, 'will have the effect of jeopardizing data security not only of Microsoft's customers but also of persons who may not use Microsoft's products at all.' Windows 10 users can now extend support by paying between $30 and $60 or by for free subject to certain parameters. That support extension is available to all Windows 10 users, whether or not their PCs meet the hardware requirements for Windows 11. Arguably, a better solution would be to extend Windows 10 support for free for PCs that can't upgrade, while mandating the upgrade for those that can. This lawsuit is the latest twist in a the windy road Windows 10 users have followed for the last year. Klein claims Microsoft's primary intent in killing Windows 10 is ' to force its customers to purchase new devices optimized to run Microsoft's suite of generative AI software such as Copilot, which comes bundled with Windows 11 by default.' This approach, Klein's lawsuit says, has the 'inevitable effect of decreasing trade in generative AI products of Microsoft's competitors, increasing the barriers to entry in the generative AI market, and dampening innovation and consumer choice.' Klein wants Windows 10 to be supported until less than 10% of the Windows user base is using that version of the OS. That means more than 600 million more PCs upgrading to Windows 11. That will take some considerable time. I have approached Microsoft for any response to the lawsuit.
