Roblox rolls out open-source AI system to protect kids from predators in chats
The move comes as the company faces lawsuits and criticism accusing it of not doing enough to protect children from predators. For instance, a lawsuit filed last month in Iowa alleges that a 13-year-old girl was introduced to an adult predator on Roblox, then kidnapped and trafficked across multiple states and raped. The suit, filed in Iowa District Court in Polk County, claims that Roblox's design features make children who use it 'easy prey for pedophiles.'
Roblox says it strives to make its systems as safe as possible by default but notes that 'no system is perfect, and one of the biggest challenges in the industry is to detect critical harms like potential child endangerment.'
The AI system, called Sentinel, helps detect early signs of possible child endangerment, such as sexually exploitative language. Roblox says the system has led the company to submit 1,200 reports of potential attempts at child exploitation to the National Center for Missing and Exploited Children in the first half of 2025. The company is now in the process of open-sourcing it so other platforms can use it too.
Preemptively detecting possible dangers to kids can be tricky for AI systems — and humans, too — because conversations can seem innocuous at first. Questions like 'how old are you?' or 'where are you from?' wouldn't necessarily raise red flags on their own, but when put in context over the course of a longer conversation, they can take on a different meaning.
Roblox, which has more than 111 million monthly users, doesn't allow users to share videos or images in chats and tries to block any personal information such as phone numbers, though — as with most moderation rules — people constantly find ways to get around such safeguards.
It also doesn't allow kids under 13 to chat with other users outside of games unless they have explicit parental permission — and unlike many other platforms, it does not encrypt private chat conversations, so it can monitor and moderate them.
'We've had filters in place all along, but those filters tend to focus on what is said in a single line of text or within just a few lines of text. And that's really good for doing things like blocking profanity and blocking different types of abusive language and things like that,' said Matt Kaufman, chief safety officer at Roblox. 'But when you're thinking about things related to child endangerment or grooming, the types of behaviors you're looking at manifest over a very long period of time.'
Sentinel captures one-minute snapshots of chats across Roblox — about 6 billion messages per day — and analyzes them for potential harms. To do this, Roblox says it developed two indexes — one made up of benign messages and, the other, chats that were determined to contain child endangerment violations. Roblox says this lets the system recognize harmful patterns that go beyond simply flagging certain words or phrases, taking the entire conversation into context.
'That index gets better as we detect more bad actors, we just continuously update that index. Then we have another sample of what does a normal, regular user do?' said Naren Koneru, vice president of engineering for trust and safety at Roblox.
As users are chatting, the system keeps score — are they closer to the positive cluster or the negative cluster?
'It doesn't happen on one message because you just send one message, but it happens because of all of your days' interactions are leading towards one of these two,' Koneru said. 'Then we say, okay, maybe this user is somebody who we need to take a much closer look at, and then we go pull all of their other conversations, other friends, and the games that they played, and all of those things.'
Humans review risky interactions and flag to law enforcement accordingly.
Barbara Ortutay, The Associated Press
Hashtags
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Globe and Mail
4 hours ago
- Globe and Mail
2 Surefire Stocks to Invest $1,000 in for the Long Haul
Key Points Alphabet can overcome an antitrust challenge thanks to strong growth prospects and substantial cash flow. Intuitive Surgical leads its niche of the medical device market and still has ample whitespace ahead. 10 stocks we like better than Alphabet › Investing in stocks is both accessible to the general public and likely to yield strong returns over the long term. That's what makes it one of the best ways to increase your wealth. However, putting your money in the right stocks is crucial. Some companies may turn out to be terrible investments, as they significantly lag behind the market and may even cease to exist. Investors can avoid that fate by investing in ETFs that track major indexes or by selecting stocks that have the potential to outperform broader equities over extended periods. These are typically well-established, highly profitable companies with solid businesses, excellent prospects, and strong competitive edges. Here are two brilliant examples: Alphabet (NASDAQ: GOOG)(NASDAQ: GOOGL), and Intuitive Surgical (NASDAQ: ISRG). Here's why putting $1,000 into either of those stocks would be a great move. 1. Alphabet Alphabet has been one of the more undervalued among the trillion-dollar tech leaders of late. In fairness, that's because it faces some potential headwinds related to antitrust lawsuits in the U.S. Regulators want the company to relinquish its Chrome browser, a key component of its advertising business. While that's worth keeping an eye on, Alphabet continues to impress with its financial results. The company's second-quarter earnings were no different. The tech giant grew its revenue by 14% year over year to $96.4 billion, while its net earnings per share (EPS) of $2.31 were up 22.2% compared to the year-ago period. Some of Alphabet's current growth drivers, including cloud computing and artificial intelligence (AI), should remain so for many years to come. The company pointed out that its cloud division, Google Cloud, now has a revenue run rate of more than $50 billion. Though Alphabet's advertising business remains its main cash cow, cloud and AI could take over in the next decade. Investors should be excited about that since it will mean that the company will become far less reliant on its advertising business, which would take the biggest hit if Alphabet loses Chrome. So, the more diversified the business becomes, the less of a risk this potential worst-case scenario of losing Chrome is. Furthermore, Alphabet is exploring other potential opportunities, including streaming through YouTube, a leader in that niche, and self-driving cars through its subsidiary, Waymo. Lastly, Alphabet has a strong competitive edge from multiple sources, including Google's network effect and strong brand name, as well as the switching costs associated with its cloud solutions. Alphabet is firing on all cylinders, and the company's long-term prospects remain bright. Even if regulators in the U.S. succeed in forcing the company to relinquish its Chrome browser, Alphabet should recover due to having multiple lucrative opportunities and significant amounts of free cash flow -- to the tune of $5.3 billion as of the end of the second quarter. That's significantly lower than the $13.5 billion it had as of the end of Q2 2024, but that's partly because Alphabet is reinvesting substantial amounts, including a projected $85 billion this year, into capital expenditures to support various growth opportunities. This investment could pay for itself several times over in the long run as the company makes further progress in cloud, AI, and other areas. Alphabet remains well-positioned to outperform the market over the next decade. Investors can get five of the company's shares for $1,000. 2. Intuitive Surgical Intuitive Surgical, the leader in the robotic-assisted surgery (RAS) market, has struggled this year. Trump's aggressive tariffs could take a bite out of the company's earnings, which is why many investors are choosing to stay away from the stock. However, the medical device specialist's outlook is still strong, at least for investors willing to be patient. For one, Intuitive Surgical continues to record excellent financial results. The company's second-quarter revenue was $2.44 billion, 21% higher than the year-ago period, thanks to a 17% year-over-year increase in the number of procedures performed with its crown jewel, the Da Vinci System. Intuitive Surgical's adjusted EPS came in at $2.19, 23% higher than Q2 2024. The company has had little competition in the RAS field, and although that's about to change, several factors should allow it to perform well over the next decade. First, it will be challenging for newcomers to catch up, as Intuitive Surgical has been earning indications for the Da Vinci System for years. It is approved for use in general surgeries, bariatric procedures, urologic procedures, gynecologic procedures, and more. Second, since the Da Vinci System is expensive and requires a learning curve, after putting considerable time and money into acquiring one and training medical staff on it, healthcare facilities will need a lot of convincing to switch to a competitor. In other words, Intuitive Surgical benefits from high switching costs. Third, the company is an innovator. It earned clearance for the fifth generation of its crown jewel last year and will undoubtedly continue to improve the device. Lastly, there is significant whitespace in the RAS market. Intuitive Surgical will benefit from the world's aging population, as older individuals are more likely to require many of the procedures performed with its Da Vinci system. Even with the current volume of surgeries, RAS procedures have captured only 5% of those eligible for robotic surgery. That points to a massive opportunity, and as a leader in the market, Intuitive Surgical should benefit. The stock may be down this year, but it could deliver superior returns over the long run. With $1,000, investors can purchase two shares. Should you invest $1,000 in Alphabet right now? Before you buy stock in Alphabet, consider this: The Motley Fool Stock Advisor analyst team just identified what they believe are the 10 best stocks for investors to buy now… and Alphabet wasn't one of them. The 10 stocks that made the cut could produce monster returns in the coming years. Consider when Netflix made this list on December 17, 2004... if you invested $1,000 at the time of our recommendation, you'd have $653,427!* Or when Nvidia made this list on April 15, 2005... if you invested $1,000 at the time of our recommendation, you'd have $1,119,863!* Now, it's worth noting Stock Advisor's total average return is 1,060% — a market-crushing outperformance compared to 182% for the S&P 500. Don't miss out on the latest top 10 list, available when you join Stock Advisor. See the 10 stocks » *Stock Advisor returns as of August 4, 2025
CBC
4 hours ago
- CBC
Regina couple's app teaches mindfulness to break the cycle of addiction
It began with moments of personal reflection. Adam Geiger had been battling a gambling addiction since he was a teenager. "I think it started innocently enough, with things like video games and whatnot," Geiger said. "Looking back, I remember myself being very anxious, always really trapped in my mind thinking of what other people thought of me." After decades of struggle, mindfulness and meditation helped him shift gears, allowing him to explore not just his behaviour, but the thoughts beneath it. "Looking at the nature of thought and what was going on, sort of beneath the surface level stuff," he said. In 2024, he and his partner Chelsea Galloway two decided to build a digital tool to support others facing similar struggles. "I think Adam and I both had this opportunity at the time to really be able to put ourselves into something that mattered to us personally," Galloway said. A year and a half later, their vision became reality with the launch of AlchemistOne, a mindfulness-focused recovery app designed to support people dealing with addictions. A shared mission born from experience Geiger's journey to AlchemistOne began decades ago, when an innocent love of video games and sports grew to compulsive gambling. "I made my first sports bet when I was 13 or 14," he said. "All those thoughts went away and it was very easy for me to escape into gambling." Temporary relief came with long-term consequences. Geiger said he spent more 20 years locked in a cycle of gambling addiction. Once he finally broke that cycle, the app seemed like a perfect opportunity to help others do the same. Galloway, the company's COO, brought both personal insight and business expertise to the project. "Addiction was just something that was really present in our lives," she said. "Adam had a really strong tech background. I had a pretty strong business background. So we came together to build the company." The pair spent months designing what would become AlchemistOne. What started as a two-person initiative now includes six full-time team members and a growing community of close to 6,000 downloads worldwide. A 3-pillar approach Geiger said that at its core, AlchemistOne is built around three key pillars of recovery: Mindfulness and meditation. Active reflection. Physical movement. Users can access a library of audio content, including guided meditations, podcast-style interviews and personal stories from people around the world who are in recovery. Geiger acknowledged an irony in people using the same phone or tablet that accessed gambling sites, social media or other addictive content as a tool for recovery. Instead of turning to a casino app or a harmful distraction, users can open AlchemistOne and engage in a quick mindfulness session. "I think often our phones and our computers are the things that we use to escape into and keeps us a lot of trouble," he said. "We definitely wanted to build that daily companion that lived in that same space that you maybe had some trouble before." Since the launch in April 2025, the response has been swift and steady the pair says. "It's really exciting that we see new members every five or 10 minutes jumping into the app and signing up," Galloway said. For both founders, the real win isn't downloads, it's impact. "We're getting that feedback from people who are saying, you know, this is resonating with me," Galloway said. "It's complicated, it's complex. And if we can just bring something to the table that helps people get through their day and potentially helps long lasting recovery, that's really the end goal."
Globe and Mail
5 hours ago
- Globe and Mail
This AI Stock Just Hit a 52-Week Low -- Here's Why That's an Opportunity
Key Points Confluent recently hit a 52-week low, as investors were not impressed by management's latest guidance. However, the company is growing at a brisk pace and building a solid revenue pipeline. An increase in AI-focused workloads should pave the way for stronger growth at Confluent, and its valuation makes it worth buying right now. 10 stocks we like better than Confluent › Data streaming platform provider Confluent (NASDAQ: CFLT) saw a sharp drop in its share price this year, which was blamed on slowing growth. Things went from bad to worse following the release of the company's latest quarterly results. Confluent reported its second-quarter results on July 30. The stock shed almost a third of its value the following day and also touched a 52-week low on Aug. 1. Investors were quick to press the panic button despite Confluent's better-than-expected results, as management's guidance didn't inspire much confidence. Savvy investors looking to buy a potential artificial intelligence (AI) winner should consider using Confluent's dip as a buying opportunity. Here's why. AI could give Confluent a nice shot in the arm Confluent reported a 20% year-over-year increase in its revenue in the previous quarter to just over $282 million. Its earnings growth was even more impressive at 50%, to $0.09 per share. Confluent management remarked on the latest earnings conference call that it is navigating a tight spending environment. The company, which operates a cloud-based data streaming platform to store, access, connect, and manage customers' data and applications in real time, says that its growth was hindered by customers' "optimization" initiatives. According to CEO Jay Kreps: In Q2, our larger customers continued their optimization efforts and adopted new use cases in a more measured pace. While we are confident that this elevated level of optimization will eventually subside, our outlook for the second half assumes consumption growth notably below what we've seen in the same period of prior years. Unsurprisingly, investors were spooked by this forecast of slowing sales growth in the second half of the year. But at the same time, one shouldn't forget that Confluent has slightly increased the lower end of its 2025 revenue guidance. What's more, the company's future revenue pipeline is improving at a brisk pace despite the controlled spending by customers. This is evident from the 31% increase in its remaining performance obligations (RPO) in the previous quarter. RPO is the total value of a company's unfulfilled contracts at the end of a period. This metric grew at a much faster pace than the growth in Confluent's top line. That's good news, as the faster growth in Confluent's revenue backlog should allow it to eventually increase its growth rate in the future once it starts fulfilling the contracts that it's signing now. Meanwhile, the company expects a big boost from the adoption of AI services in the cloud. Confluent says that it saw an increase in the number of customers using its data streaming platform to support real-time AI workloads such as agentic AI applications, code generation, content creation, chatbots, and others. In fact, Confluent says that it expects "production AI use cases to grow 10x across a few hundred customers" in 2025. The company cited several examples where customers around the globe are using its data streaming solutions to build AI agents capable of handling real-time queries. Looking ahead, there's a good chance that more customers could flock to Confluent's cloud-based platform, thanks to the sharp increase that it's predicting in real-time AI use cases. This explains why analysts expect an improvement in Confluent's earnings growth rate in the future. Data by YCharts. The valuation makes the stock an attractive buy The sharp drop in Confluent's stock price this year explains why it is trading at an attractive 5 times sales right now, less than half its average five-year sales multiple. Moreover, Confluent stock trades at a discount to the U.S. technology sector's average sales multiple of 8.4. Investors can consider buying Confluent at this discounted multiple, since it is clocking a healthy revenue growth rate despite controlled customer spending. Analysts also expect its top line to grow in the mid-teens for the next couple of years, though don't be surprised to see Confluent doing better than that, given the pace at which its revenue backlog is improving. Data by YCharts. If the company indeed manages to outpace Wall Street's revenue expectations over the next couple of years, the market could reward it with a higher sales multiple. That could pave the way for healthy stock price upside. Even if Confluent's revenue increases to $1.57 billion in 2027 (as per the previous chart) and it trades at even 7 times sales at that time (a discount to the U.S. technology sector's average), its market cap could hit $11 billion. That would be an increase of 86% from current levels. So, investors looking for an AI stock that's cheap and is trading at an attractive valuation should take a closer look at Confluent, as it seems capable of stepping on the gas in the long run. Should you invest $1,000 in Confluent right now? Before you buy stock in Confluent, consider this: The Motley Fool Stock Advisor analyst team just identified what they believe are the 10 best stocks for investors to buy now… and Confluent wasn't one of them. The 10 stocks that made the cut could produce monster returns in the coming years. Consider when Netflix made this list on December 17, 2004... if you invested $1,000 at the time of our recommendation, you'd have $653,427!* Or when Nvidia made this list on April 15, 2005... if you invested $1,000 at the time of our recommendation, you'd have $1,119,863!* Now, it's worth noting Stock Advisor's total average return is 1,060% — a market-crushing outperformance compared to 182% for the S&P 500. Don't miss out on the latest top 10 list, available when you join Stock Advisor. See the 10 stocks » *Stock Advisor returns as of August 4, 2025
