
Encountered a problematic response from an AI model? More standards and tests are needed, say researchers
The emergence of these undesirable behaviors is compounded by a lack of regulations and insufficient testing of AI models, researchers told CNBC.
Getting machine learning models to behave the way it was intended to do so is also a tall order, said Javier Rando, a researcher in AI.
"The answer, after almost 15 years of research, is, no, we don't know how to do this, and it doesn't look like we are getting better," Rando, who focuses on adversarial machine learning, told CNBC.
However, there are some ways to evaluate risks in AI, such as red teaming. The practice involves individuals testing and probing artificial intelligence systems to uncover and identify any potential harm — a modus operandi common in cybersecurity circles.
Shayne Longpre, a researcher in AI and policy and lead of the Data Provenance Initiative, noted that there are currently insufficient people working in red teams.
While AI startups are now using first-party evaluators or contracted second parties to test their models, opening the testing to third parties such as normal users, journalists, researchers, and ethical hackers would lead to a more robust evaluation, according to a paper published by Longpre and researchers.
"Some of the flaws in the systems that people were finding required lawyers, medical doctors to actually vet, actual scientists who are specialized subject matter experts to figure out if this was a flaw or not, because the common person probably couldn't or wouldn't have sufficient expertise," Longpre said.
Adopting standardized 'AI flaw' reports, incentives and ways to disseminate information on these 'flaws' in AI systems are some of the recommendations put forth in the paper.
With this practice having been successfully adopted in other sectors such as software security, "we need that in AI now," Longpre added.
Marrying this user-centred practice with governance, policy and other tools would ensure a better understanding of the risks posed by AI tools and users, said Rando.
Project Moonshot is one such approach, combining technical solutions with policy mechanisms. Launched by Singapore's Infocomm Media Development Authority, Project Moonshot is a large language model evaluation toolkit developed with industry players such as IBM and Boston-based DataRobot.
The toolkit integrates benchmarking, red teaming and testing baselines. There is also an evaluation mechanism which allows AI startups to ensure that their models can be trusted and do no harm to users, Anup Kumar, head of client engineering for data and AI at IBM Asia Pacific, told CNBC.
Evaluation is a continuous process that should be done both prior to and following the deployment of models, said Kumar, who noted that the response to the toolkit has been mixed.
"A lot of startups took this as a platform because it was open source, and they started leveraging that. But I think, you know, we can do a lot more."
Moving forward, Project Moonshot aims to include customization for specific industry use cases and enable multilingual and multicultural red teaming.
Pierre Alquier, Professor of Statistics at the ESSEC Business School, Asia-Pacific, said that tech companies are currently rushing to release their latest AI models without proper evaluation.
"When a pharmaceutical company designs a new drug, they need months of tests and very serious proof that it is useful and not harmful before they get approved by the government," he noted, adding that a similar process is in place in the aviation sector.
AI models need to meet a strict set of conditions before they are approved, Alquier added. A shift away from broad AI tools to developing ones that are designed for more specific tasks would make it easier to anticipate and control their misuse, said Alquier.
"LLMs can do too many things, but they are not targeted at tasks that are specific enough," he said. As a result, "the number of possible misuses is too big for the developers to anticipate all of them."
Such broad models make defining what counts as safe and secure difficult, according to a research that Rando was involved in.
Tech companies should therefore avoid overclaiming that "their defenses are better than they are," said Rando.
Hashtags

Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles


TechCrunch
36 minutes ago
- TechCrunch
Index Ventures' Jahanvi Sardana shares the truth about TAM and what founders should focus on instead
Early-stage founders just can't get away from TAM — the concept of having a total addressable market for their startup to disrupt and conquer. But Index Ventures partner Jahanvi Sardana has a reminder for all those founders worried about finding TAM for their product or service: many startups have emerged from markets that, at the time, were essentially nonexistent. 'What was the market for search before Google?' Sardana asked the audience at TechCrunch's 2025 All Stage event in Boston, held earlier this month. 'What was the market for operating systems before Microsoft, or the market for cloud before Amazon?' Sardana compares TAM to surfing. Every few years, there are massive waves founders must ride — first came the internet, then the mobile wave, then the cloud, and now, she said, the biggest wave of all: artificial intelligence. 'Have you shaped the right product to ride this wave?' she continued. 'That's what we call product market fit.' Which TAM bucket are you in? Sardana places TAM into three buckets: known market, emerging market, and invisible market. The first, known market, already exists, and it is when a founder seeks to replace a legacy incumbent and must prove to an investor why their startup idea is better. Techcrunch event Tech and VC heavyweights join the Disrupt 2025 agenda Netflix, ElevenLabs, Wayve, Sequoia Capital — just a few of the heavy hitters joining the Disrupt 2025 agenda. They're here to deliver the insights that fuel startup growth and sharpen your edge. Don't miss the 20th anniversary of TechCrunch Disrupt, and a chance to learn from the top voices in tech — grab your ticket now and save up to $675 before prices rise. Tech and VC heavyweights join the Disrupt 2025 agenda Netflix, ElevenLabs, Wayve, Sequoia Capital — just a few of the heavy hitters joining the Disrupt 2025 agenda. They're here to deliver the insights that fuel startup growth and sharpen your edge. Don't miss the 20th anniversary of TechCrunch Disrupt, and a chance to learn from the top voices in tech — grab your ticket now and save up to $675 before prices rise. San Francisco | REGISTER NOW 'Everyone brushes their teeth,' she said. 'You have to tell me why you're building a better toothbrush.' The emerging market is when a certain sector of the market is using a product, and there is potential for it to go mainstream. 'Think about non-alcoholic beer before it became cool,' Sardana said. Then there is the invisible market, which Sardana calls 'the biggest trap,' and 'also a little bit of a dark art.' The market doesn't exist, and a founder has to essentially create the one and provide investors with evidence of how innovative they can be. 'Think about smartphones in 2006, nobody knew they wanted them and they changed the world,' she said, later adding that 'people don't know what they're looking for and sometimes you have to show them what's possible.' The audience at All Stage, many of whom are early-stage founders, peppered Sardana with questions, largely about what investors want to see. For instance, do investors want to see a TAM slide in a pitch deck? 'It's OK to create that slide and talk about the math behind your TAM,' Sardana said, though she added that sometimes investors get annoyed when founders rely too much on industry metrics rather than having their own unique insight. Sardana also cautioned founders not to rely too much on industry reports. If a founder is too dependent on an external service to dictate how they think about the market, it can signal that they haven't thought deeply about the market they are trying to build in, she commented. 'How do you size the TAM in the marketplaces, especially big marketplaces?' one audience member asked. Well, that question hurts, Sardana quipped. After all, Index once passed on Airbnb, having believed its TAM was too small. 'The reality is Airbnb created a whole new inventory, which is now bigger than some of the largest hotel brands, and that led to a big change in behavior on how people travel,' she said, adding that marketplace TAMs are tricky. 'You want to focus on, again, what is unlocking supply, and once you unlock the supply, how will behavior change?' The audience also asked Sardana what makes a company stand out to an investor like herself. A tough one, Sardana said, but a really important one. Ultimately, if a founder can understand who the customer is and why they are willing to purchase their product, then a company should have no problem standing up to investors, she added. 'We're in the business of evaluating founders more than markets or products or anything else,' she said. 'When you talk about your market, it's really a lens on your ambition.'


The Hill
an hour ago
- The Hill
Crypto lobby gains ground under Trump
At least 27 crypto companies or advocates filed their first-ever lobbying disclosures this year across some 20 firms, reflecting an increasing appetite for influence in a more crypto-friendly Washington. The newcomers originate from all corners of the industry. There's betting website Polymarket, a gaming company that created an NFT version of the White House Easter egg hunt, and a Seychelles-based exchange that cannot operate in the U.S. market due to a federal money laundering settlement. Together, they spent nearly $2.8 million between April 1 and June 30 on lobbying landmark legislation promoting digital assets to the Treasury Department and the Securities and Exchange Commission, and a host of other issues relevant to blockchain infrastructure — an increasingly sprawling ecosystem that some hope could one day be as ubiquitous as the internet. The push has paid off for crypto so far. The GENIUS Act, a bill with bipartisan support signed by President Trump last week, has been regarded as the government's 'seal of approval' on the industry. The law sets up a regulatory framework for stablecoins, a type of cryptocurrency that is theoretically pegged to the U.S. dollar or another reference asset. The House also advanced several other landmark bills during its monumental 'crypto week,' which featured high-profile lobbying stunts such as vending machines around the Capitol and the National Mall with customized chocolate bars urging 'yes' votes, bankrolled by the crypto exchange Coinbase. Lobbying expenses that week were not covered in the second quarter disclosures. At least 73 companies or associations focused on crypto disclosed federal lobbying activities, to the tune of about $11.4 million. This total doesn't include spending from investment firms such as Andreessen Horowitz ($790,000) or BlackRock ($810,000) that have substantial crypto interests but also lobbied on a suite of other financial regulation issues. The Hill's Miriam Waldvogel has more here.


Business Journals
2 hours ago
- Business Journals
New AI platform has mapped every construction project in Texas giving contractors "God mode" for leads
A New Edge in Construction Intelligence Texas GCs have always relied on luck, timing, and relationships to win new construction projects. But a new AI-powered platform by is a game changer helping GCs discover private projects earlier, engage decision-makers sooner, and win deals ahead of competitors. One Houston-based general contractor put it plainly: 'It's God mode for leads.' a Texas-focused market intelligence platform, has mapped over 65,000 active private construction projects across the state. The platform provides structured access to early-stage signals such as rezoning activity, land title changes, permitting updates, and ownership transfers—often before plans are finalized or teams are selected. Unlike traditional platforms that rely on user input or call centers, pulls directly from city and state databases, private sources, and creates proprietary data where no structured information currently exists. Every update is tied to a traceable source document, giving users a clear paper trail for every insight. equips users to act early by connecting dots that would take a human weeks to uncover. Built for Early Signal Detection Contractors can follow civil engineers, developers, and architects, track movement across projects, and engage earlier with decision-makers. The platform helps BD teams position themselves as trusted advisors—shaping scope, not reacting to it. is the only source of early intelligence on private construction projects before the bid stage, giving contractors real-time visibility into upcoming opportunities. It surfaces what's coming—not just what's already out there. 10x Better Than Legacy Tools Traditional bid-management platforms like ConstructConnect and Dodge track public bid-stage activity. But by the time projects hit those systems, it's often too late. Private work is different. You find very few private opportunities on bid boards, and often the ones featured are much too late. 'ConstructConnect tells you what's already happening. Mercator shows you what's coming,' says founder Chloe Smith. Chloe Smith created the platform in 2020 after watching her father, a 45-year industry veteran, prepare for retirement. 'They were flying blind,' she recalls. 'Had he walked out, they wouldn't know where the work was coming from next.' 'My dad always talked to me about relationships,' she adds. 'As I matured, I realized the massive risk this imposed on his company. If he decided to retire, those relationships would disappear. Their revenue channel instantly appeared highly fragile.' She describes as a force multiplier tool for BD teams: a sales 'cheat code' that eliminates hours of manual research and replaces it with verified, real-time intelligence. And more than that, it solves a deeper pain point: the FOMO of not knowing what you don't know. 'We were tired of hearing about projects too late. With Mercator, we're in early—and it's directly impacted our win rate.' — VP, Business Development Free Permits App – Click Here Results on the Ground Texas-based users are already seeing impact. It's already being used by GCs like Joeris, Bartlett Cocke, and ChalkLine, and it's quickly becoming the de-facto standard in proactive business development across the state. A Houston interiors firm used to identify a permit revision, secure a meeting, and win the project. A Dallas GC tracked multifamily conversions early and secured two jobs before competitors were even aware. One Houston-based contractor secured a self-storage project by spotting an early land acquisition and rezoning signal. They invited the developer to tour a similar build, influenced the design, and ultimately won the job. A self-perform contractor identified a healthcare expansion weeks in advance and locked in supplier work ahead of any bid package. Internal metrics show users surface 3–5x more early-stage opportunities weekly—and spend far less time chasing dead-end leads. 'If I have one coffee meeting, I might walk away with two leads in an hour. If I'm on Mercator, I could find five in that same time.' — Director of Business Development, General Contractor 'It really allows us to parse the data, analyze it, and get high-quality leads to reps in the field. So yes, I'm a fan.' — Senior Manager, Business Development 'Mercator cuts through the noise. We don't waste time chasing dead-end leads anymore.' — Sales Manager Pricing – Click Here to Book a Live Demo Widespread Adoption and Growing Demand is actively used by members of Houston AGC, Austin AGC, San Antonio AGC, and Texo AGC/ABC. It tracks commercial, industrial, healthcare, institutional, and multifamily projects across tenant improvement, exterior renovations, core and shell, and ground up construction in every major metro in the state. The platform's growth reflects a broader shift in how construction teams are using technology. In a more competitive, margin-conscious market, timing and insight have become decisive advantages. 'AI gives people the time and space to be more strategic,' says Nihar Dalmia, a partner at Deloitte. 'It enables higher-value work and accelerates decision-making.' doesn't replace business development. It sharpens it. The best teams don't just react faster—they show up earlier, with more to offer.