
Encountered a problematic response from an AI model? More standards and tests are needed, say researchers
The emergence of these undesirable behaviors is compounded by a lack of regulations and insufficient testing of AI models, researchers told CNBC.
Getting machine learning models to behave the way it was intended to do so is also a tall order, said Javier Rando, a researcher in AI.
"The answer, after almost 15 years of research, is, no, we don't know how to do this, and it doesn't look like we are getting better," Rando, who focuses on adversarial machine learning, told CNBC.
However, there are some ways to evaluate risks in AI, such as red teaming. The practice involves individuals testing and probing artificial intelligence systems to uncover and identify any potential harm — a modus operandi common in cybersecurity circles.
Shayne Longpre, a researcher in AI and policy and lead of the Data Provenance Initiative, noted that there are currently insufficient people working in red teams.
While AI startups are now using first-party evaluators or contracted second parties to test their models, opening the testing to third parties such as normal users, journalists, researchers, and ethical hackers would lead to a more robust evaluation, according to a paper published by Longpre and researchers.
"Some of the flaws in the systems that people were finding required lawyers, medical doctors to actually vet, actual scientists who are specialized subject matter experts to figure out if this was a flaw or not, because the common person probably couldn't or wouldn't have sufficient expertise," Longpre said.
Adopting standardized 'AI flaw' reports, incentives and ways to disseminate information on these 'flaws' in AI systems are some of the recommendations put forth in the paper.
With this practice having been successfully adopted in other sectors such as software security, "we need that in AI now," Longpre added.
Marrying this user-centred practice with governance, policy and other tools would ensure a better understanding of the risks posed by AI tools and users, said Rando.
Project Moonshot is one such approach, combining technical solutions with policy mechanisms. Launched by Singapore's Infocomm Media Development Authority, Project Moonshot is a large language model evaluation toolkit developed with industry players such as IBM and Boston-based DataRobot.
The toolkit integrates benchmarking, red teaming and testing baselines. There is also an evaluation mechanism which allows AI startups to ensure that their models can be trusted and do no harm to users, Anup Kumar, head of client engineering for data and AI at IBM Asia Pacific, told CNBC.
Evaluation is a continuous process that should be done both prior to and following the deployment of models, said Kumar, who noted that the response to the toolkit has been mixed.
"A lot of startups took this as a platform because it was open source, and they started leveraging that. But I think, you know, we can do a lot more."
Moving forward, Project Moonshot aims to include customization for specific industry use cases and enable multilingual and multicultural red teaming.
Pierre Alquier, Professor of Statistics at the ESSEC Business School, Asia-Pacific, said that tech companies are currently rushing to release their latest AI models without proper evaluation.
"When a pharmaceutical company designs a new drug, they need months of tests and very serious proof that it is useful and not harmful before they get approved by the government," he noted, adding that a similar process is in place in the aviation sector.
AI models need to meet a strict set of conditions before they are approved, Alquier added. A shift away from broad AI tools to developing ones that are designed for more specific tasks would make it easier to anticipate and control their misuse, said Alquier.
"LLMs can do too many things, but they are not targeted at tasks that are specific enough," he said. As a result, "the number of possible misuses is too big for the developers to anticipate all of them."
Such broad models make defining what counts as safe and secure difficult, according to a research that Rando was involved in.
Tech companies should therefore avoid overclaiming that "their defenses are better than they are," said Rando.
Hashtags

Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Yahoo
23 minutes ago
- Yahoo
Vexev and U.S. Renal Care Complete Enrollment in Groundbreaking Trial Evaluating Use of Robotic Ultrasound Scanning for AV Fistula Mapping in Dialysis Clinics
CANSCAN study enrolled 120 patients with chronic kidney disease using the investigational, semi-autonomous VxWave™ Ultrasound Imaging System The VxWave Ultrasound Imaging System potentially provides important insights into vascular access, to possibly enable earlier creation and use of AV fistula Results to be presented at American Society of Nephrology (ASN) Annual Meeting in November SYDNEY and SAN ANTONIO, July 30, 2025 /PRNewswire/ -- Vexev, an Australian-based medical device company advancing next-generation vascular imaging, and U.S. Renal Care, a leading provider of in-center and home dialysis in the United States, today announced the successful completion of enrollment in the CANSCAN clinical trial (NCT06691672). The first-of-its-kind, multi-center feasibility study enrolled 120 patients with chronic kidney disease to evaluate the use of semi-autonomous ultrasound scanning utilizing the VxWave Ultrasound Imaging System for arteriovenous (AV) fistula mapping in dialysis clinics. "The VxWave ultrasound system may remove one of the biggest hurdles in the creation of vascular access. Patients could have vascular mapping conducted in their dialysis facility, enabling a more efficient way to obtain a permanent vascular access point," commented Varshi Broumand, MD, MHA, FACP, FASN, Interventional Nephrologist at South Texas Renal Care Group and CANSCAN Study Principal Investigator. "Performing the ultrasound scans during the clinic visit would make AV fistula mapping more convenient and timely and potentially avoid sending patients to additional imaging appointments." An AV fistula is a connection made between an artery and a vein, typically in the arm, to create an access point for patients requiring dialysis. This allows blood to flow quickly and easily during treatment, which helps the body remove toxins more effectively. The VxWave Ultrasound Imaging System is a robotic tomographic ultrasound system specifically designed for upper limb vascular imaging - integrating advanced robotics, machine learning and ultrasound signal processing to create a 3D vascular model and detailed report that potentially provides insights into vascular access points - all at the point of care. "Validating this concept is the first step toward a future where intelligent machines can perform high-precision medical imaging without a human hand on the probe. The potential for a robotic system guided by AI, delivering real clinical value - it's not science fiction anymore," said Shannon Thomas, MD, Chief Medical Officer, Vexev. "The real excitement begins when this can change lives, not just lab results. This is a huge step forward in our mission to improve the lives of millions of people worldwide living with end-stage kidney disease." "U.S. Renal Care is committed to delivering innovative treatments that address the most critical needs for people living with kidney disease," said Geoffrey A. Block, MD, FASN, Associate Chief Medical Officer & Senior Vice President, Clinical Research & Medical Affairs, U.S. Renal Care. "The CANSCAN trial of the VxWave Ultrasound Imaging System opens the possibility of integrating semi-autonomous ultrasound imaging into our workflow and improving patient outcomes, by allowing patients to receive life-saving permanent vascular access placed in the optimal location, as quickly as possible." About VexevVexev is an Australian med‑tech startup founded in 2018 by Drs. John Carroll and Eamonn Colley - University of New South Wales (UNSW) PhD graduates in vascular fluid dynamics and computational imaging. Vexev is on a mission to improve lives by developing the next generation of vascular imaging solutions. Visit to learn more. About U.S. Renal CareU.S. Renal Care, the fastest-growing dialysis provider in the nation, partners with nephrologists across 32 states in the U.S. to care for more than 36,000 people living with kidney disease. Since 2000, U.S. Renal Care has been a leader in clinical quality, innovation, and operational excellence - delivering the best experience and outcomes for our patients. Visit to learn more. Photo - - View original content: SOURCE Vexev Sign in to access your portfolio


TechCrunch
25 minutes ago
- TechCrunch
Nvidia AI chip challenger Groq said to be nearing new fundraising at $6B valuation
In Brief AI chip startup Groq is in talks to raise a fresh $600 million at a near $6 billion valuation, sources tell Bloomberg, although the deal isn't yet final and terms could change. Groq raised $640 million at a $2.8 billion valuation in November 2024, making this double the valuation in about nine months. Groq previously raised about $1 billion. The new round is led by Austin-based firm Disruptive, Bloomberg reports. The November round was led by Blackrock, with participation from Neuberger Berman, Type One Ventures, Cisco, KDDI, and Samsung Catalyst Fund. Groq was founded by Jonathan Ross, who previously worked at Google developing its Tensor Processing Unit chip. The startup emerged from stealth in 2016. This new raise comes after Groq announced in May an exclusive partnership with Bell Canada to power the telco's large AI infrastructure project. In April, Groq partnered with Meta to offer AI infrastructure to speed Llama 4 inference. Neither Disruptive nor Groq immediately returned our request for comment.


Hamilton Spectator
25 minutes ago
- Hamilton Spectator
New Niagara Falls attraction reaches for the stars
They've brought you to see the animals in Africa and Australia, digitally. Now, the owners of Hologram Zoo Niagara want to send you to space. The Space Experience is an immersive installation that brings visitors to the stars using the same holographic technology as the virtual zoo, which opened in June. Developed by Australian-based company Axiom Holographics, the attraction will create the illusion of travelling at warp speed, visiting the International Space Station and seeing a supernova up close, along with projections of the eight planets. It's at the same location as the Hologram Zoo — the corner of Victoria Avenue and Bender Street, across from Casino Niagara. Owner Marty Speelman says it's the first of several themed shows coming to the venue. 'The plan was always that every four or five months to introduce a new show,' he says. 'There's always revolving content. There's a Halloween show coming, a Christmas show coming, seasonal content as well.' As visitors walk through the attraction, motion-tracked projections and layered lighting create a sense of movement and depth around them. It makes people feel part of the environment, not just passersby, says Speelman. 'You do things like go through the tail of comet, see a star being born, you do a space walk, all those kinds of things.' While designing it, Axiom consulted with astronomers and science educators for accuracy. A narrator explains what people are seeing without 'overwhelming' detail. Speelman says the Hologram Zoo is off to a solid start, despite its location away from the main tourist hub. 'Everyone who comes through has been loving it,' he says. 'The challenge for us is, we're not on Clifton Hill. We're a couple-minute walk away. It's just getting people to know that we're there. 'We have a mix of social media marketing and we have a staff person out on the street handing out flyers.' Axiom's head of Canadian business development, John Jackson, says there are plans to open between 30 and 40 hologram zoos throughout Canada. The Space Experience is making its Canadian debut in Niagara Falls. Admission to the Space Experience is $19.99 to $24.99, with combo prices available for both attractions.