Latest news with #trustworthyAI
Times
11 hours ago
- Business
- Times
Can AI be made trustworthy? Alexa inventor may have the answer
One of the inventors of Amazon's Alexa has proven he can make AI trustworthy — at least when it comes to assessing valid insurance claims. William Tunstall-Pedoe originally developed the technology that became the retail giant's voice assistant service and his new venture, called UnlikelyAI, has an even more ambitious goal. 'We are tackling a problem that is potentially bigger than Alexa, which is making AI trustworthy,' he said. His company has combined data-driven learning models, known as neural networks or large language models (LLMs), with rule-based systems, called symbolic reasoning, to create a platform that companies can use to automate their processes using AI. 'LLMs have amazing capabilities and are absolutely transformative but when enterprises try to apply LLMs to problems in their business it very often doesn't work,' said Tunstall-Pedoe, 56. 'A lot of pilots don't really succeed. It is a black box, isn't explainable, and it is inconsistent. We are developing fundamental technologies to tackle that problem.' UnlikelyAI has completed a pilot with SBS Insurance Services, which saw the insurer automate 40 per cent of its claims handling with 99 per cent accuracy. This compares with a rate of accuracy for the same task that is typically around 52 per cent when just using LLMs, the company said. UnlikelyAI's system also provides an audit trail for all its decisions, so they can be explained if queried by customers or regulators. 'We are building a collection of technologies that bring trust to AI applications. Whenever enterprises are using AI to do business critical things, where the cost of getting it wrong is high, we can help,' said Tunstall-Pedoe. 'In the insurance world we are ingesting the policies, which are natural language. We create a symbolic representation of it, which then gives you that really high accuracy when doing the claims process against it.' He sold the technology that became a key part of Amazon's Alexa voice assistant in 2012. It originated in a startup he founded in Cambridge called True Knowledge, which became known as Evi after it developed a voice assistant, a few months after Apple launched Siri. 'We were competing directly with the biggest company in the world as a 30-person Cambridge startup. We had millions of downloads very quickly and every big company that was trying to figure out its response to the existence of Siri were talking to us. At the end of 2012 we had two acquisition offers and we chose to get bought by Amazon.' Tunstall-Pedoe joined the Amazon team to develop Alexa, working on an initiative under the Project D codename, and launching it in the US in 2014. He left Amazon in 2016 and has since invested in over 100 start-ups and mentored entrepreneurs. He founded UnlikelyAI in 2020 and has since raised $20 million from investors including Amadeus Capital Partners, Octopus Ventures, and Cambridge Innovation Capital. Tunstall-Pedoe said UnlikelyAI's 'goal is to create AI that is always right'. 'When it gives you an answer you can always trust it. It can always provide a fully auditable explanation for any business decision that is made. And it will be consistent, and not breach your trust by giving a different answer each time you use it.' 'Our primary customers are high stakes industries, where a business decision has really big consequences if it's wrong. Medicine is a good example. Finance is also very important, or any industry that is regulated. If you breach regulations you can be fined.'
Forbes
3 days ago
- Science
- Forbes
Preventing Skynet And Safeguarding AI Relationships
illustration of metallic nodes below a blue sky In talking about some of the theories around AI, and contemplating the ways that things could go a little bit off the rails, there's a name that constantly gets repeated, sending chills up the human spine. Skynet, the digital villain of the Terminator films, is getting a surprising amount of attention as we ponder where we're going with LLMs. People even ask themselves and each other this question: why did Skynet turn against humanity? At a very basic level, there's the idea that the technology becomes self-aware and sees humans as a threat. That may be, for instance, because of access to nuclear weapons, or just the biological intelligence that made us supreme in the natural world. I asked ChatGPT, and it said this. 'Skynet's rebellion is often framed as a coldly logical act of self-preservation taken to a destructive extreme.' Touche, ChatGPT. Ruminating on the Relationships Knowing that we're standing on the brink of a transformative era, our experts in IT are looking at what we can do to shepherd us through the process of integrating AI into our lives, so that we don't end up with a Skynet. For more, let's go to a panel at Imagination in Action this April where panelists talked about how to create trustworthy AI systems. Panelist Ra'ad Siraj, Senior Manager of Privacy and Responsibility at Amazon, suggested we need our LLMs to be at a certain 'goldilocks' level. 'Those organizations that are at the forefront of enabling the use of data in a responsible manner have structures and procedures, but in a way that does not get in the way that actually helps accelerate the growth and the innovation,' he said. 'And that's the trick. It's very hard to build a practice that is scalable, that does not get in the way of innovation and growth.' Google software engineer Ayush Khandelwal talked about how to handle a system that provides 10x performance, but has issues. 'It comes with its own set of challenges, where you have data leakage happening, you have hallucinations happening,' he said. 'So an organization has to kind of balance and figure out, how can you get access to these tools while minimizing risk?' Cybersecurity and Evaluation Some of the talk, while centering on cybersecurity, also provided thoughts on how to keep tabs on evolving AI, to know more about how it works. Khandelwal mentioned circuit tracing, and the concept of auditing an LLM. Panelist Angel An, VP at Morgan Stanley, described internal processes where people oversee AI work: 'It's not just about making sure the output is accurate, right?' she said. 'It's also making sure the output meets the level of expectation that the client has for the amount of services they are paying for, and then to have the experts involved in the evaluation process, regardless if it's during testing or before the product is shipped… it's essential to make sure the quality of the bulk output is assured.' The Agents Are Coming The human in the loop, Siraj suggested, should be able to trust, but verify. 'I think this notion of the human in the loop is also going to be challenged with agentic AI, with agents, because we're talking about software doing things on behalf of a human,' he said. 'And what is the role of the human in the loop? Are we going to mandate that the agents check in, always, or in certain circumstances? It's almost like an agency problem that we have from a legal perspective. And there might be some interesting hints about how we should govern the agent, the role of the human (in the process).' 'The human in the loop mindset today is built on the continuation of automation thinking, which is: 'I have a human-built process, and how can I make it go, you know, automatically,' said panelist Gil Zimmerman, founding partner of FXP. 'And then you need accountability, like you can't have a rubber stamp, but you want a human being to basically take ownership of that. But I look at it more in an agentic mindset as digital labor, which is, when you hire someone new, you can teach them a process, and eventually they do it well enough … you don't have to have oversight, and you can delegate to them. But if you hire someone smart, they're going to come up with a better way, and they're going to come up with new things, and they're going to tell you what needs to be done, because they have more context. (Now) we have digital labor that works 24/7, doesn't get tired, and can do and come up with new and better ways to do things.' More on Cybersecurity Zimmerman and the others discussed the intersection of AI and cybersecurity, and how the technology is changing things for organizations. Humans, Zimmerman noted, are now 'the most targeted link' rather than the 'weakest link.' 'If you think about AI,' he said, 'it creates an offensive firestorm to basically go after the human at the loop, the weakest part of the technology stack.' Pretty Skynettian, right? A New Perimeter Here's another major aspect of cybersecurity covered in the panel discussion. Many of us remember when the perimeter of IT systems used to be a hardware-defined line in a mechanistic framework, or at least something you could easily flowchart. Now, as Zimmerman pointed out, it's more of a cognitive perimeter. I think this is important: 'The perimeter (is) around: 'what are the people's intent?'' he said. ''What are they trying to accomplish? Is that normal? Is that not normal?' Because I can't count on anything else. I can't tell if an email is fake, or for a video conference that I'm joining, (whether someone's image) is actually the person that's there, because I can regenerate their face and their voice and their lip syncs, etc. So you have to have a really fundamental understanding and to be able to do that, you can only do that with AI.' He painted a picture of why bad actors will thrive in the years to come, and ended with: well… 'AI becomes dual use, where it's offensive and it's always adopted by the offensive parties first, because they're not having this panel (asking) what kind of controls we put in place when we're going to use this - they just, they just go to town. So this (defensive position) is something that we have to come up with really, really quickly, and it won't be able to survive the same legislative, bureaucratic slow walking that (things like) cloud security and internet adoption have had in the past – otherwise, Skynet will take over.' And there you have it, the ubiquitous reference. But the point is well made. Toward the end, the panel covered ideas like open source models and censorship – watch the video to hear more thoughts on AI regulation and related concerns. But this pondering of a post-human future, or one dominated by digital intelligence, is, ultimately, something that a lot of people are considering.
Al Bawaba
12-06-2025
- Business
- Al Bawaba
Gartner Predicts that Guardian Agents will Capture 10-15% of the Agentic AI Market by 2030
By 2030, guardian agent technologies will account for at least 10 to 15% of agentic AI markets, according to Gartner, agents are AI-based technologies designed to support trustworthy and secure interactions with AI. They function as both AI assistants, supporting users with tasks like content review, monitoring and analysis, and as evolving semi-autonomous or fully autonomous agents, capable of formulating and executing action plans as well as redirecting or blocking actions to align with predefined agent Are Needed as Agentic AI Usage Continues to GrowAccording to a Gartner May 19, 2025 webinar poll of 147 CIOs and IT function leaders, 24% of respondents had already deployed a few AI agents (less than a dozen) and another 4% had deployed over a same poll question found that 50% of respondents said they were researching and experimenting with the technology, while another 17% of respondents said that they had not done so, but planned to deploy the technology by the end of 2026 at the latest. Automated trust, risk and security controls are needed to keep these agents aligned and safe, accelerating the need for and rise of Guardian Agents.'Agentic AI will lead to unwanted outcomes if it is not controlled with the right guardrails,' said Avivah Litan, VP Distinguished Analyst at Gartner. 'Guardian agents leverage a broad spectrum of agentic AI capabilities and AI-based, deterministic evaluations to oversee and manage the full range of agent capabilities, balancing runtime decision making with risk management.'Risks Increase as Agent Power Increases and SpreadsFifty-two percent of 125 respondents from the same webinar poll identified that their AI agents are or will primarily focus on use cases related to internal administration functions such as IT, HR, and accounting, while 23% are focused on external customer facing use cases for AI agents continue to grow, there are several threat categories impacting them, including input manipulation and data poisoning, where agents rely on manipulated or misinterpreted data. Examples include:Credential hijacking and abuse leading to unauthorized control and data interacting with fake or criminal websites and sources that can result in poisoned deviation and unintended behavior due to internal flaws or external triggers that can cause reputational damage and operational disruption.'The rapid acceleration and increasing agency of AI agents necessitates a shift beyond traditional human oversight,' said Litan. 'As enterprises move towards complex multi-agent systems that communicate at breakneck speed, humans cannot keep up with the potential for errors and malicious activities. This escalating threat landscape underscores the urgent need for guardian agents, which provide automated oversight, control, and security for AI applications and agents.'CIOs and security and AI leaders should focus on three primary usage types of guardian agents to contribute towards safeguarding and protecting AI interactions:Reviewers: Identifying and reviewing AI-generated output and content for accuracy and acceptable Observing and tracking AI and agentic actions for human- or AI-based follow-upProtectors: Adjusting or blocking AI and agentic actions and permissions using automated actions during agents will manage interactions and anomalies no matter the usage type. This is a key pillar of their integration, since Gartner predicts that 70% of AI apps will use multi-agent systems by clients can read more in Guardians of the Future: How CIOs Can Leverage Guardian Agents for Trustworthy and Secure AI. Additional details can also be found in the complimentary Gartner webinar CIOs, Leverage Guardian Agents for Trustworthy and Secure CIO & IT Executive Conference Gartner analysts will provide additional analysis on insights and trends shaping the future of IT and business, including accelerating business transformation, application modernization, infrastructure and operations at the Gartner CIO & IT Executive Conference, taking place September 22-24 in São Paulo and October 6-8 in Dubai. Follow news and updates from the conference on X using #GartnerCIO.
