Why RAG Alone Isn't Enough To Achieve Real ROI In The Agentic AI Era

Forbes

a day ago

Alon Goren, CEO and Cofounder of AnswerRocket, transforming your analytics with AI.
If you looked under the hood of generative AI (GenAI) technologies over the last year or so, you probably came across the concept of retrieval augmented generation (RAG). RAG has gained a lot of buzz, celebrated as a helpful innovation that enables AI to deliver more accurate, powerful insights from data.
But there's a problem: RAG isn't the AI panacea that many organizations have envisioned. Although RAG-based solutions showed early promise in simple AI applications, they can fall short of board-level expectations for transformative enterprise value and ROI.
RAG Use Cases
As NVIDIA explained, 'RAG is a technique for enhancing the accuracy and reliability of generative AI models with information fetched from specific and relevant data sources.' It works in three steps:
• Retrieval: A user queries a model, which searches its knowledge base for relevant context.
• Augmentation: The retrieved data is added as context to the prompt.
• Generation: A generative AI model produces a response to the user based on both the input query and the retrieved context.
Essentially, RAG can produce more accurate AI responses by incorporating additional context. It's valuable for many basic applications, like finding information buried deep in lengthy static documents (like spreadsheets or Word documents). But it struggles when users try to push it beyond its limits, such as depending on RAG alone for hardcore data analytics or highly complex statistical computations. Basically, RAG tends to fall short for hefty, data-driven analytics workflows that involve many different steps. And it's those types of use cases that organizations are imagining when they think of AI as a true business differentiator.
A good example is the use of RAG to generate SQL queries. Although RAG-created SQL queries may look right for general business analysts, they could be asking the wrong questions and ultimately getting incorrect answers.
And that's one of the key dangers of AI—getting bad 'insights' with high confidence. When people leverage RAG for use cases, it's not suited to—such as analytics applications with dynamic content, custom business logic or complex math—they're liable to get error-prone and out-of-context results, possibly without knowing it.
AI Agents: An Alternative To RAG
RAG is valuable for some use cases. But for organizations to get the broad ROI they're looking for from artificial intelligence, AI agents are often the better option. Agentic AI refers to systems with genuine autonomy. They can make decisions and take action independently. They don't just follow pre-defined rulesets, and they don't require input from humans.
What makes AI agents so special is the freedom they offer. With agentic AI, models have real agency. For example, if a user comes to an agent and asks, 'Why are my sales declining? What's going on?' The model can essentially tell itself, 'I'm not really sure. I need to consult some other sources.' Then, it can analyze sources like the internet to learn about policy driving consumer behavior, databases for trends in product evolution, syndicated data to look at the overall market and other relevant information. AI agents have the freedom to leverage additional tools (including RAG solutions) and do more research so they can provide accurate, deep insights and answers to complicated questions. RAG models, on the other hand, have been architected to force a potentially incorrect or incomplete response based on limited data.
Is This Agentic AI Really Agentic?
The agentic AI space is heating up quickly. More and more vendors are rolling out AI agents or incorporating agentic AI technology into their platforms. Organizations should look hard at these technologies. Early in the process, they should be careful to make sure that the agentic AI they're evaluating is truly agentic.
Like every major new enterprise tech innovation (especially anything involving AI), there's a lot of marketing hype surrounding AI agents, and the label is often slapped onto solutions that really don't fit the bill. Here's a quick checklist that will tell you if AI is really agentic.
Agentic AI should:
• Make decisions on its own.
• Be capable of handling complex tasks.
• Leverage external resources to get the job done better.
• Clearly explain how it arrived at its conclusions or why it took action.
• Identify its own errors or inaccuracies and course-correct in real time to improve results.
• Use verifiers to ensure it stays within proper guardrails.
Transformative AI Requires More Than Basic Capabilities
The effective use of AI begins with understanding what a model can really do and not leaning on it for much beyond that. You wouldn't ask your company's internal HR chatbot to analyze and provide insights about the business' sales data. So, you also shouldn't lean on RAG for more challenging data analysis or to try and uncover multifaceted insights.
Corporate boards and the C-suite are looking at AI to transform their organizations, delivering incredible new insights and recommendations for their businesses. RAG alone can't provide that. But AI agents can.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?