AI Inferencing Is Growing In Importance—And RAG Is Fueling Its Rise

Forbes

24-07-2025

As the AI infrastructure market evolves, we've been hearing a lot more about AI inference—the last step in the AI technology infrastructure chain to deliver fine-tuned answers to the prompts given to an AI model, whether that's by a human or agentic AI.
Inferencing will grow in importance in delivering specialized and fine-tuned AI intelligence. It requires a different technology infrastructure stack from that used for training AI models. One of the building blocks is a technology called Retrieval-Augmented Generation, which is a way to infuse large language models with additional data. RAG can also help drive down the costs of serving up AI inference.
RAG is growing in importance as more enterprises need to collect and apply specific data to AI applications to ensure depth and accuracy. RAG-based models use specific custom data to apply up-to-date knowledge to generate more accurate, relevant, and trustworthy responses. In a new Futuriom research report, "RAGs to Riches: The New Era of AI Inferencing," analyst Craig Matsumoto recently dove into RAG and found it will drive a lot of technology subsegments, including vector databases, edge networking, and AI security.
Why Inferencing Is Booming
First, let's talk about why the inference market represents the next leg in AI infrastructure growth. Agentic AI is being unleashed by organizations worldwide. AI-driven agents can satisfy more complex queries that require multiple steps. But inferencing is the crucial step to customizing these agentic software models for specific use cases.
Inferencing is the heart of enterprise AI. Enterprises will still train specialized models, but they can't reap the benefits of AI until they become experts at inference.
AI infrastructure companies believe that the inference market could dwarf the size of the AI training market. NVIDIA's CEO Jensen Huang said during his GTC 2025 keynote that the rise of agentic AI, expected to be a cornerstone of inference, could push AI's computing requirements to grow roughly 100-fold in as little as a year. AMD expects the AI inference market to grow at 80% per year 'for the next few years,' CEO Lisa Su said recently, although she offered neither market sizes nor a timeframe for context.
Some of these estimates are probably extreme, as it does not seem economically possible for AI capital spending to grow by an order of magnitude in a year, but there is no doubt that the center of gravity has shifted toward inference.
What makes inferencing special is that it can be placed anywhere—on a chip or chips in a phone, in a manufacturing facility, or in small-to-medium-sized datacenters. It typically doesn't require the most expensive GPUs needed for LLM training.
AI is seeing wide adoption across industries, but enterprises are fine-tuning LLMs for different use ... More cases.
What About RAG?
Inferencing brings us to RAG, which helps add relevance to LLMs and small language models for specialized applications. RAG is a less costly way to add data to AI models and a better way to accommodate dynamic, real-time data.
Training is slow and expensive, whereas RAG operates on the fly during inferencing, infusing a model with information relevant to the current query.
Visa, for example, has set up employees to use RAG routinely. They can query six different LLMs—popular foundation models such as ChatGPT and Claude—that sit behind Visa's firewall, accessed through one interface. Fraud detection is one obvious use case, but employees also use the LLMs with RAG to hunt for domain-specific knowledge.
RAG is also useful for updating data. One problem with AI models is that they are trained over time and quickly become out-of-date. This doesn't matter if you are looking for less recent information, but it's a shortcoming for dynamic, real-time information.
Databases Need an Update
Applying RAG to AI models to get the best inferencing means that data architectures need to adapt. This explains why large data management companies, including Oracle, Snowflake, and Databricks, are racing to explain how their architectures are being adapted to serve AI—even though in many cases their databases were built for applications that now like legacy approaches.
Vector databases are an area of growing interest. Vector databases are good at storing and labeling unstructured data, such as documents and images, using vectors and semantic search. This enables enterprises to store diverse data and perform multimedia searches. This has led major data players such as Oracle, Databricks, and Snowflake to incorporate vector support into their products.
But RAG and inferencing have an impact that stretches well beyond vector databases. According to the Futuriom RAG report, here are some of the other ways that RAG will influence the technology markets:
Interest in SLMs. Foundational AI models form the basis of RAG, enabling AI applications to understand queries, process retrieved data, and generate responses. For specific industries or applications, LLMs can be stripped down and customized to SLMs, which can operate more economically. Technology companies to watch: Amazon, Anthropic, Arcee.ai, Hugging Face, OpenAI, Microsoft, and Stability AI.
Database Evolution. As mentioned above, RAG is transforming the database and data management market at a very high speed. In addition to Oracle, Snowflake, and Databricks, some companies to watch include DDN, LanceDB, MinIO, MongoDB, Pinecone, VAST Data, Weka, and Vectara.
The Model Context Protocol (MCP) is accelerating the maturity of agentic AI. It's an esoteric under-the-covers protocol, but developers have leapt onto MCP to make AI handle sophisticated tasks.
Key technology players: Virtually every infrastructure company is employing MCP to enable agentic AI.
Security. Security is growing in importance for RAG and also for MCP. Companies will need to allay security concerns with the proper architecture and security tools.
Technology companies to watch: Many security companies are focused on AI, but we have recently seen some interesting AI data security launches from companies such as Aryaka, Aviatrix, Cloudflare, Eclypsium, Fortanix, and Teleport. Large AI providers such as Amazon have also released security services for RAG applications, such as Amazon Bedrock Guardrails for sensitive data redaction and Knowledge Bases for managing RAG workflows.
Hybrid, edge, and multicloud networking. The same security needs translate to the network. More secure application-layer networking and infrastructure will be needed for inferencing to help connect and feed RAG applications with data.
Key networking companies to watch in connecting AI applications: Arista, Aviatrix, Cloudflare, Cisco, F5, Alkira, Aryaka, Aviz Networks, Hedgehog, Juniper Networks (HPE), Palo Alto Networks, and Versa Networks.