Artificial Intelligence

Improve RAG Data Ingestion with PowerScale’s New RAG Connector

Dell's RAG connector for PowerScale accelerates data ingestion and processing to enable real time RAG.
By   |  

This blog post is co-authored by Gaurav Chawla, VP | Fellow, Dell Technologies

At GTC 2025, Dell is introducing an open-source Dell PowerScale-specific connector for RAG applications that will help customers free up CPU and GPU resources on their AI compute clusters, reduce network and storage I/O load and greatly accelerate data processing time.

Dell customers can integrate the PowerScale RAG connector with RAG Frameworks like LangChain or with the NVIDIA AI Enterprise software to further optimize data processing using NVIDIA NIM microservices. These capabilities enable scaling agentic AI deployments using Dell AI Data Platform with NVIDIA and integrates with hardware and software services in NVIDIA AI Data Platform reference design.

Let’s look at the problem this connector solves, how it works, and how developers can use it in their RAG applications.

Why is this important?

Today, when developers ingest data using their favorite RAG framework and build RAG applications, they are faced with a challenge – how do I keep the RAG application up to date with the latest changes to the dataset?

Typically, developers set up a data processing pipeline where they ingest source data on set intervals from a storage system like PowerScale, a key storage infrastructure option of the Dell AI Factory with NVIDIA. This pipeline creates and updates chunks and embeddings that the RAG application uses. Creating chunks and embeddings is computationally expensive and requires CPU and GPU resources.

When the RAG application needs to ingest millions of documents and terabytes of data, it creates heavy load on the compute, network, and storage systems, especially when the same documents show up multiple times.  The PowerScale RAG Connector dramatically improves performance by reducing the amount of data that needs to be processed. The connector intelligently identifies the files that were already processed, and more importantly, which files need to be processed. Best yet, the connector integrates with RAG frameworks like LangChain, generic Python classes, and NVIDIA NeMo Retriever and NIM microservices.

How does it work? With PowerScale’s latest software release, admins can use the new MetadataIQ feature where filesystem metadata is saved periodically to an external Elasticsearch database. The PowerScale RAG connector uses the information in the database to keep track of the files that are already processed to improve the data ingestion and processing in RAG applications.

Figure 1: System design of a typical customer RAG application when using NVIDIA AI Enterprise software with PowerScale RAG Connector.
Figure 2: System design of a typical customer RAG application. In purple is the PowerScale RAG connector that allows files to be skipped that were already processed.

In both diagrams:

  • Developers will ingest the data from PowerScale using Dell open-source python-based RAG Connector.
  • The connector will communicate with the Metadata Repo (database) which tracks new, modified and updated files.
  • Once the database operation is performed, the RAG Connector will return the results of only new and modified files to the RAG framework, skipping any files that have not been modified.
  • The new and modified files will be processed normally through the RAG application. These files can be chunked and embedded using standard methods or by leveraging NVIDIA NeMo Retriever.
  • Insights can be extracted quickly and accurately from large amounts of data, using the NeMo Retriever collection of NIM microservices for extraction, embedding and reranking.

As a result, RAG applications only process new and modified files which free up CPU, GPU and network resources for other computational tasks.

When using Dell RAG PowerScale connector with NVIDIA AI Enterprise software, like NeMo Retriever, customers will retain the benefits of Dell’s connector and leverage the best of breed RAG capabilities from NVIDIA.

Getting Started

Developers can get immediate access to the PowerScale RAG connector by:

About the Author: Adam Brenner

Adam Brenner is a Senior Principal Engineering Technologist at Dell Technologies in the ISG (Infrastructure Solutions Group) Chief Technology Office. Adam has 10+ years of experience in the technology industry and has been with Dell since 2014. Over his career at Dell, Adam has produced prototypes and contributed to our Data Protection, Security, Storage and Data Management products. He currently focuses on AI with our storage products. Adam has contributed to open-source projects and is an active member in Toastmasters where he earned the Distinguished Toastmaster achievement. Adam has 50+ granted patents and graduated with a B.S. in Computer Science from the University of California at Irvine. Adam’s motto is that he loves to fix things, mostly because he is the one breaking them.