The PowerScale RAG Connector is an open-source Python library designed to enhance RAG application performance during data ingestion by skipping files that have already been processed. It leverages PowerScale's unique capability to identify changed files within its storage system and provides this information in an easily consumable format via MetadataIQ.

Developers can integrate the PowerScale RAG Connector directly within LangChain or LlamaIndex as supported document loaders, or use it independently as a generic Python class. Version 2 introduces stable file identity tracking using logical inode numbers (lin) to handle file modifications and prevent stale content in vector databases. This document demonstrates these integration options in the following sections.

The intended audience for this document includes software developers, machine learning scientists, and AI developers who will utilize files from PowerScale in the development of a RAG application.

This guide is divided into two sections: setting up the environment and using the connector. Note that system administration privileges are required for the initial configuration on PowerScale, which may need to be performed by PowerScale administrators.

The following table provides definitions for some of the terms that are used in this document.

Dell Technologies and the authors of this document welcome your feedback on this document. Contact the Dell Technologies team by email.

Author

Adam Brenner

Contributors

Michael Horgan, Sanne Bloemsma

MetadataIQ is a new feature introduced in PowerScale OneFS 9.10 that copies filesystem metadata (such as file path, size, owner, type, etc.) to an external Elasticsearch database. MetadataIQ continuously updates the Elasticsearch index when files are added, modified, or deleted on OneFS. It can be configured to monitor the entire filesystem (/ifs) or specific portions of it, saving metadata to Elasticsearch.

MetadataIQ is highly optimized, utilizing internal OneFS features to quickly and efficiently detect modified files using OneFS snapshots and its changelist. It automatically manages the lifecycle of PowerScale snapshots without impacting performance, usage, cloud pools, or daily cluster operations.

The initial configuration of MetadataIQ is performed via the PowerScale command line interface and requires elevated administrative privileges. This one-time setup configures the connection to Elasticsearch. An automated job scheduler then runs continuously, requiring no user involvement to keep Elasticsearch updated with the latest filesystem changes.

Additionally, it is the customer's responsibility to deploy, configure, and monitor Elasticsearch. This guide will cover the basics of deploying Elasticsearch for development purposes only.

Elasticsearch can be deployed in various scenarios, including bare metal, virtual machines, containers, and as-a-service. All these deployment options are compatible with MetadataIQ. MetadataIQ requires HTTP or HTTPS connectivity to Elasticsearch with a specific API key. This guide will cover deploying Elasticsearch on Ubuntu, running either on bare metal or a virtual machine.

The recommend way to install Elasticsearch on Ubuntu by using built in package manager, apt.

First step is to download and install the Elasticsearch public signing key.

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

Next step is to install configure the package manager sources list to download the prebuild deb packages directly from Elastic

sudo apt-get install apt-transport-https

echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

Finally, we can install Elasticsearch using the apt package manager command

sudo apt-get update && sudo apt-get install elasticsearch

If no errors exist with the installation process, Elasticsearch should now be installed. To enable it on system startup, run the following command:

sudo systemctl daemon-reload && sudo systemctl enable elasticsearch.service

Start the Elasticsearch manually and take careful note of the first-time deployment information such as the username and password. This information will be required to configure PowerScale to write filesystem information to Elasticsearch.

sudo systemctl start elasticsearch.service

You can verify Elasticsearch is online by running the following command:

curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic:${ELASTIC_PASSWORD} https://localhost:9200 

PowerScale administrators must perform an initial configuration of MetadataIQ to copy filesystem metadata to Elasticsearch. This requires using the root account on PowerScale via an SSH connection.

The first step is to find the PowerScale cluster name and unique ID. We can do this by running the following python code on PowerScale (all on one line):

python -c "from isi.config.arrcfg import config_load; c = config_load(); print(f'isi-metadataiq-index.{c.name}.{c.guid}'[:255].lower())"

This command will output the index name PowerScale will use to write its metadata to. More importantly, we need to this index name to Elasticsearch in order to generate a unique API key for PowerScale to use.

Example value could be:

isi-metadataiq-index.ioflash.04bf1be5f95e87b4fa6512225f9ec530ca70

On your Ubuntu environment where Elasticsearch is installed, run the following REST API command using curl

curl -L 'https://localhost:9200/_security/api_key' \
-H 'Content-Type: application/json' \
-u elastic:${ELASTIC_PASSWORD} \
-d '{
  "name": "pscale_metadataiq",
  "role_descriptors": { 
    "pscale_metadataiq_descriptors": {
      "cluster": ["all"],
      "indices": [
        {
          "names": ["isi-metadataiq-index.ioflash.04bf1be5f95e87b4fa6512225f9ec530ca70"],
          "privileges": ["all"]
        }
      ]
    }
  }
}'

This will return JSON response and the value we need to provide PowerScale is the encoded field. Example response below:

{
    "id": "Gm9yP5UBJofu5tMOYfOQ",
    "name": "pscale_metadataiq",
    "api_key": "-GUoFsMDSMOxLH_ooYhlGA",
    "encoded": "R205eVA1VUJKb2Z1NXRNT1lmT1E6LUdVb0ZzTURTTU94TEhfb29ZaGxHQQ=="
}

On PowerScale, we need to run the following command:

isi metadataiq settings modify \
  --hostname https://ip-address-or-fqdn-of-elasticsearch/ \
  --verify-certificate true \
  --host-port 9200 \
  --path /ifs/ \
  --schedule "every day every 12 hours" \
  --ca-certificate-path "/ifs/es-http-ca-cert.crt" \
  --api-key "R205eVA1VUJKb2Z1NXRNT1lmT1E6LUdVb0ZzTURTTU94TEhfb29ZaGxHQQ==" \
  --verbose

isi services -a isi_metadataiq_producer_d enable
isi services -a isi_metadataiq_consumer_d enable

Remember: to use the encoded field from Elasticsearch as the value to the api-key on the PowerScale command to setup MetadataIQ

The PowerScale RAG connector can be used in four different use cases. Select the use case that best fits your needs

The four different use cases are:

1. Integrated with LangChain
2. Integrated with LlamaIndex
3. Integrated with Nvidia AI Enterprise Software
   1. Example: nv-ingest, NeMo Retriever, and so forth
4. Standalone using Generic Python Interfaces

Using the PowerScale RAG connector with Langchain is a drop in replacement to any other document loader from LangChain.

pip install –upgrade –quiet powerscale-rag-connector[langchain]

Then in your python code add the import line:

from powerscale_rag_connector import PowerScaleUnstructuredLoader

With the libraries installed and used in your LangChain RAG application, you can use the document loader like any other:

loader = PowerScaleUnstructuredLoader(
    es_host_url=”https://ip-address-or-fqdn-of-elasticsearch:9200/”,
    es_index_name="isi-metadataiq-index.ioflash.04bf1be5f95e87b4fa6512225f9ec530ca70", # replace
    es_api_key="-GUoFsMDSMOxLH_ooYhlGA", # replace
    folder_path="/ifs/", # replace
    force_scan=False,
    verify_ssl=True
)

# The loader returns a standard Document Loader following LangChain interface
for file in loader.lazy_load():
    # pass file to your LangChain pipeline to store in your chunk and vector store
	 # The file metadata includes: source, snapshot, lin (logical inode number), and change_types

Using the PowerScale RAG connector with LlamaIndex is a drop in replacement to any other document reader from LlamaIndex.

pip install –upgrade –quiet powerscale-rag-connector[llamaindex]

Then in your python code add the import line:

from powerscale_rag_connector import PowerScaleUnstructuredReader

With the libraries installed and used in your LlamaIndex RAG application, you can use the reader like any other:

reader = PowerScaleUnstructuredReader(
    es_host_url="https://ip-address-or-fqdn-of-elasticsearch:9200/",
    es_index_name="isi-metadataiq-index.ioflash.04bf1be5f95e87b4fa6512225f9ec530ca70",  # replace
    es_api_key="-GUoFsMDSMOxLH_ooYhlGA",  # replace
    folder_path="/ifs/",  # replace
    force_scan=False,
    verify_ssl=True,
    mode="elements",  # 'single' for whole doc, 'elements' for granular chunks
)
 
# The reader returns standard LlamaIndex Document objects
for document in reader.load_data():
    # pass document to your LlamaIndex pipeline to store in your chunk and vector store
    # The document metadata includes: source, snapshot, lin (logical inode number), and change_types

We can use the PowerScale RAG connector with Nvidia AI Enterprise Software like nv-ingest, retriever, etc.

First step is to download our python package:

pip install –upgrade –quiet powerscale-rag-connector

Next is to import our python library in your code

from powerscale_rag_connector import PowerScalePathLoader

Then in your python code load our PowerScale RAG Connector library to feed the data to Nvidia AI Enterprise Software. In this example we are using nv-ingest but the same logic could apply to NeMo retriever.

def get_powerscale_files() -> Iterator[Path]:
  loader = PowerScalePathLoader(
    es_host_url="https://ip-address-or-fqdn-of-elasticsearch:9200/",
    es_index_name="isi-metadataiq-index.ioflash.04bf1be5f95e87b4fa6512225f9ec530ca70",  # replace
    es_api_key="-GUoFsMDSMOxLH_ooYhlGA",  # replace
    folder_path="/ifs/",  # replace
    force_scan=False,
    verify_ssl=True
  )

  # The loader returns tuples of (Path, snapshot_id, lin, change_types)
  for file_path, snapshot, lin, change_types in loader.lazy_load():
    yield file_path


for file_path in get_powerscale_files():

  ingestor = Ingestor(
    message_client_allocator=RestClient,
    message_client_hostname="ip-address-or-fqdn-of-nv-ingest",  # replace
    message_client_port=7670,
    message_client_kwargs={"api_version": "v2"},
  )

  # Extract text/tables/charts and split into chunks
  results = (
    ingestor.files([str(file_path)])
    .extract(
      extract_text=True,
      extract_tables=True,
      extract_charts=True,
      extract_images=False,
    )
    .pdf_split_config(pages_per_chunk=64)
    .ingest(return_full_response=True)
  )

  # use results in the rest of your code to generate embeddings
  # (e.g., NVIDIAEmbeddings) and store in a vector database

Lastly, we can use the PowerScale RAG connector in a standalone python program for general use.

First step is to download our python package:

pip install –upgrade –quiet powerscale-rag-connector

Next is to import our python library in your code

from powerscale_rag_connector import PowerScalePathLoader

Next is to use call into our RAG connector:

def get_powerscale_files() -> Iterator[Path]:
  loader = PowerScalePathLoader(
    es_host_url=”https://ip-address-or-fqdn-of-elasticsearch:9200/”,
    es_index_name="isi-metadataiq-index.ioflash.04bf1be5f95e87b4fa6512225f9ec530ca70", # replace
    es_api_key="-GUoFsMDSMOxLH_ooYhlGA", # replace
    folder_path="/ifs/", # replace
    force_scan=False,
    verify_ssl=True
  )

  # The loader returns tuples of (Path, snapshot_id, lin, change_types), but we only need the Path
  for file_tuple in loader.lazy_load():
    # Use this tuple in your generic python code to create chunks and embeddings
    yield file_tuple[0]

From our python examples we were able to show four different methods of using our RAG Connector for PowerScale that loads data.

The following Dell Technologies documentation provides other information related to this document. Access to these documents depends on your login credentials. If you do not have access to a document, contact your Dell Technologies representative.

• Dell Data Lakehouse and PowerScale MetadataIQ for AI Datasets from Unstructured Data

The following documentation on the MetadataIQ provides additional and relevant information.

• PowerScale OneFS 9.10.0.0 Documentation
• OneFS 9.10 | Dell PowerScale OneFS Release Notes

All the examples listed in this document are also available on our GitHub project.

• GitHub - dell/powerscale-rag-connector: Dell PowerScale RAG Connector · GitHub