How to Use Open WebUI with vLLM on the Dell Pro Max 16 Plus MB16250 with Qualcomm Inference Card in Linux

Prerequisites

Complete the one-time device setup How to Setup the Dell Pro Max 16 Plus with Qualcomm Inference Card in Linux.

You must pass a Huffingface token (HF_TOKEN) to use this model. First, log in to Huggingface (create a free account if you do not have one). You then must accept the model use terms at the model link.

Your model access approval takes 5-10 minutes, and you receive an email confirmation. Once this is complete, you next create a token. From the Huggingface.

1. Click your profile icon in the upper right corner of the Huggingface website.
2. From the drop-down menu, select Settings.
3. In the left side navigation panel, click Access Tokens.
4. Click the + Create new token button.
5. Provide a name for your token (for example, "my-hf-token").
6. Select the Read button (do not select Fine-grained or Write buttons).
7. Click the Create token button.
8. The token value is displayed. Copy this value immediately and save it in a secure location, as it will not be shown again once you leave the page.

Now in an Ubuntu terminal use huggingface-cli login to enter your token that allows you to use the Llama-3.2-1B-Instruct model from Huggingface. If you do not have huggingface-cli installed, you may need to install using pip install huggingface-hub. Sample output.

Sample use of my-hf-token HF_TOKEN

dell@dell:~$ huggingface-cli login
⚠️  Warning: 'huggingface-cli login' is deprecated. Use 'hf auth login' instead.

    _|    _|  _|    _|    _|_|_|    _|_|_|  _|_|_|  _|      _|    _|_|_|      _|_|_|_|    _|_|      _|_|_|  _|_|_|_|
    _|    _|  _|    _|  _|        _|          _|    _|_|    _|  _|            _|        _|    _|  _|        _|
    _|_|_|_|  _|    _|  _|  _|_|  _|  _|_|    _|    _|  _|  _|  _|  _|_|      _|_|_|    _|_|_|_|  _|        _|_|_|
    _|    _|  _|    _|  _|    _|  _|    _|    _|    _|    _|_|  _|    _|      _|        _|    _|  _|        _|
    _|    _|    _|_|      _|_|_|    _|_|_|  _|_|_|  _|      _|    _|_|_|      _|        _|    _|    _|_|_|  _|_|_|_|

    A token is already saved on your machine. Run `hf auth whoami` to get more information or `hf auth logout` if you want to log out.
    Setting a new token will erase the existing one.
    To log in, `huggingface_hub` requires a token generated from https://huggingface.co/settings/tokens .
Enter your token (input will not be visible): 
Add token as git credential? (Y/n) y
Token is valid (permission: read).
The token `my-hf-token` has been saved to /home/dell/.cache/huggingface/stored_tokens
Your token has been saved in your configured git credential helpers (store).
Your token has been saved to /home/dell/.cache/huggingface/token
Login successful.
The current active token is: `my-hf-token`
dell@dell:~$ HF_TOKEN=<INSERT YOUR TOKEN HERE>

Download the Llama 3.2:1B Instruct prebuilt QPC

For this playbook we are going to use the prebuilt QPC for the Llama 3.2:1B Instruct-Zentree-Qualcinn Precompiled Model Catalog for Cloud AI Accelerators.Warning: This is a nonsecure link, use at your own risk.

Follow the steps below to download and extract the QPC from the archive. The AI100 inferencing card uses up to two SoCs (System on Chip) so we download that specific prebuilt version using the Zentree download link address. If you do not have curl already installed, you can install it in the terminal using the command sudo apt install curl.

  # Use existing Downloads folder for model downloads
cd ~/Downloads
# Download QPC archive from Zentree as llama3.2-1b-instruct.tar.gz
curl -fSL https://dc00tk1pxen80.cloudfront.net/SDK1.20.4/meta-llama/Llama-3.2-1B-Instruct/Llama-3.2-1B-Instruct_qpc_16cores_128pl_8192cl_1fbs_2devices_mxfp6_mxint8.tar.gz -o llama3.2-1b-instruct.tar.gz
# Extract QPC archive
tar -xzvf llama3.2-1b-instruct.tar.gz
# Rename extracted directory to llama3.2-1b-instruct-qpc
mv $(tar -tzf llama3.2-1b-instruct.tar.gz | head -1 | cut -f1 -d"/") llama3.2-1b-instruct-qpc

Serve the Llama 3.2:1B Instruct model using vLLM

Open Terminal and run the following command.

If you have not previously used this version of the AI100 Docker container, it fetches ~14GB before opening the Docker container. Be sure that you have setup your HF_TOKEN before opening the Docker container.

docker run --rm -it \
  --name qaic-bench \
  --device=/dev/accel/accel1 \
  --device=/dev/accel/accel2 \
  --network host \
  --ulimit nofile=1048576 \
  -e OMP_NUM_THREADS=8 \
  -e HF_TOKEN=$HF_TOKEN \
  -v ~/Downloads/llama3.2-1b-instruct-qpc:/root/qpc \
  ghcr.io/quic/cloud_ai_inference_ubuntu24:1.20.6.0 \
  /opt/vllm-env/bin/vllm serve meta-llama/Llama-3.2-1B-Instruct \
    --max-num-seq 1 \
    --max-model-len 8192 \
    --max-seq_len-to-capture 128 \
    --device qaic \
    --device-group 0,1 \
    --quantization mxfp6 \
    --kv-cache-dtype mxint8 \
    --override-qaic-config "qpc_path=/root/qpc/qpc"

The model is ready for inferencing once you see the message "Application startup complete."

To stop vLLM (unloading the model and closing the container), press Ctrl+C in this terminal window. Larger models may take up to 1 minute to unload.

References: Docker - Qualcomm® Cloud AI SDK User Guide and vLLM - Qualcomm® Cloud AI SDK User Guide

Launch Open WebUI Container

Open a new terminal and run the following command.

docker run -d \
  -p 3000:8080 \
  -e WEBUI_AUTH=False \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui \
  ghcr.io/open-webui/open-webui:main-slim

This fetches a container with Open WebUI if you have not previously downloaded it. After ~20 seconds Open WebUI is available in your web browser at the following link: http://localhost:3000/

http://localhost:3000/

The container continues to run in the background until you stop the container. Use the following commands in a terminal to stop, restart, or update the container.

To stop the container: docker rm -f open-webui

To restart an existing container: docker start open-webui

To update the container: docker pull ghcr.io/open-webui/open-webui:main-slim

Reference: Quick Start | Open WebUI

Configure Open WebUI

Use your browser for the following link http://localhost:3000/admin/settings/connections

1. Add an OpenAI API Connection - on the line Manage OpenAI API Connections press the + for connection type on the right side.
2. Toggle Connection Type to Local.
3. Set the URL to http://host.docker.internal:8000/v1
4. Press Save.

Reference: Quick Start / Open WebUI

Use Open WebUI

Use your browser to connect to this link http://localhost:3000/ and start chatting, Open WebUI auto-detects the model. You can keep the container active and change models underneath running using vLLM, or stop the container using the commands section 4: Launch Open WebUI Container.

Reference: Home | Open WebUI