Como executar benchmarks usando vLLM no Dell Pro Max 16 Plus com a placa de inferência Qualcomm no Linux

Summary: Aprenda a executar referências de desempenho usando vLLM no Dell Pro Max 16 Plus com a placa de inferência Qualcomm no Linux.

Affected Products

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.

Check out other resources

Instructions

Fazer download de um QPC (Qualcomm Program Container, contêiner de programa Qualcomm)
Baixe um modelo de IA pré-configurado QPC do catálogo de modelos pré-compilados da Zentree - Zentree-Qualcomm Pre-compiled Model Catalog for Cloud AI Accelerators

Nota: Certifique-se de que o QPC seja compatível com o Dell Pro Max procurando SoCs / Tensor slicing = 2
```
tar -xzvf ~/Downloads/qpc_name.tar.gz
```
Atenda ao LLM (Large Language Model) usando o vLLM.
Abra um Terminal e execute o seguinte comando (substitua: <QPC_DIR>, <MODEL_ID>, <CTX_LEN>, <BATCH_SIZE>e <PREFILL_SEQ_LEN>).
Nota: Isso <MODEL_ID>, <CTX_LEN>, <BATCH_SIZE>e <PREFILL_SEQ_LEN> dependem da configuração de compilação QPC. Para QPCs criados e hospedados pela Zentree, esses valores são documentados em seu site. O mapeamento é <BATCH_SIZE> com "Tamanho completo do lote", <CTX_LEN> com "Context Length (CL)" e <PREFILL_SEQ_LEN> com "Chunking Prompt Length" da tabela de configurações QPC do modelo Zentree. Observe também que alguns modelos exigem que você configure um token Huggingface que você precisará passar para o contêiner para acessar.
```
    docker run --rm -it \

    --name qaic-bench

    --device=/dev/accel/accel1 \

    --device=/dev/accel/accel2 \

    --network host \

    --ulimit nofile=1048576 \

    -e OMP_NUM_THREADS=8 \

    -e HF_TOKEN=$HF_TOKEN \

    -v ~/Downloads/<QPC_DIR>:/root/qpc \

    ghcr.io/quic/cloud_ai_inference_ubuntu24:1.20.6.0 \

    /opt/vllm-env/bin/vllm serve <MODEL_ID> \

    --max-num-seq <BATCH_SIZE> \

    --max-model-len <CTX_LEN> \

    --max-seq_len-to-capture <PREFILL_SEQ_LEN> \

    --device qaic \

    --device-group 0,1 \

    --quantization mxfp6 \

    --kv-cache-dtype mxint8 \

    --override-qaic-config "qpc_path=/root/qpc/qpc"
```
Nota: Todos os QPCs hospedados pela Zentree para o Dell Pro Max são desenvolvidos com num_devices=2, mas se um modelo for criado com num_devices=1Então --device-group deve ser definido como 0 ou 1.

Todos os QPCs hospedados pela Zentree para o Dell Pro Max são desenvolvidos com mxfp6_matmul=true, mas se um modelo for criado com mxfp6_matmul=falseEntão --quantization mxfp6 deve ser descartado.

Todos os QPCs hospedados pela Zentree para o Dell Pro Max são desenvolvidos com mxint8_kv_cache=true, mas se um modelo for criado com mxint8_kv_cache=falseEntão --kv-cache-dtype mxint8 deve ser descartado.

Quando você vir a mensagem "Inicialização do aplicativo concluída", o modelo estará pronto para inferência.

Para interromper o vLLM (descarregando o modelo e fechando o contêiner), pressione Ctrl+C nesta janela do Terminal . Modelos maiores podem levar até 1 minuto para serem descarregados.

Referências: Docker – Guia do usuário do Qualcomm Cloud AI SDK e vLLM - Qualcomm Cloud AI SDK User Guide

Executar análises comparativas.
Abra um novo Terminal e execute o seguinte comando (substitua: <MODEL_ID> e <BATCH_SIZE> para corresponder ao modelo servido na etapa 2). Os valores para num-prompts, random-input-lene random-output-len pode ser ajustado conforme desejado.


    docker exec -it qaic-bench \

    /opt/vllm-env/bin/vllm bench serve \

    --model  \

    --num-prompts 25 \

    --max-concurrency  \

    --random-input-len 256 \

    --random-output-len 256

Depois que a referência de desempenho for concluída, o vLLM imprimirá as medições resultantes.

Nota: Para derivar o throughput (tokens/segundo ou TPS) apenas para o estágio de decodificação (geração de token), excluindo o estágio de preenchimento prévio (processamento de prompt), você pode calcular o TPS usando 1.000 ÷ TPOT médio (ms).

Por exemplo, um TPOT médio de 20ms equivale a uma taxa de transferência de decodificação de 50 tokens/segundo (TPS).

Exemplo de trabalho:

Faça download do QPC pré-construído Qwen3-30B-A3B-Instruct-2507.
Para este guia estratégico usamos o QPC pré-construído para o Qwen3 30B A3B Instruct 2507

Esse hiperlink direcionará você para um site fora da Dell Technologies.

.
Há dois modelos predefinidos disponíveis que funcionam na placa de inferência AI100 que oferece suporte a 2 SoCs - um com um 4K e outro com um comprimento de contexto de 8K. Para este guia estratégico é usado o modelo de comprimento de contexto 4K e 2 SoC.
Siga as etapas abaixo para baixar e extrair o QPC do arquivo. Note que se você ainda não tiver curl instalado, você pode instalá-lo no terminal usando o comando (sudo apt install curl).


        # Use existing Downloads folder for model downloads

        cd ~/Downloads

        # Download QPC archive from Zentree as qwen3-30b-a3b-instruct-2507.tar.gz

        curl -fSL https://dc00tk1pxen80.cloudfront.net/SDK1.20.4/Qwen/Qwen3-30B-A3B-Instruct-2507/

        Qwen3-30B-A3B-Instruct-2507_qpc_16cores_1pl_4096cl_1bs_2devices_mxfp6_mxint8.tar.gz -o qwen3-30b-a3b-instruct-2507.tar.gz

        # Extract QPC archive

        tar -xzvf qwen3-30b-a3b-instruct-2507.tar.gz

        # Rename extracted directory to qwen3-30b-a3b-instruct-2507-qpc

        mv $(tar -tzf qwen3-30b-a3b-instruct-2507.tar.gz | head -1 | cut -f1 -d"/") qwen3-30b-a3b-instruct-2507-qpc

Atenda ao modelo Qwen3:30B usando vLLM.
Abra um Terminal e execute o seguinte comando. Observe da receita geral acima que <QPC_DIR> é mapeado usando o "-v ~/Downloads/qwen3-30b-a3b-instruct-2507-qpc:/root/qpc" linha para o diretório QPC que o modelo é baixado para, <MODEL_ID> está mapeado para "Qwen/Qwen3-30B-A3B-Instruct-2507," <CTX_LEN> está mapeado para 4096, <BATCH_SIZE> é mapeado para 1 e <PREFILL_SEQ_LEN> é mapeado para 1 (tudo a partir da página do modelo Zentree para o modelo Qwen3:30B usando um comprimento de contexto 4K).

Observe que, se você não tiver usado anteriormente essa versão do contêiner do Docker AI100, ela buscará ~14 GB antes de abrir o contêiner do Docker.
```
        docker run --rm -it \

        --name qaic-bench \

        --device=/dev/accel/accel1 \

        --device=/dev/accel/accel2 \

        --network host \

        --ulimit nofile=1048576 \

        -e OMP_NUM_THREADS=8 \

        -v ~/Downloads/qwen3-30b-a3b-instruct-2507-qpc:/root/qpc \

        ghcr.io/quic/cloud_ai_inference_ubuntu24:1.20.6.0 \

        /opt/vllm-env/bin/vllm serve Qwen/Qwen3-30B-A3B-Instruct-2507 \

        --max-num-seq 1 \

        --max-model-len 4096 \

        --max-seq_len-to-capture 1 \

        --device qaic \

        --device-group 0,1 \

        --quantization mxfp6 \

        --kv-cache-dtype mxint8 \

        --override-qaic-config "qpc_path=/root/qpc/qpc"
```
O modelo estará pronto para inferência assim que você vir a mensagem "Application startup complete."

Nota: Para interromper o vLLM (descarregar o modelo e fechar o contêiner), pressione Ctrl+C nesta janela do terminal. Modelos maiores podem levar até 1 minuto para serem descarregados.

Referências: Docker – Guia do usuário do Qualcomm Cloud AI SDK e vLLM - Qualcomm Cloud AI SDK User Guide
Executar análises comparativas.
Abra um novo Terminal e execute o seguinte comando (substitua: <MODEL_ID> Para --model e <BATCH_SIZE> Para --max-concurrency para corresponder ao modelo servido na etapa 2). Os valores para num-prompts, random-input-lene random-output-len pode ser ajustado conforme desejado.
```
        docker exec -it qaic-bench \

        /opt/vllm-env/bin/vllm bench serve \

        --model Qwen/Qwen3-30B-A3B-Instruct-2507 \

        --num-prompts 25 \

        --max-concurrency 1 \

        --random-input-len 256 \

        --random-output-len 256
```

Captura de saída de amostra.
Exemplo de saída para servir Qwen3:30B usando vLLM.
Exemplo de resultado para vLLM:


        dell@dell:~/Downloads/qwen3-30b-a3b-instruct-2507-qpc$ docker run --rm -it \

        --name qaic-bench \

        --device=/dev/accel/accel1 \

        --device=/dev/accel/accel2 \

        --network host \

        --ulimit nofile=1048576 \

        -e OMP_NUM_THREADS=8 \

        -v ~/Downloads/qwen3-30b-a3b-instruct-2507-qpc:/root/qpc \

        ghcr.io/quic/cloud_ai_inference_ubuntu24:1.20.6.0 \

        /opt/vllm-env/bin/vllm serve Qwen/Qwen3-30B-A3B-Instruct-2507 \

        --max-num-seq 1 \

        --max-model-len 4096 \

        --max-seq_len-to-capture 1 \

        --device qaic \

        --device-group 0,1 \

        --quantization mxfp6 \

        --kv-cache-dtype mxint8 \

        --override-qaic-config "qpc_path=/root/qpc/qpc"

        loading /opt/qti-aic/dev/lib/x86_64/libQAic.so

        INFO 01-15 17:46:28 [__init__.py:269] Automatically detected platform qaic.


        
        INFO 01-15 17:46:30 [api_server.py:1043] vLLM API server version 0.8.6.dev0+gba41cc90e.d20251229



        INFO 01-15 17:46:30 [api_server.py:1044] args: Namespace(subparser='serve', model_tag='Qwen/Qwen3-30B-A3B-Instruct-2507', config='', host=None, port=8000, uvicorn_log_level='info', disable_uvicorn_access_log=False, allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='Qwen/Qwen3-30B-A3B-Instruct-2507', task='auto', tokenizer=None, hf_config_path=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, load_format='auto', download_dir=None, model_loader_extra_config={}, use_tqdm_on_load=True, config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', max_model_len=4096, guided_decoding_backend='auto', reasoning_parser=None, logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, data_parallel_size=1, enable_expert_parallel=False, max_parallel_loading_workers=None, ray_workers_use_nsight=False, disable_custom_all_reduce=False, block_size=None, gpu_memory_utilization=0.9, swap_space=4, kv_cache_dtype='mxint8', num_gpu_blocks_override=None, enable_prefix_caching=None, prefix_caching_hash_algo='builtin', cpu_offload_gb=0, calculate_kv_scales=False, disable_sliding_window=False, use_v2_block_manager=True, seed=None, max_logprobs=20, disable_log_stats=False, quantization='mxfp6', rope_scaling=None, rope_theta=None, hf_token=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=1, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config={}, limit_mm_per_prompt={}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=None, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=None, max_prompt_adapters=1, max_prompt_adapter_token=0, device='qaic', speculative_config=None, ignore_patterns=[], served_model_name=None, qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, max_num_batched_tokens=None, max_num_seqs=1, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, num_lookahead_slots=0, scheduler_delay_factor=0.0, preemption_mode=None, num_scheduler_steps=1, multi_step_stream_outputs=True, scheduling_policy='fcfs', enable_chunked_prefill=None, disable_chunked_mm_input=False, scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', worker_extension_cls='', generation_config='auto', override_generation_config=None, enable_sleep_mode=False, additional_config=None, enable_reasoning=False, disable_cascade_attn=False, override_qaic_config={'qpc_path': '/root/qpc/qpc'}, device_group=[0, 1], disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, enable_server_load_tracking=False, dispatch_function=<function ServeSubcommand.cmd at 0x7416f976a710>)




        config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 963/963 [00:00<00:00, 6.38MB/s]


        
        INFO 01-15 17:46:34 [config.py:728] This model supports multiple tasks: {'score', 'generate', 'classify', 'embed', 'reward'}. Defaulting to 'generate'.



        WARNING 01-15 17:46:34 [arg_utils.py:1711] --kv-cache-dtype is not supported by the V1 Engine. Falling back to V0.


        
        INFO 01-15 17:46:34 [config.py:1415] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor


        
        INFO 01-15 17:46:34 [config.py:1820] Disabled the custom all-reduce kernel because it is not supported on current platform.


        
        config_1m.json: 77.3kB [00:00, 163MB/s]


        
        generation_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████| 239/239 [00:00<00:00, 2.47MB/s]


        
        model.safetensors.index.json: 1.70MB [00:00, 57.5MB/s]


        
        tokenizer.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 11.4M/11.4M [00:02<00:00, 5.23MB/s]


        
        tokenizer_config.json: 9.38kB [00:00, 41.9MB/s]


        
        vocab.json: 2.78MB [00:00, 25.4MB/s]


        
        INFO 01-15 17:46:38 [api_server.py:246] Started engine process with PID 51


        
        merges.txt: 1.67MB [00:00, 11.8MB/s]


        
        loading /opt/qti-aic/dev/lib/x86_64/libQAic.so


        
        INFO 01-15 17:46:40 [__init__.py:269] Automatically detected platform qaic.



        INFO 01-15 17:46:41 [llm_engine.py:247] Initializing a V0 LLM engine (v0.8.6.dev0+gba41cc90e.d20251229) with config: model='Qwen/Qwen3-30B-A3B-Instruct-2507', speculative_config=None, tokenizer='Qwen/Qwen3-30B-A3B-Instruct-2507', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=mxfp6, enforce_eager=False, kv_cache_dtype=mxint8,  device_config=cpu, decoding_config=DecodingConfig(guided_decoding_backend='auto', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=Qwen/Qwen3-30B-A3B-Instruct-2507, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=None, chunked_prefill_enabled=False, use_async_output_proc=False, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[1],"max_capture_size":1}, use_cached_outputs=True,



        WARNING 01-15 17:46:42 [qaic.py:100] Pin memory is not supported on Qaic.



        WARNING 01-15 17:46:42 [registry.py:217] Model class <class 'vllm.model_executor.models.mllama.MllamaForConditionalGeneration'> already has a multi-modal processor registered to <vllm.multimodal.registry.MultiModalRegistry object at 0x7f2ffb2ddc90>. It is overwritten by the new one.



        WARNING 01-15 17:46:42 [registry.py:217] Model class <class 'vllm.model_executor.models.mllama4.Llama4ForConditionalGeneration'> already has a multi-modal processor registered to <vllm.multimodal.registry.MultiModalRegistry object at 0x7f2ffb2ddc90>. It is overwritten by the new one.



        WARNING 01-15 17:46:42 [qaic_worker.py:102] Reducing Torch parallelism from 8 threads to 8 to avoid unnecessary CPU contention. Set VLLM_QAIC_MAX_CPU_THREADS to tune this value as needed.



        INFO 01-15 17:46:42 [qaic_worker.py:87] Setting RLIMIT_STACK to 16.0MB.


        
        INFO 01-15 17:46:44 [parallel_state.py:1004] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0


        
        loading /opt/qti-aic/dev/lib/x86_64/libQAic.so


        
        INFO 01-15 17:46:44 [executor_base.py:112] # qaic blocks: 1, # CPU blocks: 0


        
        INFO 01-15 17:46:44 [executor_base.py:117] Maximum concurrency for 4096 tokens per request: 1.00x


        
        {'prefill_seq_len': 1, 'ctx_len': 4096, 'batch_size': 1, 'full_batch_size': 1, 'device_group': [0, 1], 'num_devices': 2, 'num_cores': 16, 'mxfp6_matmul': True, 'mxint8_kv_cache': True, 'aic_enable_depth_first': True, 'prefill_only': None, 'compile_only': False}



        INFO 01-15 17:46:44 [qaic.py:959] Using qpc:-/root/qpc/qpc


        
        Stages:  1


        
        INFO 01-15 17:46:44 [qserve_model_runner.py:66] Loading QPC...


        
        INFO 01-15 17:46:44 [qserve_model_runner.py:67] This may take some time, please don't press CTRL-C during this phase...



        INFO 01-15 17:46:52 [qserve_model_runner.py:88] Successfully loaded QPC in 7.791425864999837 secs



        WARNING 01-15 17:46:52 [qserve_model_runner.py:122] User entered `include_sampler`=True. But the provided QPC is not compiled to run sampling on the QAIC device. Falling back to the PyTorch backend.


        
        INFO 01-15 17:46:52 [qserve_model_runner.py:162] Warming-up with Dummy run...


        
        INFO 01-15 17:46:52 [qserve_model_runner.py:178] Successfully finished Dummy run in 0.04556767500025671 secs


        
        INFO 01-15 17:46:52 [llm_engine.py:444] init engine (profile, create kv cache, warmup model) took 7.86 seconds


        
        WARNING 01-15 17:46:53 [config.py:1251] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.


        
        INFO 01-15 17:46:53 [serving_chat.py:124] Using default chat sampling params from model: {'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}


        
        INFO 01-15 17:46:53 [serving_completion.py:61] Using default completion sampling params from model: {'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}


        
        INFO 01-15 17:46:53 [api_server.py:1090] Starting vLLM API server on http://0.0.0.0:8000


        
        INFO 01-15 17:46:53 [launcher.py:28] Available routes are:


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /openapi.json, Methods: GET, HEAD


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /docs, Methods: GET, HEAD


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /docs/oauth2-redirect, Methods: GET, HEAD


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /redoc, Methods: GET, HEAD


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /health, Methods: GET


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /load, Methods: GET


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /ping, Methods: POST, GET


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /tokenize, Methods: POST


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /detokenize, Methods: POST


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /v1/models, Methods: GET



        INFO 01-15 17:46:53 [launcher.py:36] Route: /version, Methods: GET


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /v1/chat/completions, Methods: POST


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /v1/completions, Methods: POST


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /v1/embeddings, Methods: POST


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /pooling, Methods: POST


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /score, Methods: POST


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /v1/score, Methods: POST


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /v1/audio/transcriptions, Methods: POST


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /rerank, Methods: POST


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /v1/rerank, Methods: POST


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /v2/rerank, Methods: POST


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /invocations, Methods: POST


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /metrics, Methods: GET


        
        INFO:     Started server process [1]


        
        INFO:     Waiting for application startup.


        
        INFO:     Application startup complete.

Exemplo de resultado de benchmarking para qaic-bench.

Exemplo de saída para qaic-bench


        dell@dell:~$ docker exec -it qaic-bench \

        /opt/vllm-env/bin/vllm bench serve \

        --model Qwen/Qwen3-30B-A3B-Instruct-2507 \

        --num-prompts 25 \

        --max-concurrency 1 \

        --random-input-len 256 \

        --random-output-len 256



        loading /opt/qti-aic/dev/lib/x86_64/libQAic.so


        
        INFO 01-15 17:47:44 [__init__.py:269] Automatically detected platform qaic.



        Namespace(subparser='bench', bench_type='serve', dispatch_function=<function BenchmarkServingSubcommand.cmd at 0x71abbd45e9e0>, endpoint_type='openai-comp', label=None, base_url=None, host='127.0.0.1', port=8000, endpoint='/v1/completions', dataset_name='random', max_concurrency=1, model='Qwen/Qwen3-30B-A3B-Instruct-2507', tokenizer=None, best_of=1, use_beam_search=False, num_prompts=25, logprobs=None, request_rate=inf, burstiness=1.0, seed=0, trust_remote_code=False, disable_tqdm=False, profile=False, save_result=False, metadata=None, result_dir=None, result_filename=None, ignore_eos=False, percentile_metrics='ttft,tpot,itl', metric_percentiles='99', goodput=None, random_input_len=256, random_output_len=256, random_range_ratio=1.0, random_prefix_len=0, tokenizer_mode='auto', served_model_name=None, lora_modules=None)



        Starting initial single prompt test run...


        
        Initial test run completed. Starting main benchmark run...


        
        Traffic request rate: inf


        
        Burstiness factor: 1.0 (Poisson process)


        
        Maximum request concurrency: 1



        100%|████████████████████████████████████████████████████████████████████████████████████| 25/25 [05:00<00:00, 12.02s/it]



        ============ Serving Benchmark Result ============

        Successful requests:                     25        

        Benchmark duration (s):                  300.53    

        Total input tokens:                      6400      

        Total generated tokens:                  6400      

        Request throughput (req/s):              0.08      

        Output token throughput (tok/s):         21.30     

        Total Token throughput (tok/s):          42.59     

        ---------------Time to First Token----------------

        Mean TTFT (ms):                          5915.88   

        Median TTFT (ms):                        5803.64   

        P99 TTFT (ms):                           7153.97   

        -----Time per Output Token (excl. 1st token)------

        Mean TPOT (ms):                          23.94     

        Median TPOT (ms):                        23.82     

        P99 TPOT (ms):                           25.18     

        ---------------Inter-token Latency----------------

        Mean ITL (ms):                           23.94     

        Median ITL (ms):                         23.78     

        P99 ITL (ms):                            25.77     

        ==================================================

        dell@dell:~$

Para calcular os resultados de TPS (tokens/segundo) para os tokens gerados por saída, use 1000 ÷ TPOT médio (23,94 de cima) = 41,8 TPS.

Voltar ao início

Affected Products

Dell Pro Max 16 Plus MB16250

Article Number: 000446730

Article Type: How To

Last Modified: 24 May 2026

Version: 5

Check if your device is covered by Support Services.

Como executar benchmarks usando vLLM no Dell Pro Max 16 Plus com a placa de inferência Qualcomm no Linux

Summary: Aprenda a executar referências de desempenho usando vLLM no Dell Pro Max 16 Plus com a placa de inferência Qualcomm no Linux.

Instructions

Affected Products

Instructions

Affected Products

Article Properties

Find answers to your questions from other Dell users

Support Services

Article Properties

Find answers to your questions from other Dell users

Support Services

Como executar benchmarks usando vLLM no Dell Pro Max 16 Plus com a placa de inferência Qualcomm no Linux

Summary: Aprenda a executar referências de desempenho usando vLLM no Dell Pro Max 16 Plus com a placa de inferência Qualcomm no Linux.

Detailed Article

Instructions

Affected Products

Instructions

Affected Products

Article Properties

Find answers to your questions from other Dell users

Support Services

Article Properties

Find answers to your questions from other Dell users

Support Services