Sådan kører du benchmarks ved hjælp af vLLM på Dell Pro Max 16 Plus med Qualcomm Inference Card i Linux

Summary: Få mere at vide om, hvordan du kører benchmarks ved hjælp af vLLM på Dell Pro Max 16 Plus med Qualcomm Inference Card i Linux.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Instructions

  1. Download en Qualcomm-programbeholder (QPC.)
    Download en færdigbygget AI-model QPC fra Zentrees forudkompilerede modelkatalog - Zentree-Qualcomm Pre-compiled Model Catalog for Cloud AI Accelerators Dette hyperlink fører dig til et websted uden for Dell Technologies.
    Bemærk: Sørg for, at QPC'en er kompatibel med Dell Pro Max ved at kigge efter SoCs / Tensor slicing = 2 
    tar -xzvf ~/Downloads/qpc_name.tar.gz
  2. Server Large Language Model (LLM) ved hjælp af vLLM.
    Åbn en terminal , og kør følgende kommando (erstat <QPC_DIR>, <MODEL_ID>, <CTX_LEN>, <BATCH_SIZE>og <PREFILL_SEQ_LEN>).
    Bemærk: At <MODEL_ID>, <CTX_LEN>, <BATCH_SIZE>og <PREFILL_SEQ_LEN> afhænger af QPC-kompileringskonfigurationen. For QPC'er, der er bygget og hostet af Zentree, dokumenteres disse værdier på deres hjemmeside. Kortlægningen er <BATCH_SIZE> med "Fuld batchstørrelse," <CTX_LEN> med "Kontekstlængde (CL)", og <PREFILL_SEQ_LEN> med "Chunking Prompt Length" fra Zentree-modellen QPC Configurations-tabellen. Bemærk også, at nogle modeller kræver, at du konfigurerer et Huggingface-token, som du skal passere ind i containeren for at få adgang. 
    
    docker run --rm -it \

    --name qaic-bench

    --device=/dev/accel/accel1 \

    --device=/dev/accel/accel2 \

    --network host \

    --ulimit nofile=1048576 \

    -e OMP_NUM_THREADS=8 \

    -e HF_TOKEN=$HF_TOKEN \

    -v ~/Downloads/<QPC_DIR>:/root/qpc \

    ghcr.io/quic/cloud_ai_inference_ubuntu24:1.20.6.0 \

    /opt/vllm-env/bin/vllm serve <MODEL_ID> \

    --max-num-seq <BATCH_SIZE> \

    --max-model-len <CTX_LEN> \

    --max-seq_len-to-capture <PREFILL_SEQ_LEN> \

    --device qaic \

    --device-group 0,1 \

    --quantization mxfp6 \

    --kv-cache-dtype mxint8 \

    --override-qaic-config "qpc_path=/root/qpc/qpc"
     
    Bemærk: Alle Zentree-hostede QPC'er til Dell Pro Max er bygget med num_devices=2, men hvis en model er bygget med num_devices=1Derefter --device-group skal indstilles til 0 eller 1.

    Alle Zentree-hostede QPC'er til Dell Pro Max er bygget med mxfp6_matmul=true, men hvis en model er bygget med mxfp6_matmul=falseDerefter --quantization mxfp6 bør droppes.

    Alle Zentree-hostede QPC'er til Dell Pro Max er bygget med mxint8_kv_cache=true, men hvis en model er bygget med mxint8_kv_cache=falseDerefter --kv-cache-dtype mxint8 bør droppes.

    Når du ser meddelelsen "Programstart fuldført", er modellen klar til slutning.

    For at stoppe vLLM (derfor aflæse modellen og lukke beholderen), tryk på Ctrl + C i dette terminalvindue . Det kan tage op til 1 minut at læsse større modeller af.

    Referencer: Docker – Brugervejledning Dette hyperlink fører dig til et websted uden for Dell Technologies.til Qualcomm Cloud AI SDK og vLLM – Brugervejledning Dette hyperlink fører dig til et websted uden for Dell Technologies.til Qualcomm Cloud AI SDK
     
  3. Kør benchmarking.
    Åbn en ny terminal , og kør følgende kommando (erstat <MODEL_ID> og <BATCH_SIZE> for at matche den model, der serveres i trin 2). Værdierne for num-prompts, random-input-lenog random-output-len kan justeres efter ønske. 
    
    docker exec -it qaic-bench \

    /opt/vllm-env/bin/vllm bench serve \

    --model  \

    --num-prompts 25 \

    --max-concurrency  \

    --random-input-len 256 \

    --random-output-len 256

    Når benchmarket er afsluttet, udskriver vLLM de resulterende målinger.
     
    Bemærk: Hvis du kun vil udlede gennemløb (tokens/sekund eller TPS) for afkodningsfasen (generering af token) eksklusive fasen for forhåndsudfyldning (hurtig behandling), kan du beregne TPS ved hjælp af 1000 ÷ Mean TPOT (ms).

    For eksempel svarer en gennemsnitlig TPOT på 20ms til en afkodningsgennemstrømning på 50 tokens / sekund (TPS).

    Arbejdseksempel:
    1. Download den færdigbyggede QPC Qwen3-30B-A3B-Instruct-2507.
      Til denne playbook bruger vi den forudbyggede QPC til Qwen3 30B A3B Instruct 2507 Dette hyperlink fører dig til et websted uden for Dell Technologies. .
      Der er 2 forudbyggede modeller tilgængelige, der fungerer på AI100-inferenskortet, der understøtter 2 SoC'er - en med en 4K og en anden med en 8K kontekstlængde. Til denne playbook bruges 2 SoC, 4K kontekstlængdemodellen.
      Følg nedenstående trin for at downloade og udpakke QPC fra arkivet. Bemærk, at hvis du ikke allerede har curl installeret, kan du installere det i terminalen ved hjælp af kommandoen (sudo apt install curl). 
      
        # Use existing Downloads folder for model downloads

        cd ~/Downloads

        # Download QPC archive from Zentree as qwen3-30b-a3b-instruct-2507.tar.gz

        curl -fSL https://dc00tk1pxen80.cloudfront.net/SDK1.20.4/Qwen/Qwen3-30B-A3B-Instruct-2507/

        Qwen3-30B-A3B-Instruct-2507_qpc_16cores_1pl_4096cl_1bs_2devices_mxfp6_mxint8.tar.gz -o qwen3-30b-a3b-instruct-2507.tar.gz

        # Extract QPC archive

        tar -xzvf qwen3-30b-a3b-instruct-2507.tar.gz

        # Rename extracted directory to qwen3-30b-a3b-instruct-2507-qpc

        mv $(tar -tzf qwen3-30b-a3b-instruct-2507.tar.gz | head -1 | cut -f1 -d"/") qwen3-30b-a3b-instruct-2507-qpc
    2. Server Qwen3:30B-modellen ved hjælp af vLLM.
      Åbn en terminal , og kør følgende kommando. Bemærk fra den generelle opskrift ovenfor, at <QPC_DIR> er kortlagt ved hjælp af "-v ~/Downloads/qwen3-30b-a3b-instruct-2507-qpc:/root/qpc" linje til QPC-mappen, som modellen downloades til, <MODEL_ID> er knyttet til "Qwen/Qwen3-30B-A3B-Instruct-2507," <CTX_LEN> er kortlagt til 4096, <BATCH_SIZE> er knyttet til 1, og <PREFILL_SEQ_LEN> er kortlagt til 1 (alle fra Zentree-modelsiden for Qwen3:30B-modellen ved hjælp af en 4K-kontekstlængde).

      Bemærk, at hvis du ikke tidligere har brugt denne version af AI100 Docker-beholderen, hentes ~14 GB, før du åbner Docker-beholderen. 
      
        docker run --rm -it \

        --name qaic-bench \

        --device=/dev/accel/accel1 \

        --device=/dev/accel/accel2 \

        --network host \

        --ulimit nofile=1048576 \

        -e OMP_NUM_THREADS=8 \

        -v ~/Downloads/qwen3-30b-a3b-instruct-2507-qpc:/root/qpc \

        ghcr.io/quic/cloud_ai_inference_ubuntu24:1.20.6.0 \

        /opt/vllm-env/bin/vllm serve Qwen/Qwen3-30B-A3B-Instruct-2507 \

        --max-num-seq 1 \

        --max-model-len 4096 \

        --max-seq_len-to-capture 1 \

        --device qaic \

        --device-group 0,1 \

        --quantization mxfp6 \

        --kv-cache-dtype mxint8 \

        --override-qaic-config "qpc_path=/root/qpc/qpc"

      Modellen er klar til slutning, når du ser meddelelsen "Application startup complete."
       
      Bemærk: For at stoppe vLLM (aflæsning af modellen og lukning af beholderen) skal du trykke på Ctrl + C i dette terminalvindue. Det kan tage op til 1 minut at læsse større modeller af.

      Referencer: Docker – Brugervejledning Dette hyperlink fører dig til et websted uden for Dell Technologies.til Qualcomm Cloud AI SDK og vLLM – Brugervejledning Dette hyperlink fører dig til et websted uden for Dell Technologies.til Qualcomm Cloud AI SDK

       
    3. Kør benchmarking.
      Åbn en ny terminal , og kør følgende kommando (erstat <MODEL_ID> For --model og <BATCH_SIZE> For --max-concurrency for at matche den model, der serveres i trin 2). Værdierne for num-prompts, random-input-lenog random-output-len kan justeres efter ønske. 
      
        docker exec -it qaic-bench \

        /opt/vllm-env/bin/vllm bench serve \

        --model Qwen/Qwen3-30B-A3B-Instruct-2507 \

        --num-prompts 25 \

        --max-concurrency 1 \

        --random-input-len 256 \

        --random-output-len 256
    4. Optagelse af prøveoutput.
      Prøveoutput til servering af Qwen3:30B ved hjælp af vLLM.
      Eksempel på output til vLLM: 
      
        dell@dell:~/Downloads/qwen3-30b-a3b-instruct-2507-qpc$ docker run --rm -it \

        --name qaic-bench \

        --device=/dev/accel/accel1 \

        --device=/dev/accel/accel2 \

        --network host \

        --ulimit nofile=1048576 \

        -e OMP_NUM_THREADS=8 \

        -v ~/Downloads/qwen3-30b-a3b-instruct-2507-qpc:/root/qpc \

        ghcr.io/quic/cloud_ai_inference_ubuntu24:1.20.6.0 \

        /opt/vllm-env/bin/vllm serve Qwen/Qwen3-30B-A3B-Instruct-2507 \

        --max-num-seq 1 \

        --max-model-len 4096 \

        --max-seq_len-to-capture 1 \

        --device qaic \

        --device-group 0,1 \

        --quantization mxfp6 \

        --kv-cache-dtype mxint8 \

        --override-qaic-config "qpc_path=/root/qpc/qpc"

        loading /opt/qti-aic/dev/lib/x86_64/libQAic.so

        INFO 01-15 17:46:28 [__init__.py:269] Automatically detected platform qaic.


        
        INFO 01-15 17:46:30 [api_server.py:1043] vLLM API server version 0.8.6.dev0+gba41cc90e.d20251229



        INFO 01-15 17:46:30 [api_server.py:1044] args: Namespace(subparser='serve', model_tag='Qwen/Qwen3-30B-A3B-Instruct-2507', config='', host=None, port=8000, uvicorn_log_level='info', disable_uvicorn_access_log=False, allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='Qwen/Qwen3-30B-A3B-Instruct-2507', task='auto', tokenizer=None, hf_config_path=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, load_format='auto', download_dir=None, model_loader_extra_config={}, use_tqdm_on_load=True, config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', max_model_len=4096, guided_decoding_backend='auto', reasoning_parser=None, logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, data_parallel_size=1, enable_expert_parallel=False, max_parallel_loading_workers=None, ray_workers_use_nsight=False, disable_custom_all_reduce=False, block_size=None, gpu_memory_utilization=0.9, swap_space=4, kv_cache_dtype='mxint8', num_gpu_blocks_override=None, enable_prefix_caching=None, prefix_caching_hash_algo='builtin', cpu_offload_gb=0, calculate_kv_scales=False, disable_sliding_window=False, use_v2_block_manager=True, seed=None, max_logprobs=20, disable_log_stats=False, quantization='mxfp6', rope_scaling=None, rope_theta=None, hf_token=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=1, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config={}, limit_mm_per_prompt={}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=None, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=None, max_prompt_adapters=1, max_prompt_adapter_token=0, device='qaic', speculative_config=None, ignore_patterns=[], served_model_name=None, qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, max_num_batched_tokens=None, max_num_seqs=1, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, num_lookahead_slots=0, scheduler_delay_factor=0.0, preemption_mode=None, num_scheduler_steps=1, multi_step_stream_outputs=True, scheduling_policy='fcfs', enable_chunked_prefill=None, disable_chunked_mm_input=False, scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', worker_extension_cls='', generation_config='auto', override_generation_config=None, enable_sleep_mode=False, additional_config=None, enable_reasoning=False, disable_cascade_attn=False, override_qaic_config={'qpc_path': '/root/qpc/qpc'}, device_group=[0, 1], disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, enable_server_load_tracking=False, dispatch_function=<function ServeSubcommand.cmd at 0x7416f976a710>)




        config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 963/963 [00:00<00:00, 6.38MB/s]


        
        INFO 01-15 17:46:34 [config.py:728] This model supports multiple tasks: {'score', 'generate', 'classify', 'embed', 'reward'}. Defaulting to 'generate'.



        WARNING 01-15 17:46:34 [arg_utils.py:1711] --kv-cache-dtype is not supported by the V1 Engine. Falling back to V0.


        
        INFO 01-15 17:46:34 [config.py:1415] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor


        
        INFO 01-15 17:46:34 [config.py:1820] Disabled the custom all-reduce kernel because it is not supported on current platform.


        
        config_1m.json: 77.3kB [00:00, 163MB/s]


        
        generation_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████| 239/239 [00:00<00:00, 2.47MB/s]


        
        model.safetensors.index.json: 1.70MB [00:00, 57.5MB/s]


        
        tokenizer.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 11.4M/11.4M [00:02<00:00, 5.23MB/s]


        
        tokenizer_config.json: 9.38kB [00:00, 41.9MB/s]


        
        vocab.json: 2.78MB [00:00, 25.4MB/s]


        
        INFO 01-15 17:46:38 [api_server.py:246] Started engine process with PID 51


        
        merges.txt: 1.67MB [00:00, 11.8MB/s]


        
        loading /opt/qti-aic/dev/lib/x86_64/libQAic.so


        
        INFO 01-15 17:46:40 [__init__.py:269] Automatically detected platform qaic.



        INFO 01-15 17:46:41 [llm_engine.py:247] Initializing a V0 LLM engine (v0.8.6.dev0+gba41cc90e.d20251229) with config: model='Qwen/Qwen3-30B-A3B-Instruct-2507', speculative_config=None, tokenizer='Qwen/Qwen3-30B-A3B-Instruct-2507', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=mxfp6, enforce_eager=False, kv_cache_dtype=mxint8,  device_config=cpu, decoding_config=DecodingConfig(guided_decoding_backend='auto', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=Qwen/Qwen3-30B-A3B-Instruct-2507, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=None, chunked_prefill_enabled=False, use_async_output_proc=False, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[1],"max_capture_size":1}, use_cached_outputs=True,



        WARNING 01-15 17:46:42 [qaic.py:100] Pin memory is not supported on Qaic.



        WARNING 01-15 17:46:42 [registry.py:217] Model class <class 'vllm.model_executor.models.mllama.MllamaForConditionalGeneration'> already has a multi-modal processor registered to <vllm.multimodal.registry.MultiModalRegistry object at 0x7f2ffb2ddc90>. It is overwritten by the new one.



        WARNING 01-15 17:46:42 [registry.py:217] Model class <class 'vllm.model_executor.models.mllama4.Llama4ForConditionalGeneration'> already has a multi-modal processor registered to <vllm.multimodal.registry.MultiModalRegistry object at 0x7f2ffb2ddc90>. It is overwritten by the new one.



        WARNING 01-15 17:46:42 [qaic_worker.py:102] Reducing Torch parallelism from 8 threads to 8 to avoid unnecessary CPU contention. Set VLLM_QAIC_MAX_CPU_THREADS to tune this value as needed.



        INFO 01-15 17:46:42 [qaic_worker.py:87] Setting RLIMIT_STACK to 16.0MB.


        
        INFO 01-15 17:46:44 [parallel_state.py:1004] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0


        
        loading /opt/qti-aic/dev/lib/x86_64/libQAic.so


        
        INFO 01-15 17:46:44 [executor_base.py:112] # qaic blocks: 1, # CPU blocks: 0


        
        INFO 01-15 17:46:44 [executor_base.py:117] Maximum concurrency for 4096 tokens per request: 1.00x


        
        {'prefill_seq_len': 1, 'ctx_len': 4096, 'batch_size': 1, 'full_batch_size': 1, 'device_group': [0, 1], 'num_devices': 2, 'num_cores': 16, 'mxfp6_matmul': True, 'mxint8_kv_cache': True, 'aic_enable_depth_first': True, 'prefill_only': None, 'compile_only': False}



        INFO 01-15 17:46:44 [qaic.py:959] Using qpc:-/root/qpc/qpc


        
        Stages:  1


        
        INFO 01-15 17:46:44 [qserve_model_runner.py:66] Loading QPC...


        
        INFO 01-15 17:46:44 [qserve_model_runner.py:67] This may take some time, please don't press CTRL-C during this phase...



        INFO 01-15 17:46:52 [qserve_model_runner.py:88] Successfully loaded QPC in 7.791425864999837 secs



        WARNING 01-15 17:46:52 [qserve_model_runner.py:122] User entered `include_sampler`=True. But the provided QPC is not compiled to run sampling on the QAIC device. Falling back to the PyTorch backend.


        
        INFO 01-15 17:46:52 [qserve_model_runner.py:162] Warming-up with Dummy run...


        
        INFO 01-15 17:46:52 [qserve_model_runner.py:178] Successfully finished Dummy run in 0.04556767500025671 secs


        
        INFO 01-15 17:46:52 [llm_engine.py:444] init engine (profile, create kv cache, warmup model) took 7.86 seconds


        
        WARNING 01-15 17:46:53 [config.py:1251] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.


        
        INFO 01-15 17:46:53 [serving_chat.py:124] Using default chat sampling params from model: {'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}


        
        INFO 01-15 17:46:53 [serving_completion.py:61] Using default completion sampling params from model: {'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}


        
        INFO 01-15 17:46:53 [api_server.py:1090] Starting vLLM API server on http://0.0.0.0:8000


        
        INFO 01-15 17:46:53 [launcher.py:28] Available routes are:


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /openapi.json, Methods: GET, HEAD


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /docs, Methods: GET, HEAD


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /docs/oauth2-redirect, Methods: GET, HEAD


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /redoc, Methods: GET, HEAD


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /health, Methods: GET


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /load, Methods: GET


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /ping, Methods: POST, GET


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /tokenize, Methods: POST


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /detokenize, Methods: POST


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /v1/models, Methods: GET



        INFO 01-15 17:46:53 [launcher.py:36] Route: /version, Methods: GET


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /v1/chat/completions, Methods: POST


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /v1/completions, Methods: POST


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /v1/embeddings, Methods: POST


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /pooling, Methods: POST


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /score, Methods: POST


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /v1/score, Methods: POST


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /v1/audio/transcriptions, Methods: POST


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /rerank, Methods: POST


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /v1/rerank, Methods: POST


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /v2/rerank, Methods: POST


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /invocations, Methods: POST


        
        INFO 01-15 17:46:53 [launcher.py:36] Route: /metrics, Methods: GET


        
        INFO:     Started server process [1]


        
        INFO:     Waiting for application startup.


        
        INFO:     Application startup complete.

      Prøve benchmarking output for qaic-bænk.

      Prøveoutput til QIC-bænk 
      
        dell@dell:~$ docker exec -it qaic-bench \

        /opt/vllm-env/bin/vllm bench serve \

        --model Qwen/Qwen3-30B-A3B-Instruct-2507 \

        --num-prompts 25 \

        --max-concurrency 1 \

        --random-input-len 256 \

        --random-output-len 256



        loading /opt/qti-aic/dev/lib/x86_64/libQAic.so


        
        INFO 01-15 17:47:44 [__init__.py:269] Automatically detected platform qaic.



        Namespace(subparser='bench', bench_type='serve', dispatch_function=<function BenchmarkServingSubcommand.cmd at 0x71abbd45e9e0>, endpoint_type='openai-comp', label=None, base_url=None, host='127.0.0.1', port=8000, endpoint='/v1/completions', dataset_name='random', max_concurrency=1, model='Qwen/Qwen3-30B-A3B-Instruct-2507', tokenizer=None, best_of=1, use_beam_search=False, num_prompts=25, logprobs=None, request_rate=inf, burstiness=1.0, seed=0, trust_remote_code=False, disable_tqdm=False, profile=False, save_result=False, metadata=None, result_dir=None, result_filename=None, ignore_eos=False, percentile_metrics='ttft,tpot,itl', metric_percentiles='99', goodput=None, random_input_len=256, random_output_len=256, random_range_ratio=1.0, random_prefix_len=0, tokenizer_mode='auto', served_model_name=None, lora_modules=None)



        Starting initial single prompt test run...


        
        Initial test run completed. Starting main benchmark run...


        
        Traffic request rate: inf


        
        Burstiness factor: 1.0 (Poisson process)


        
        Maximum request concurrency: 1



        100%|████████████████████████████████████████████████████████████████████████████████████| 25/25 [05:00<00:00, 12.02s/it]



        ============ Serving Benchmark Result ============

        Successful requests:                     25        

        Benchmark duration (s):                  300.53    

        Total input tokens:                      6400      

        Total generated tokens:                  6400      

        Request throughput (req/s):              0.08      

        Output token throughput (tok/s):         21.30     

        Total Token throughput (tok/s):          42.59     

        ---------------Time to First Token----------------

        Mean TTFT (ms):                          5915.88   

        Median TTFT (ms):                        5803.64   

        P99 TTFT (ms):                           7153.97   

        -----Time per Output Token (excl. 1st token)------

        Mean TPOT (ms):                          23.94     

        Median TPOT (ms):                        23.82     

        P99 TPOT (ms):                           25.18     

        ---------------Inter-token Latency----------------

        Mean ITL (ms):                           23.94     

        Median ITL (ms):                         23.78     

        P99 ITL (ms):                            25.77     

        ==================================================

        dell@dell:~$

      For at beregne TPS-resultaterne (tokens/sekund) for de outputgenererede tokens skal du bruge 1000 ÷ gennemsnitlige TPOT (23,94 ovenfra) = 41,8 TPS.

Tilbage til toppen

Affected Products

Dell Pro Max 16 Plus MB16250
Article Properties
Article Number: 000446730
Article Type: How To
Last Modified: 24 May 2026
Version:  5
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.