Come eseguire i benchmark utilizzando vLLM su Dell Pro Max 16 Plus con la scheda di inferenza Qualcomm in Linux
Summary: Informazioni su come eseguire benchmark utilizzando vLLM su Dell Pro Max 16 Plus con la scheda di inferenza Qualcomm in Linux.
This article applies to
This article does not apply to
This article is not tied to any specific product.
Not all product versions are identified in this article.
Instructions
- Scaricare un contenitore di programma Qualcomm (QPC).
Scarica un modello AI QPC precompilato dal catalogo dei modelli precompilato di Zentree - Zentree-Qualcomm Catalogo dei modelli precompilato per gli acceleratoriAI cloud
Nota: Accertarsi che il QPC sia compatibile con Dell Pro Max cercandoSoCs / Tensor slicing = 2tar -xzvf ~/Downloads/qpc_name.tar.gz - Serve LLM (Large Language Model) utilizzando vLLM.
Aprire un terminale ed eseguire il seguente comando (sostituire<QPC_DIR>,<MODEL_ID>,<CTX_LEN>,<BATCH_SIZE>e<PREFILL_SEQ_LEN>).Nota: Quella<MODEL_ID>,<CTX_LEN>,<BATCH_SIZE>e<PREFILL_SEQ_LEN>dipendono dalla configurazione di compilazione QPC. Per i QPC creati e ospitati da Zentree, questi valori sono documentati sul loro sito web. Il mapping è<BATCH_SIZE>con "Full Batch Size",<CTX_LEN>con "Context Length (CL)," e<PREFILL_SEQ_LEN>con "Chunking Prompt Length" dalla tabella delle configurazioni QPC del modello Zentree. Si noti inoltre che alcuni modelli richiedono la configurazione di un token Huggingface che sarà necessario passare nel contenitore per accedere.docker run --rm -it \ --name qaic-bench --device=/dev/accel/accel1 \ --device=/dev/accel/accel2 \ --network host \ --ulimit nofile=1048576 \ -e OMP_NUM_THREADS=8 \ -e HF_TOKEN=$HF_TOKEN \ -v ~/Downloads/<QPC_DIR>:/root/qpc \ ghcr.io/quic/cloud_ai_inference_ubuntu24:1.20.6.0 \ /opt/vllm-env/bin/vllm serve <MODEL_ID> \ --max-num-seq <BATCH_SIZE> \ --max-model-len <CTX_LEN> \ --max-seq_len-to-capture <PREFILL_SEQ_LEN> \ --device qaic \ --device-group 0,1 \ --quantization mxfp6 \ --kv-cache-dtype mxint8 \ --override-qaic-config "qpc_path=/root/qpc/qpc"Nota: Tutti i QPC ospitati su Zentree per Dell Pro Max sono realizzati con:num_devices=2, ma se un modello è compilato connum_devices=1Poi--device-groupdovrebbe essere impostato su0oppure1.
Tutti i QPC ospitati su Zentree per Dell Pro Max sono realizzati con:mxfp6_matmul=true, ma se un modello è compilato conmxfp6_matmul=falsePoi--quantization mxfp6dovrebbe essere abbandonato.
Tutti i QPC ospitati su Zentree per Dell Pro Max sono realizzati con:mxint8_kv_cache=true, ma se un modello è compilato conmxint8_kv_cache=falsePoi--kv-cache-dtype mxint8dovrebbe essere abbandonato.
Una volta visualizzato il messaggio "Application startup complete", il modello è pronto per l'inferenza.
Per arrestare vLLM (quindi scaricare il modello e chiudere il contenitore), premere Ctrl+C in questa finestra del Terminale . Lo scaricamento dei modelli più grandi può richiedere fino a 1 minuto.
Riferimenti: Docker: Guida
- Eseguire il benchmarking.
Apri un nuovo Terminale ed esegui il seguente comando (sostituisci<MODEL_ID>e<BATCH_SIZE>in modo che corrisponda al modello servito nel passaggio 2). I valori pernum-prompts,random-input-lenerandom-output-lenpuò essere regolato a piacere.docker exec -it qaic-bench \ /opt/vllm-env/bin/vllm bench serve \ --model \ --num-prompts 25 \ --max-concurrency \ --random-input-len 256 \ --random-output-len 256
Una volta completato il benchmark, vLLM stampa le metriche risultanti.
Nota: Per ricavare il throughput (token/secondo o TPS) solo per la fase di decodifica (generazione di token) esclusa la fase di precompilazione (elaborazione di prompt), è possibile calcolare TPS utilizzando 1000 ÷ TPOT medio (ms).
Ad esempio, un TPOT medio di 20 ms equivale a un throughput di decodifica di 50 token al secondo (TPS).
Esempio funzionante:- Scaricare il QPC precostruito Qwen3-30B-A3B-Instruct-2507.
Per questo playbook utilizziamo il QPC precostruito per Qwen3 30B A3B Instruct 2507
Sono disponibili 2 modelli predefiniti che funzionano sulla scheda di inferenza AI100 che supporta 2 SoC, uno con 4K e un altro con una lunghezza di contesto di 8K. Per questo playbook viene utilizzato il modello di lunghezza del contesto 2 SoC, 4K.
Seguire la procedura riportata di seguito per scaricare ed estrarre il QPC dall'archivio. Si noti che se non si dispone di curl già installato, è possibile installarlo nel terminale utilizzando il comando (sudo apt install curl).# Use existing Downloads folder for model downloads cd ~/Downloads # Download QPC archive from Zentree as qwen3-30b-a3b-instruct-2507.tar.gz curl -fSL https://dc00tk1pxen80.cloudfront.net/SDK1.20.4/Qwen/Qwen3-30B-A3B-Instruct-2507/ Qwen3-30B-A3B-Instruct-2507_qpc_16cores_1pl_4096cl_1bs_2devices_mxfp6_mxint8.tar.gz -o qwen3-30b-a3b-instruct-2507.tar.gz # Extract QPC archive tar -xzvf qwen3-30b-a3b-instruct-2507.tar.gz # Rename extracted directory to qwen3-30b-a3b-instruct-2507-qpc mv $(tar -tzf qwen3-30b-a3b-instruct-2507.tar.gz | head -1 | cut -f1 -d"/") qwen3-30b-a3b-instruct-2507-qpc - Servire il modello Qwen3:30B utilizzando vLLM.
Apri un terminale ed esegui il seguente comando. Nota dalla ricetta generale sopra che<QPC_DIR>è mappato utilizzando il "-v ~/Downloads/qwen3-30b-a3b-instruct-2507-qpc:/root/qpc" alla directory QPC in cui viene scaricato il modello,<MODEL_ID>è mappato a "Qwen/Qwen3-30B-A3B-Instruct-2507,"<CTX_LEN>è mappato a 4096,<BATCH_SIZE>è mappato a 1 e<PREFILL_SEQ_LEN>è mappato a 1 (tutto dalla pagina del modello Zentree per il modello Qwen3:30B utilizzando una lunghezza di contesto 4K).
Si noti che se non è stata utilizzata in precedenza questa versione del contenitore Docker AI100, vengono recuperati ~14 GB prima di aprire il contenitore Docker.docker run --rm -it \ --name qaic-bench \ --device=/dev/accel/accel1 \ --device=/dev/accel/accel2 \ --network host \ --ulimit nofile=1048576 \ -e OMP_NUM_THREADS=8 \ -v ~/Downloads/qwen3-30b-a3b-instruct-2507-qpc:/root/qpc \ ghcr.io/quic/cloud_ai_inference_ubuntu24:1.20.6.0 \ /opt/vllm-env/bin/vllm serve Qwen/Qwen3-30B-A3B-Instruct-2507 \ --max-num-seq 1 \ --max-model-len 4096 \ --max-seq_len-to-capture 1 \ --device qaic \ --device-group 0,1 \ --quantization mxfp6 \ --kv-cache-dtype mxint8 \ --override-qaic-config "qpc_path=/root/qpc/qpc"
Il modello è pronto per l'inferenza una volta visualizzato il messaggio "Application startup complete".
Nota: Per arrestare vLLM (scarico del modello e chiusura del contenitore), premere Ctrl+C in questa finestra del terminale. Lo scaricamento dei modelli più grandi può richiedere fino a 1 minuto.
Riferimenti: Docker: Guida
- Eseguire il benchmarking.
Apri un nuovo Terminale ed esegui il seguente comando (sostituisci<MODEL_ID>Per--modele<BATCH_SIZE>Per--max-concurrencyin modo che corrisponda al modello servito nel passaggio 2). I valori pernum-prompts,random-input-lenerandom-output-lenpuò essere regolato a piacere.docker exec -it qaic-bench \ /opt/vllm-env/bin/vllm bench serve \ --model Qwen/Qwen3-30B-A3B-Instruct-2507 \ --num-prompts 25 \ --max-concurrency 1 \ --random-input-len 256 \ --random-output-len 256 - Acquisizione dell'output di esempio.
Output di esempio per la distribuzione di Qwen3:30B con vLLM.
Output di esempio per vLLM:dell@dell:~/Downloads/qwen3-30b-a3b-instruct-2507-qpc$ docker run --rm -it \ --name qaic-bench \ --device=/dev/accel/accel1 \ --device=/dev/accel/accel2 \ --network host \ --ulimit nofile=1048576 \ -e OMP_NUM_THREADS=8 \ -v ~/Downloads/qwen3-30b-a3b-instruct-2507-qpc:/root/qpc \ ghcr.io/quic/cloud_ai_inference_ubuntu24:1.20.6.0 \ /opt/vllm-env/bin/vllm serve Qwen/Qwen3-30B-A3B-Instruct-2507 \ --max-num-seq 1 \ --max-model-len 4096 \ --max-seq_len-to-capture 1 \ --device qaic \ --device-group 0,1 \ --quantization mxfp6 \ --kv-cache-dtype mxint8 \ --override-qaic-config "qpc_path=/root/qpc/qpc" loading /opt/qti-aic/dev/lib/x86_64/libQAic.so INFO 01-15 17:46:28 [__init__.py:269] Automatically detected platform qaic. INFO 01-15 17:46:30 [api_server.py:1043] vLLM API server version 0.8.6.dev0+gba41cc90e.d20251229 INFO 01-15 17:46:30 [api_server.py:1044] args: Namespace(subparser='serve', model_tag='Qwen/Qwen3-30B-A3B-Instruct-2507', config='', host=None, port=8000, uvicorn_log_level='info', disable_uvicorn_access_log=False, allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='Qwen/Qwen3-30B-A3B-Instruct-2507', task='auto', tokenizer=None, hf_config_path=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, load_format='auto', download_dir=None, model_loader_extra_config={}, use_tqdm_on_load=True, config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', max_model_len=4096, guided_decoding_backend='auto', reasoning_parser=None, logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, data_parallel_size=1, enable_expert_parallel=False, max_parallel_loading_workers=None, ray_workers_use_nsight=False, disable_custom_all_reduce=False, block_size=None, gpu_memory_utilization=0.9, swap_space=4, kv_cache_dtype='mxint8', num_gpu_blocks_override=None, enable_prefix_caching=None, prefix_caching_hash_algo='builtin', cpu_offload_gb=0, calculate_kv_scales=False, disable_sliding_window=False, use_v2_block_manager=True, seed=None, max_logprobs=20, disable_log_stats=False, quantization='mxfp6', rope_scaling=None, rope_theta=None, hf_token=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=1, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config={}, limit_mm_per_prompt={}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=None, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=None, max_prompt_adapters=1, max_prompt_adapter_token=0, device='qaic', speculative_config=None, ignore_patterns=[], served_model_name=None, qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, max_num_batched_tokens=None, max_num_seqs=1, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, num_lookahead_slots=0, scheduler_delay_factor=0.0, preemption_mode=None, num_scheduler_steps=1, multi_step_stream_outputs=True, scheduling_policy='fcfs', enable_chunked_prefill=None, disable_chunked_mm_input=False, scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', worker_extension_cls='', generation_config='auto', override_generation_config=None, enable_sleep_mode=False, additional_config=None, enable_reasoning=False, disable_cascade_attn=False, override_qaic_config={'qpc_path': '/root/qpc/qpc'}, device_group=[0, 1], disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, enable_server_load_tracking=False, dispatch_function=<function ServeSubcommand.cmd at 0x7416f976a710>) config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 963/963 [00:00<00:00, 6.38MB/s] INFO 01-15 17:46:34 [config.py:728] This model supports multiple tasks: {'score', 'generate', 'classify', 'embed', 'reward'}. Defaulting to 'generate'. WARNING 01-15 17:46:34 [arg_utils.py:1711] --kv-cache-dtype is not supported by the V1 Engine. Falling back to V0. INFO 01-15 17:46:34 [config.py:1415] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor INFO 01-15 17:46:34 [config.py:1820] Disabled the custom all-reduce kernel because it is not supported on current platform. config_1m.json: 77.3kB [00:00, 163MB/s] generation_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████| 239/239 [00:00<00:00, 2.47MB/s] model.safetensors.index.json: 1.70MB [00:00, 57.5MB/s] tokenizer.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 11.4M/11.4M [00:02<00:00, 5.23MB/s] tokenizer_config.json: 9.38kB [00:00, 41.9MB/s] vocab.json: 2.78MB [00:00, 25.4MB/s] INFO 01-15 17:46:38 [api_server.py:246] Started engine process with PID 51 merges.txt: 1.67MB [00:00, 11.8MB/s] loading /opt/qti-aic/dev/lib/x86_64/libQAic.so INFO 01-15 17:46:40 [__init__.py:269] Automatically detected platform qaic. INFO 01-15 17:46:41 [llm_engine.py:247] Initializing a V0 LLM engine (v0.8.6.dev0+gba41cc90e.d20251229) with config: model='Qwen/Qwen3-30B-A3B-Instruct-2507', speculative_config=None, tokenizer='Qwen/Qwen3-30B-A3B-Instruct-2507', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=mxfp6, enforce_eager=False, kv_cache_dtype=mxint8, device_config=cpu, decoding_config=DecodingConfig(guided_decoding_backend='auto', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=Qwen/Qwen3-30B-A3B-Instruct-2507, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=None, chunked_prefill_enabled=False, use_async_output_proc=False, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[1],"max_capture_size":1}, use_cached_outputs=True, WARNING 01-15 17:46:42 [qaic.py:100] Pin memory is not supported on Qaic. WARNING 01-15 17:46:42 [registry.py:217] Model class <class 'vllm.model_executor.models.mllama.MllamaForConditionalGeneration'> already has a multi-modal processor registered to <vllm.multimodal.registry.MultiModalRegistry object at 0x7f2ffb2ddc90>. It is overwritten by the new one. WARNING 01-15 17:46:42 [registry.py:217] Model class <class 'vllm.model_executor.models.mllama4.Llama4ForConditionalGeneration'> already has a multi-modal processor registered to <vllm.multimodal.registry.MultiModalRegistry object at 0x7f2ffb2ddc90>. It is overwritten by the new one. WARNING 01-15 17:46:42 [qaic_worker.py:102] Reducing Torch parallelism from 8 threads to 8 to avoid unnecessary CPU contention. Set VLLM_QAIC_MAX_CPU_THREADS to tune this value as needed. INFO 01-15 17:46:42 [qaic_worker.py:87] Setting RLIMIT_STACK to 16.0MB. INFO 01-15 17:46:44 [parallel_state.py:1004] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0 loading /opt/qti-aic/dev/lib/x86_64/libQAic.so INFO 01-15 17:46:44 [executor_base.py:112] # qaic blocks: 1, # CPU blocks: 0 INFO 01-15 17:46:44 [executor_base.py:117] Maximum concurrency for 4096 tokens per request: 1.00x {'prefill_seq_len': 1, 'ctx_len': 4096, 'batch_size': 1, 'full_batch_size': 1, 'device_group': [0, 1], 'num_devices': 2, 'num_cores': 16, 'mxfp6_matmul': True, 'mxint8_kv_cache': True, 'aic_enable_depth_first': True, 'prefill_only': None, 'compile_only': False} INFO 01-15 17:46:44 [qaic.py:959] Using qpc:-/root/qpc/qpc Stages: 1 INFO 01-15 17:46:44 [qserve_model_runner.py:66] Loading QPC... INFO 01-15 17:46:44 [qserve_model_runner.py:67] This may take some time, please don't press CTRL-C during this phase... INFO 01-15 17:46:52 [qserve_model_runner.py:88] Successfully loaded QPC in 7.791425864999837 secs WARNING 01-15 17:46:52 [qserve_model_runner.py:122] User entered `include_sampler`=True. But the provided QPC is not compiled to run sampling on the QAIC device. Falling back to the PyTorch backend. INFO 01-15 17:46:52 [qserve_model_runner.py:162] Warming-up with Dummy run... INFO 01-15 17:46:52 [qserve_model_runner.py:178] Successfully finished Dummy run in 0.04556767500025671 secs INFO 01-15 17:46:52 [llm_engine.py:444] init engine (profile, create kv cache, warmup model) took 7.86 seconds WARNING 01-15 17:46:53 [config.py:1251] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. INFO 01-15 17:46:53 [serving_chat.py:124] Using default chat sampling params from model: {'temperature': 0.7, 'top_k': 20, 'top_p': 0.8} INFO 01-15 17:46:53 [serving_completion.py:61] Using default completion sampling params from model: {'temperature': 0.7, 'top_k': 20, 'top_p': 0.8} INFO 01-15 17:46:53 [api_server.py:1090] Starting vLLM API server on http://0.0.0.0:8000 INFO 01-15 17:46:53 [launcher.py:28] Available routes are: INFO 01-15 17:46:53 [launcher.py:36] Route: /openapi.json, Methods: GET, HEAD INFO 01-15 17:46:53 [launcher.py:36] Route: /docs, Methods: GET, HEAD INFO 01-15 17:46:53 [launcher.py:36] Route: /docs/oauth2-redirect, Methods: GET, HEAD INFO 01-15 17:46:53 [launcher.py:36] Route: /redoc, Methods: GET, HEAD INFO 01-15 17:46:53 [launcher.py:36] Route: /health, Methods: GET INFO 01-15 17:46:53 [launcher.py:36] Route: /load, Methods: GET INFO 01-15 17:46:53 [launcher.py:36] Route: /ping, Methods: POST, GET INFO 01-15 17:46:53 [launcher.py:36] Route: /tokenize, Methods: POST INFO 01-15 17:46:53 [launcher.py:36] Route: /detokenize, Methods: POST INFO 01-15 17:46:53 [launcher.py:36] Route: /v1/models, Methods: GET INFO 01-15 17:46:53 [launcher.py:36] Route: /version, Methods: GET INFO 01-15 17:46:53 [launcher.py:36] Route: /v1/chat/completions, Methods: POST INFO 01-15 17:46:53 [launcher.py:36] Route: /v1/completions, Methods: POST INFO 01-15 17:46:53 [launcher.py:36] Route: /v1/embeddings, Methods: POST INFO 01-15 17:46:53 [launcher.py:36] Route: /pooling, Methods: POST INFO 01-15 17:46:53 [launcher.py:36] Route: /score, Methods: POST INFO 01-15 17:46:53 [launcher.py:36] Route: /v1/score, Methods: POST INFO 01-15 17:46:53 [launcher.py:36] Route: /v1/audio/transcriptions, Methods: POST INFO 01-15 17:46:53 [launcher.py:36] Route: /rerank, Methods: POST INFO 01-15 17:46:53 [launcher.py:36] Route: /v1/rerank, Methods: POST INFO 01-15 17:46:53 [launcher.py:36] Route: /v2/rerank, Methods: POST INFO 01-15 17:46:53 [launcher.py:36] Route: /invocations, Methods: POST INFO 01-15 17:46:53 [launcher.py:36] Route: /metrics, Methods: GET INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete.
Esempio di output di benchmarking per qaic-bench.
Esempio di output per qaic-benchdell@dell:~$ docker exec -it qaic-bench \ /opt/vllm-env/bin/vllm bench serve \ --model Qwen/Qwen3-30B-A3B-Instruct-2507 \ --num-prompts 25 \ --max-concurrency 1 \ --random-input-len 256 \ --random-output-len 256 loading /opt/qti-aic/dev/lib/x86_64/libQAic.so INFO 01-15 17:47:44 [__init__.py:269] Automatically detected platform qaic. Namespace(subparser='bench', bench_type='serve', dispatch_function=<function BenchmarkServingSubcommand.cmd at 0x71abbd45e9e0>, endpoint_type='openai-comp', label=None, base_url=None, host='127.0.0.1', port=8000, endpoint='/v1/completions', dataset_name='random', max_concurrency=1, model='Qwen/Qwen3-30B-A3B-Instruct-2507', tokenizer=None, best_of=1, use_beam_search=False, num_prompts=25, logprobs=None, request_rate=inf, burstiness=1.0, seed=0, trust_remote_code=False, disable_tqdm=False, profile=False, save_result=False, metadata=None, result_dir=None, result_filename=None, ignore_eos=False, percentile_metrics='ttft,tpot,itl', metric_percentiles='99', goodput=None, random_input_len=256, random_output_len=256, random_range_ratio=1.0, random_prefix_len=0, tokenizer_mode='auto', served_model_name=None, lora_modules=None) Starting initial single prompt test run... Initial test run completed. Starting main benchmark run... Traffic request rate: inf Burstiness factor: 1.0 (Poisson process) Maximum request concurrency: 1 100%|████████████████████████████████████████████████████████████████████████████████████| 25/25 [05:00<00:00, 12.02s/it] ============ Serving Benchmark Result ============ Successful requests: 25 Benchmark duration (s): 300.53 Total input tokens: 6400 Total generated tokens: 6400 Request throughput (req/s): 0.08 Output token throughput (tok/s): 21.30 Total Token throughput (tok/s): 42.59 ---------------Time to First Token---------------- Mean TTFT (ms): 5915.88 Median TTFT (ms): 5803.64 P99 TTFT (ms): 7153.97 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 23.94 Median TPOT (ms): 23.82 P99 TPOT (ms): 25.18 ---------------Inter-token Latency---------------- Mean ITL (ms): 23.94 Median ITL (ms): 23.78 P99 ITL (ms): 25.77 ================================================== dell@dell:~$
Per calcolare i risultati TPS (token/secondo) per i token generati dall'output, utilizzare 1000 ÷ TPOT medio (23,94 dall'alto) = 41,8 TPS.
- Scaricare il QPC precostruito Qwen3-30B-A3B-Instruct-2507.
Affected Products
Dell Pro Max 16 Plus MB16250Article Properties
Article Number: 000446730
Article Type: How To
Last Modified: 24 May 2026
Version: 5
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.