Linux에서 Qualcomm 추론 카드와 함께 Dell Pro Max 16 Plus에서 vLLM을 사용하여 벤치마크를 실행하는 방법
Summary: Linux에서 Qualcomm 추론 카드와 함께 Dell Pro Max 16 Plus에서 vLLM을 사용하여 벤치마크를 실행하는 방법을 알아보십시오.
This article applies to
This article does not apply to
This article is not tied to any specific product.
Not all product versions are identified in this article.
Instructions
- QPC(Qualcomm Program Container)를 다운로드합니다.
Zentree의 사전 컴파일된 모델 카탈로그에서 사전 구축된 AI 모델 QPC 다운로드 - Zentree-Qualcomm 클라우드 AI 가속기를 위한 사전 컴파일된 모델 카탈로그
참고: QPC가 Dell Pro Max와 호환되는지 확인하려면SoCs / Tensor slicing = 2tar -xzvf ~/Downloads/qpc_name.tar.gz - vLLM을 사용하여 LLM(Large Language Model)을 제공합니다.
터미널을 열고 다음 명령을 실행합니다(<QPC_DIR>,<MODEL_ID>,<CTX_LEN>,<BATCH_SIZE>및<PREFILL_SEQ_LEN>)를 제공해야 합니다.참고: 저것<MODEL_ID>,<CTX_LEN>,<BATCH_SIZE>및<PREFILL_SEQ_LEN>QPC 컴파일 구성에 따라 다릅니다. Zentree에서 구축하고 호스팅하는 QPC의 경우 이러한 값은 웹 사이트에 문서화되어 있습니다. 매핑은 다음과 같습니다.<BATCH_SIZE>"전체 배치 크기"를 사용하면<CTX_LEN>"컨텍스트 길이(CL)"와 함께<PREFILL_SEQ_LEN>Zentree 모델 QPC 구성 테이블의 "청킹 프롬프트 길이"를 사용합니다. 또한 일부 모델에서는 액세스하기 위해 컨테이너에 전달해야 하는 Huggingface 토큰을 설정해야 합니다.docker run --rm -it \ --name qaic-bench --device=/dev/accel/accel1 \ --device=/dev/accel/accel2 \ --network host \ --ulimit nofile=1048576 \ -e OMP_NUM_THREADS=8 \ -e HF_TOKEN=$HF_TOKEN \ -v ~/Downloads/<QPC_DIR>:/root/qpc \ ghcr.io/quic/cloud_ai_inference_ubuntu24:1.20.6.0 \ /opt/vllm-env/bin/vllm serve <MODEL_ID> \ --max-num-seq <BATCH_SIZE> \ --max-model-len <CTX_LEN> \ --max-seq_len-to-capture <PREFILL_SEQ_LEN> \ --device qaic \ --device-group 0,1 \ --quantization mxfp6 \ --kv-cache-dtype mxint8 \ --override-qaic-config "qpc_path=/root/qpc/qpc"참고: 모든 Dell Pro Max용 Zentree 호스팅 QPC는num_devices=2하지만 모델이num_devices=1다음--device-group로 설정해야 합니다.0또는1을 클릭합니다.
모든 Dell Pro Max용 Zentree 호스팅 QPC는mxfp6_matmul=true하지만 모델이mxfp6_matmul=false다음--quantization mxfp6삭제해야 합니다.
모든 Dell Pro Max용 Zentree 호스팅 QPC는mxint8_kv_cache=true하지만 모델이mxint8_kv_cache=false다음--kv-cache-dtype mxint8삭제해야 합니다.
"응용 프로그램 시작 완료"라는 메시지가 표시되면 모델을 유추할 준비가 된 것입니다.
vLLM을 중지하려면(따라서 모델을 언로드하고 컨테이너를 닫으려면) 이 터미널 창에서 Ctrl+C를 누릅니다. 더 큰 모델은 언로드하는 데 최대 1분이 걸릴 수 있습니다.
참고 자료: Docker - Qualcomm Cloud AI SDK 사용자 가이드
- 벤치마킹을 실행합니다.
새 터미널 을 열고 다음 명령을 실행합니다(<MODEL_ID>및<BATCH_SIZE>2단계에서 제공된 모델과 일치하도록 함). 에 대한 값num-prompts,random-input-len및random-output-len원하는대로 조정할 수 있습니다.docker exec -it qaic-bench \ /opt/vllm-env/bin/vllm bench serve \ --model \ --num-prompts 25 \ --max-concurrency \ --random-input-len 256 \ --random-output-len 256
벤치마크가 완료되면 vLLM이 결과 메트릭을 출력합니다.
참고: 미리 채우기(프롬프트 처리) 단계를 제외한 디코딩(토큰 생성) 단계에서만 처리량(토큰/초 또는 TPS)을 도출하려면 1000 ÷ 평균 TPOT(ms)를 사용하여 TPS를 계산할 수 있습니다.
예를 들어, 20ms의 평균 TPOT는 50 토큰/초(TPS)의 디코딩 처리량과 같습니다.
작업 예 :- Qwen3-30B-A3B-Instruct-2507 사전 구축된 QPC를 다운로드합니다.
이 플레이북에서는 Qwen3 30B A3B Instruct 2507
2개의 SoC를 지원하는 AI100 추론 카드에서 작동하는 사전 구축된 모델은 2개(하나는 4K, 다른 하나는 컨텍스트 길이가 8K)입니다. 이 플레이북에는 2개의 SoC, 4K 컨텍스트 길이 모델이 사용됩니다.
아래 단계에 따라 아카이브에서 QPC를 다운로드하고 압축을 풉니다. curl이 아직 설치되어 있지 않은 경우 터미널에서 다음 명령(sudo apt install curl)를 제공해야 합니다.# Use existing Downloads folder for model downloads cd ~/Downloads # Download QPC archive from Zentree as qwen3-30b-a3b-instruct-2507.tar.gz curl -fSL https://dc00tk1pxen80.cloudfront.net/SDK1.20.4/Qwen/Qwen3-30B-A3B-Instruct-2507/ Qwen3-30B-A3B-Instruct-2507_qpc_16cores_1pl_4096cl_1bs_2devices_mxfp6_mxint8.tar.gz -o qwen3-30b-a3b-instruct-2507.tar.gz # Extract QPC archive tar -xzvf qwen3-30b-a3b-instruct-2507.tar.gz # Rename extracted directory to qwen3-30b-a3b-instruct-2507-qpc mv $(tar -tzf qwen3-30b-a3b-instruct-2507.tar.gz | head -1 | cut -f1 -d"/") qwen3-30b-a3b-instruct-2507-qpc - vLLM을 사용하여 Qwen3:30B 모델을 제공합니다.
터미널을 열고 다음 명령을 실행합니다. 위의 일반 레시피에서 다음을 참고하십시오.<QPC_DIR>는 "를 사용하여 매핑됩니다.-v ~/Downloads/qwen3-30b-a3b-instruct-2507-qpc:/root/qpc" 줄을 모델이 다운로드되는 QPC 디렉토리에 추가하고,<MODEL_ID>"에 매핑됩니다.Qwen/Qwen3-30B-A3B-Instruct-2507,"<CTX_LEN>4096에 매핑되고,<BATCH_SIZE>는 1에 매핑되고,<PREFILL_SEQ_LEN>는 1에 매핑됩니다(모두 4K 컨텍스트 길이를 사용하는 Qwen3:30B 모델에 대한 Zentree 모델 페이지에서 가져옴).
이전에 이 버전의 AI100 Docker 컨테이너를 사용하지 않은 경우 Docker 컨테이너를 열기 전에 ~14GB를 가져옵니다.docker run --rm -it \ --name qaic-bench \ --device=/dev/accel/accel1 \ --device=/dev/accel/accel2 \ --network host \ --ulimit nofile=1048576 \ -e OMP_NUM_THREADS=8 \ -v ~/Downloads/qwen3-30b-a3b-instruct-2507-qpc:/root/qpc \ ghcr.io/quic/cloud_ai_inference_ubuntu24:1.20.6.0 \ /opt/vllm-env/bin/vllm serve Qwen/Qwen3-30B-A3B-Instruct-2507 \ --max-num-seq 1 \ --max-model-len 4096 \ --max-seq_len-to-capture 1 \ --device qaic \ --device-group 0,1 \ --quantization mxfp6 \ --kv-cache-dtype mxint8 \ --override-qaic-config "qpc_path=/root/qpc/qpc"
"라는 메시지가 표시되면 모델은 추론할 준비가 된 것입니다.Application startup complete"로 변경되었습니다.
참고: vLLM을 중지하려면(모델 언로드 및 컨테이너 닫기) 이 터미널 창에서 Ctrl+C를 누릅니다. 더 큰 모델은 언로드하는 데 최대 1분이 걸릴 수 있습니다.
참조: Docker - Qualcomm Cloud AI SDK 사용자 가이드
- 벤치마킹을 실행합니다.
새 터미널 을 열고 다음 명령을 실행합니다(<MODEL_ID>예를 들어,--model및<BATCH_SIZE>예를 들어,--max-concurrency2단계에서 제공된 모델과 일치하도록 함). 에 대한 값num-prompts,random-input-len및random-output-len원하는대로 조정할 수 있습니다.docker exec -it qaic-bench \ /opt/vllm-env/bin/vllm bench serve \ --model Qwen/Qwen3-30B-A3B-Instruct-2507 \ --num-prompts 25 \ --max-concurrency 1 \ --random-input-len 256 \ --random-output-len 256 - 샘플 출력 캡처.
vLLM을 사용하여 Qwen3:30B를 제공하기 위한 샘플 출력입니다.
vLLM의 샘플 출력:dell@dell:~/Downloads/qwen3-30b-a3b-instruct-2507-qpc$ docker run --rm -it \ --name qaic-bench \ --device=/dev/accel/accel1 \ --device=/dev/accel/accel2 \ --network host \ --ulimit nofile=1048576 \ -e OMP_NUM_THREADS=8 \ -v ~/Downloads/qwen3-30b-a3b-instruct-2507-qpc:/root/qpc \ ghcr.io/quic/cloud_ai_inference_ubuntu24:1.20.6.0 \ /opt/vllm-env/bin/vllm serve Qwen/Qwen3-30B-A3B-Instruct-2507 \ --max-num-seq 1 \ --max-model-len 4096 \ --max-seq_len-to-capture 1 \ --device qaic \ --device-group 0,1 \ --quantization mxfp6 \ --kv-cache-dtype mxint8 \ --override-qaic-config "qpc_path=/root/qpc/qpc" loading /opt/qti-aic/dev/lib/x86_64/libQAic.so INFO 01-15 17:46:28 [__init__.py:269] Automatically detected platform qaic. INFO 01-15 17:46:30 [api_server.py:1043] vLLM API server version 0.8.6.dev0+gba41cc90e.d20251229 INFO 01-15 17:46:30 [api_server.py:1044] args: Namespace(subparser='serve', model_tag='Qwen/Qwen3-30B-A3B-Instruct-2507', config='', host=None, port=8000, uvicorn_log_level='info', disable_uvicorn_access_log=False, allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='Qwen/Qwen3-30B-A3B-Instruct-2507', task='auto', tokenizer=None, hf_config_path=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, load_format='auto', download_dir=None, model_loader_extra_config={}, use_tqdm_on_load=True, config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', max_model_len=4096, guided_decoding_backend='auto', reasoning_parser=None, logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, data_parallel_size=1, enable_expert_parallel=False, max_parallel_loading_workers=None, ray_workers_use_nsight=False, disable_custom_all_reduce=False, block_size=None, gpu_memory_utilization=0.9, swap_space=4, kv_cache_dtype='mxint8', num_gpu_blocks_override=None, enable_prefix_caching=None, prefix_caching_hash_algo='builtin', cpu_offload_gb=0, calculate_kv_scales=False, disable_sliding_window=False, use_v2_block_manager=True, seed=None, max_logprobs=20, disable_log_stats=False, quantization='mxfp6', rope_scaling=None, rope_theta=None, hf_token=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=1, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config={}, limit_mm_per_prompt={}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=None, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=None, max_prompt_adapters=1, max_prompt_adapter_token=0, device='qaic', speculative_config=None, ignore_patterns=[], served_model_name=None, qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, max_num_batched_tokens=None, max_num_seqs=1, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, num_lookahead_slots=0, scheduler_delay_factor=0.0, preemption_mode=None, num_scheduler_steps=1, multi_step_stream_outputs=True, scheduling_policy='fcfs', enable_chunked_prefill=None, disable_chunked_mm_input=False, scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', worker_extension_cls='', generation_config='auto', override_generation_config=None, enable_sleep_mode=False, additional_config=None, enable_reasoning=False, disable_cascade_attn=False, override_qaic_config={'qpc_path': '/root/qpc/qpc'}, device_group=[0, 1], disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, enable_server_load_tracking=False, dispatch_function=<function ServeSubcommand.cmd at 0x7416f976a710>) config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 963/963 [00:00<00:00, 6.38MB/s] INFO 01-15 17:46:34 [config.py:728] This model supports multiple tasks: {'score', 'generate', 'classify', 'embed', 'reward'}. Defaulting to 'generate'. WARNING 01-15 17:46:34 [arg_utils.py:1711] --kv-cache-dtype is not supported by the V1 Engine. Falling back to V0. INFO 01-15 17:46:34 [config.py:1415] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor INFO 01-15 17:46:34 [config.py:1820] Disabled the custom all-reduce kernel because it is not supported on current platform. config_1m.json: 77.3kB [00:00, 163MB/s] generation_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████| 239/239 [00:00<00:00, 2.47MB/s] model.safetensors.index.json: 1.70MB [00:00, 57.5MB/s] tokenizer.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 11.4M/11.4M [00:02<00:00, 5.23MB/s] tokenizer_config.json: 9.38kB [00:00, 41.9MB/s] vocab.json: 2.78MB [00:00, 25.4MB/s] INFO 01-15 17:46:38 [api_server.py:246] Started engine process with PID 51 merges.txt: 1.67MB [00:00, 11.8MB/s] loading /opt/qti-aic/dev/lib/x86_64/libQAic.so INFO 01-15 17:46:40 [__init__.py:269] Automatically detected platform qaic. INFO 01-15 17:46:41 [llm_engine.py:247] Initializing a V0 LLM engine (v0.8.6.dev0+gba41cc90e.d20251229) with config: model='Qwen/Qwen3-30B-A3B-Instruct-2507', speculative_config=None, tokenizer='Qwen/Qwen3-30B-A3B-Instruct-2507', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=mxfp6, enforce_eager=False, kv_cache_dtype=mxint8, device_config=cpu, decoding_config=DecodingConfig(guided_decoding_backend='auto', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=Qwen/Qwen3-30B-A3B-Instruct-2507, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=None, chunked_prefill_enabled=False, use_async_output_proc=False, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[1],"max_capture_size":1}, use_cached_outputs=True, WARNING 01-15 17:46:42 [qaic.py:100] Pin memory is not supported on Qaic. WARNING 01-15 17:46:42 [registry.py:217] Model class <class 'vllm.model_executor.models.mllama.MllamaForConditionalGeneration'> already has a multi-modal processor registered to <vllm.multimodal.registry.MultiModalRegistry object at 0x7f2ffb2ddc90>. It is overwritten by the new one. WARNING 01-15 17:46:42 [registry.py:217] Model class <class 'vllm.model_executor.models.mllama4.Llama4ForConditionalGeneration'> already has a multi-modal processor registered to <vllm.multimodal.registry.MultiModalRegistry object at 0x7f2ffb2ddc90>. It is overwritten by the new one. WARNING 01-15 17:46:42 [qaic_worker.py:102] Reducing Torch parallelism from 8 threads to 8 to avoid unnecessary CPU contention. Set VLLM_QAIC_MAX_CPU_THREADS to tune this value as needed. INFO 01-15 17:46:42 [qaic_worker.py:87] Setting RLIMIT_STACK to 16.0MB. INFO 01-15 17:46:44 [parallel_state.py:1004] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0 loading /opt/qti-aic/dev/lib/x86_64/libQAic.so INFO 01-15 17:46:44 [executor_base.py:112] # qaic blocks: 1, # CPU blocks: 0 INFO 01-15 17:46:44 [executor_base.py:117] Maximum concurrency for 4096 tokens per request: 1.00x {'prefill_seq_len': 1, 'ctx_len': 4096, 'batch_size': 1, 'full_batch_size': 1, 'device_group': [0, 1], 'num_devices': 2, 'num_cores': 16, 'mxfp6_matmul': True, 'mxint8_kv_cache': True, 'aic_enable_depth_first': True, 'prefill_only': None, 'compile_only': False} INFO 01-15 17:46:44 [qaic.py:959] Using qpc:-/root/qpc/qpc Stages: 1 INFO 01-15 17:46:44 [qserve_model_runner.py:66] Loading QPC... INFO 01-15 17:46:44 [qserve_model_runner.py:67] This may take some time, please don't press CTRL-C during this phase... INFO 01-15 17:46:52 [qserve_model_runner.py:88] Successfully loaded QPC in 7.791425864999837 secs WARNING 01-15 17:46:52 [qserve_model_runner.py:122] User entered `include_sampler`=True. But the provided QPC is not compiled to run sampling on the QAIC device. Falling back to the PyTorch backend. INFO 01-15 17:46:52 [qserve_model_runner.py:162] Warming-up with Dummy run... INFO 01-15 17:46:52 [qserve_model_runner.py:178] Successfully finished Dummy run in 0.04556767500025671 secs INFO 01-15 17:46:52 [llm_engine.py:444] init engine (profile, create kv cache, warmup model) took 7.86 seconds WARNING 01-15 17:46:53 [config.py:1251] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. INFO 01-15 17:46:53 [serving_chat.py:124] Using default chat sampling params from model: {'temperature': 0.7, 'top_k': 20, 'top_p': 0.8} INFO 01-15 17:46:53 [serving_completion.py:61] Using default completion sampling params from model: {'temperature': 0.7, 'top_k': 20, 'top_p': 0.8} INFO 01-15 17:46:53 [api_server.py:1090] Starting vLLM API server on http://0.0.0.0:8000 INFO 01-15 17:46:53 [launcher.py:28] Available routes are: INFO 01-15 17:46:53 [launcher.py:36] Route: /openapi.json, Methods: GET, HEAD INFO 01-15 17:46:53 [launcher.py:36] Route: /docs, Methods: GET, HEAD INFO 01-15 17:46:53 [launcher.py:36] Route: /docs/oauth2-redirect, Methods: GET, HEAD INFO 01-15 17:46:53 [launcher.py:36] Route: /redoc, Methods: GET, HEAD INFO 01-15 17:46:53 [launcher.py:36] Route: /health, Methods: GET INFO 01-15 17:46:53 [launcher.py:36] Route: /load, Methods: GET INFO 01-15 17:46:53 [launcher.py:36] Route: /ping, Methods: POST, GET INFO 01-15 17:46:53 [launcher.py:36] Route: /tokenize, Methods: POST INFO 01-15 17:46:53 [launcher.py:36] Route: /detokenize, Methods: POST INFO 01-15 17:46:53 [launcher.py:36] Route: /v1/models, Methods: GET INFO 01-15 17:46:53 [launcher.py:36] Route: /version, Methods: GET INFO 01-15 17:46:53 [launcher.py:36] Route: /v1/chat/completions, Methods: POST INFO 01-15 17:46:53 [launcher.py:36] Route: /v1/completions, Methods: POST INFO 01-15 17:46:53 [launcher.py:36] Route: /v1/embeddings, Methods: POST INFO 01-15 17:46:53 [launcher.py:36] Route: /pooling, Methods: POST INFO 01-15 17:46:53 [launcher.py:36] Route: /score, Methods: POST INFO 01-15 17:46:53 [launcher.py:36] Route: /v1/score, Methods: POST INFO 01-15 17:46:53 [launcher.py:36] Route: /v1/audio/transcriptions, Methods: POST INFO 01-15 17:46:53 [launcher.py:36] Route: /rerank, Methods: POST INFO 01-15 17:46:53 [launcher.py:36] Route: /v1/rerank, Methods: POST INFO 01-15 17:46:53 [launcher.py:36] Route: /v2/rerank, Methods: POST INFO 01-15 17:46:53 [launcher.py:36] Route: /invocations, Methods: POST INFO 01-15 17:46:53 [launcher.py:36] Route: /metrics, Methods: GET INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete.
qaic-bench에 대한 샘플 벤치마킹 출력.
qaic-bench의 샘플 출력dell@dell:~$ docker exec -it qaic-bench \ /opt/vllm-env/bin/vllm bench serve \ --model Qwen/Qwen3-30B-A3B-Instruct-2507 \ --num-prompts 25 \ --max-concurrency 1 \ --random-input-len 256 \ --random-output-len 256 loading /opt/qti-aic/dev/lib/x86_64/libQAic.so INFO 01-15 17:47:44 [__init__.py:269] Automatically detected platform qaic. Namespace(subparser='bench', bench_type='serve', dispatch_function=<function BenchmarkServingSubcommand.cmd at 0x71abbd45e9e0>, endpoint_type='openai-comp', label=None, base_url=None, host='127.0.0.1', port=8000, endpoint='/v1/completions', dataset_name='random', max_concurrency=1, model='Qwen/Qwen3-30B-A3B-Instruct-2507', tokenizer=None, best_of=1, use_beam_search=False, num_prompts=25, logprobs=None, request_rate=inf, burstiness=1.0, seed=0, trust_remote_code=False, disable_tqdm=False, profile=False, save_result=False, metadata=None, result_dir=None, result_filename=None, ignore_eos=False, percentile_metrics='ttft,tpot,itl', metric_percentiles='99', goodput=None, random_input_len=256, random_output_len=256, random_range_ratio=1.0, random_prefix_len=0, tokenizer_mode='auto', served_model_name=None, lora_modules=None) Starting initial single prompt test run... Initial test run completed. Starting main benchmark run... Traffic request rate: inf Burstiness factor: 1.0 (Poisson process) Maximum request concurrency: 1 100%|████████████████████████████████████████████████████████████████████████████████████| 25/25 [05:00<00:00, 12.02s/it] ============ Serving Benchmark Result ============ Successful requests: 25 Benchmark duration (s): 300.53 Total input tokens: 6400 Total generated tokens: 6400 Request throughput (req/s): 0.08 Output token throughput (tok/s): 21.30 Total Token throughput (tok/s): 42.59 ---------------Time to First Token---------------- Mean TTFT (ms): 5915.88 Median TTFT (ms): 5803.64 P99 TTFT (ms): 7153.97 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 23.94 Median TPOT (ms): 23.82 P99 TPOT (ms): 25.18 ---------------Inter-token Latency---------------- Mean ITL (ms): 23.94 Median ITL (ms): 23.78 P99 ITL (ms): 25.77 ================================================== dell@dell:~$
출력 생성 토큰에 대한 TPS(토큰/초) 결과를 계산하려면 1000 ÷ 평균 TPOT(위에서 23.94) = 41.8TPS를 사용합니다.
- Qwen3-30B-A3B-Instruct-2507 사전 구축된 QPC를 다운로드합니다.
Affected Products
Dell Pro Max 16 Plus MB16250Article Properties
Article Number: 000446730
Article Type: How To
Last Modified: 24 May 2026
Version: 5
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.