Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1             Please see https://github.com/pytorch/ao/issues/2919 for more info
WARNING 12-19 17:11:45 [attention.py:82] Using VLLM_V1_USE_PREFILL_DECODE_ATTENTION environment variable is deprecated and will be removed in v0.14.0 or v1.0.0, whichever is soonest. Please use --attention-config.use_prefill_decode_attention command line argument or AttentionConfig(use_prefill_decode_attention=...) config field instead.
(APIServer pid=76251) INFO 12-19 17:11:45 [api_server.py:1351] vLLM API server version 0.13.0rc2.dev112+g763963aa7.d20251213
(APIServer pid=76251) INFO 12-19 17:11:45 [utils.py:253] non-default args: {'model_tag': 'cpatonn/Qwen3-Next-80B-A3B-Instruct-AWQ-4bit', 'host': '127.0.0.1', 'model': 'cpatonn/Qwen3-Next-80B-A3B-Instruct-AWQ-4bit', 'trust_remote_code': True, 'max_model_len': 16384, 'tensor_parallel_size': 2, 'gpu_memory_utilization': 0.98, 'max_num_seqs': 32}
(APIServer pid=76251) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=76251) INFO 12-19 17:11:49 [model.py:514] Resolved architecture: Qwen3NextForCausalLM
(APIServer pid=76251) INFO 12-19 17:11:49 [model.py:1636] Using max model len 16384
(APIServer pid=76251) INFO 12-19 17:11:49 [scheduler.py:228] Chunked prefill is enabled with max_num_batched_tokens=2048.
(APIServer pid=76251) INFO 12-19 17:11:49 [config.py:312] Disabling cascade attention since it is not supported for hybrid models.
(APIServer pid=76251) INFO 12-19 17:11:49 [config.py:439] Setting attention block size to 544 tokens to ensure that attention page size is >= mamba page size.
(APIServer pid=76251) INFO 12-19 17:11:49 [config.py:463] Padding mamba page size by 1.49% to ensure that mamba page size and attention page size are exactly equal.
Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1             Please see https://github.com/pytorch/ao/issues/2919 for more info
(EngineCore_DP0 pid=76413) INFO 12-19 17:11:53 [core.py:93] Initializing a V1 LLM engine (v0.13.0rc2.dev112+g763963aa7.d20251213) with config: model='cpatonn/Qwen3-Next-80B-A3B-Instruct-AWQ-4bit', speculative_config=None, tokenizer='cpatonn/Qwen3-Next-80B-A3B-Instruct-AWQ-4bit', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=16384, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=True, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False), seed=0, served_model_name=cpatonn/Qwen3-Next-80B-A3B-Instruct-AWQ-4bit, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 64, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False}, 'local_cache_dir': None}
(EngineCore_DP0 pid=76413) WARNING 12-19 17:11:53 [multiproc_executor.py:884] Reducing Torch parallelism from 24 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1             Please see https://github.com/pytorch/ao/issues/2919 for more info
Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1             Please see https://github.com/pytorch/ao/issues/2919 for more info
INFO 12-19 17:11:56 [parallel_state.py:1203] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:50157 backend=nccl
INFO 12-19 17:11:56 [parallel_state.py:1203] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:50157 backend=nccl
INFO 12-19 17:11:57 [pynccl.py:111] vLLM is using nccl==2.27.3
INFO 12-19 17:11:57 [parallel_state.py:1411] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 1, EP rank 1
INFO 12-19 17:11:57 [parallel_state.py:1411] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
(Worker_TP0 pid=76495) INFO 12-19 17:11:58 [gpu_model_runner.py:3562] Starting to load model cpatonn/Qwen3-Next-80B-A3B-Instruct-AWQ-4bit...
(Worker_TP1 pid=76496) WARNING 12-19 17:11:58 [compressed_tensors.py:742] Acceleration for non-quantized schemes is not supported by Compressed Tensors. Falling back to UnquantizedLinearMethod
(Worker_TP1 pid=76496) INFO 12-19 17:11:58 [layer.py:372] Enabled separate cuda stream for MoE shared_experts
(Worker_TP1 pid=76496) INFO 12-19 17:11:58 [compressed_tensors_moe.py:188] Using CompressedTensorsWNA16MoEMethod
(Worker_TP0 pid=76495) WARNING 12-19 17:11:58 [compressed_tensors.py:742] Acceleration for non-quantized schemes is not supported by Compressed Tensors. Falling back to UnquantizedLinearMethod
(Worker_TP0 pid=76495) INFO 12-19 17:11:58 [layer.py:372] Enabled separate cuda stream for MoE shared_experts
(Worker_TP0 pid=76495) INFO 12-19 17:11:58 [compressed_tensors_moe.py:188] Using CompressedTensorsWNA16MoEMethod
(Worker_TP1 pid=76496) INFO 12-19 17:11:58 [rocm.py:306] Using Rocm Attention backend on V1 engine.
(Worker_TP0 pid=76495) INFO 12-19 17:11:58 [rocm.py:306] Using Rocm Attention backend on V1 engine.
(Worker_TP0 pid=76495) 
Loading safetensors checkpoint shards:   0% Completed | 0/10 [00:00<?, ?it/s]
(Worker_TP0 pid=76495) 
Loading safetensors checkpoint shards:  10% Completed | 1/10 [00:03<00:32,  3.65s/it]
(Worker_TP0 pid=76495) 
Loading safetensors checkpoint shards:  20% Completed | 2/10 [00:07<00:29,  3.64s/it]
(Worker_TP0 pid=76495) 
Loading safetensors checkpoint shards:  30% Completed | 3/10 [00:11<00:26,  3.79s/it]
(Worker_TP0 pid=76495) 
Loading safetensors checkpoint shards:  40% Completed | 4/10 [00:15<00:23,  3.97s/it]
(Worker_TP0 pid=76495) 
Loading safetensors checkpoint shards:  50% Completed | 5/10 [00:18<00:18,  3.76s/it]
(Worker_TP0 pid=76495) 
Loading safetensors checkpoint shards:  60% Completed | 6/10 [00:22<00:15,  3.85s/it]
(Worker_TP0 pid=76495) 
Loading safetensors checkpoint shards:  70% Completed | 7/10 [00:26<00:11,  3.83s/it]
(Worker_TP0 pid=76495) 
Loading safetensors checkpoint shards:  80% Completed | 8/10 [00:30<00:07,  3.84s/it]
(Worker_TP0 pid=76495) 
Loading safetensors checkpoint shards: 100% Completed | 10/10 [00:34<00:00,  2.97s/it]
(Worker_TP0 pid=76495) 
Loading safetensors checkpoint shards: 100% Completed | 10/10 [00:34<00:00,  3.46s/it]
(Worker_TP0 pid=76495) 
(Worker_TP0 pid=76495) INFO 12-19 17:12:33 [default_loader.py:308] Loading weights took 34.64 seconds
(Worker_TP0 pid=76495) INFO 12-19 17:12:34 [gpu_model_runner.py:3659] Model loading took 23.5020 GiB memory and 35.701153 seconds
(Worker_TP0 pid=76495) INFO 12-19 17:12:38 [backends.py:634] Using cache directory: /home/kyuz0/.cache/vllm/torch_compile_cache/671837fea6/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=76495) INFO 12-19 17:12:38 [backends.py:694] Dynamo bytecode transform time: 4.08 s
(Worker_TP1 pid=76496) INFO 12-19 17:12:40 [backends.py:261] Cache the graph of compile range (1, 2048) for later use
(Worker_TP0 pid=76495) INFO 12-19 17:12:40 [backends.py:261] Cache the graph of compile range (1, 2048) for later use
(Worker_TP1 pid=76496) WARNING 12-19 17:12:41 [fused_moe.py:888] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/opt/venv/lib/python3.13/site-packages/vllm/model_executor/layers/fused_moe/configs/E=512,N=256,device_name=AMD-gfx1201,dtype=int4_w4a16.json']
(Worker_TP0 pid=76495) WARNING 12-19 17:12:41 [fused_moe.py:888] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/opt/venv/lib/python3.13/site-packages/vllm/model_executor/layers/fused_moe/configs/E=512,N=256,device_name=AMD-gfx1201,dtype=int4_w4a16.json']
(Worker_TP0 pid=76495) INFO 12-19 17:12:43 [backends.py:278] Compiling a graph for compile range (1, 2048) takes 2.59 s
(Worker_TP0 pid=76495) INFO 12-19 17:12:43 [monitor.py:34] torch.compile takes 6.67 s in total
(Worker_TP1 pid=76496) WARNING 12-19 17:12:43 [decorators.py:528] Cannot save aot compilation to path /home/kyuz0/.cache/vllm/torch_aot_compile/8cfc5b4a85b0195581f74785744c13daeda3dd4bc8885cc84b5001d538a63819/rank_1_0/model, error: 
(Worker_TP0 pid=76495) WARNING 12-19 17:12:43 [decorators.py:528] Cannot save aot compilation to path /home/kyuz0/.cache/vllm/torch_aot_compile/8cfc5b4a85b0195581f74785744c13daeda3dd4bc8885cc84b5001d538a63819/rank_0_0/model, error: 
(Worker_TP0 pid=76495) INFO 12-19 17:12:44 [gpu_worker.py:375] Available KV cache memory: 7.09 GiB
(EngineCore_DP0 pid=76413) INFO 12-19 17:12:45 [kv_cache_utils.py:1291] GPU KV cache size: 154,496 tokens
(EngineCore_DP0 pid=76413) INFO 12-19 17:12:45 [kv_cache_utils.py:1296] Maximum concurrency for 16,384 tokens per request: 33.47x
(Worker_TP0 pid=76495) 
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/11 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   9%|▉         | 1/11 [00:00<00:05,  1.95it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  18%|█▊        | 2/11 [00:01<00:04,  1.83it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  27%|██▋       | 3/11 [00:01<00:04,  1.91it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  36%|███▋      | 4/11 [00:02<00:03,  1.93it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  45%|████▌     | 5/11 [00:02<00:03,  1.95it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  55%|█████▍    | 6/11 [00:03<00:02,  1.95it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  64%|██████▎   | 7/11 [00:03<00:02,  1.97it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  73%|███████▎  | 8/11 [00:04<00:01,  1.98it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  82%|████████▏ | 9/11 [00:04<00:01,  1.95it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  91%|█████████ | 10/11 [00:05<00:00,  1.92it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 11/11 [00:05<00:00,  1.98it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 11/11 [00:05<00:00,  1.95it/s]
(Worker_TP0 pid=76495) 
Capturing CUDA graphs (decode, FULL):   0%|          | 0/7 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):   0%|          | 0/7 [00:00<?, ?it/s]
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] WorkerProc hit an exception.
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] Traceback (most recent call last):
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/language/core.py", line 43, in wrapper
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return fn(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/language/core.py", line 1638, in arange
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return _semantic.arange(start, end)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~~^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/language/semantic.py", line 583, in arange
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     raise ValueError("arange's range must be a power of 2")
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ValueError: arange's range must be a power of 2
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] 
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] The above exception was the direct cause of the following exception:
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] 
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] Traceback (most recent call last):
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py", line 821, in worker_busy_loop
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     output = func(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_worker.py", line 459, in compile_or_warm_up_model
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     cuda_graph_memory_bytes = self.model_runner.capture_model()
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4586, in capture_model
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     self._capture_cudagraphs(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ~~~~~~~~~~~~~~~~~~~~~~~~^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         compilation_cases=compilation_cases_decode,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         cudagraph_runtime_mode=CUDAGraphMode.FULL,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         uniform_decode=True,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4664, in _capture_cudagraphs
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     self._dummy_run(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ~~~~~~~~~~~~~~~^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         num_tokens,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ...<6 lines>...
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         activate_lora=activate_lora,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return func(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4198, in _dummy_run
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     outputs = self.model(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         input_ids=input_ids,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ...<3 lines>...
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         **model_kwargs,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/compilation/cuda_graph.py", line 220, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self.runnable(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return forward_call(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/model_executor/models/qwen3_next.py", line 1231, in forward
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     hidden_states = self.model(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         input_ids, positions, intermediate_tensors, inputs_embeds
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/compilation/decorators.py", line 376, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self.aot_compiled_fn(self, *args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/_dynamo/aot_compile.py", line 124, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self.fn(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/model_executor/models/qwen3_next.py", line 997, in forward
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     def forward(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/compilation/caching.py", line 54, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self.optimized_call(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 936, in call_wrapped
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self._wrapped_call(self, *args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 455, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     raise e
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 442, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return forward_call(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "<eval_with_key>.99", line 333, in forward
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     submod_7 = self.submod_7(getitem_21, s72, getitem_22, getitem_23, getitem_24);  getitem_21 = getitem_22 = getitem_23 = submod_7 = None
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 936, in call_wrapped
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self._wrapped_call(self, *args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 455, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     raise e
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 442, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return forward_call(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "<eval_with_key>.107", line 5, in forward
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     unified_attention_with_output = torch.ops.vllm.unified_attention_with_output(query_8, key_8, value_9, output_5, 'model.layers.3.self_attn.attn');  query_8 = key_8 = value_9 = output_5 = unified_attention_with_output = None
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/_ops.py", line 1209, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self._op(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/attention/utils/kv_transfer_utils.py", line 39, in wrapper
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return func(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/attention/layer.py", line 923, in unified_attention_with_output
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     self.impl.forward(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ~~~~~~~~~~~~~~~~~^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         self,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ...<7 lines>...
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         output_block_scale=output_block_scale,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/attention/backends/rocm_attn.py", line 337, in forward
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     chunked_prefill_paged_decode(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         query=query[:num_actual_tokens],
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ...<17 lines>...
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         sinks=self.sinks,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/attention/ops/chunked_prefill_paged_decode.py", line 356, in chunked_prefill_paged_decode
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     kernel_paged_attention_2d[
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ~~~~~~~~~~~~~~~~~~~~~~~~~~
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ...<3 lines>...
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ~
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ](
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ~^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         output_ptr=output,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ...<37 lines>...
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         USE_FP8=output_scale is not None,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/runtime/jit.py", line 419, in <lambda>
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]                                    ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/runtime/jit.py", line 733, in run
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     kernel = self._do_compile(key, signature, device, constexprs, options, attrs, warmup)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/runtime/jit.py", line 861, in _do_compile
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     kernel = self.compile(src, target=target, options=options.__dict__)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/compiler/compiler.py", line 300, in compile
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     module = src.make_ir(target, options, codegen_fns, module_map, context)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/compiler/compiler.py", line 80, in make_ir
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return ast_to_ttir(self.fn, self, context=context, options=options, codegen_fns=codegen_fns,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]                        module_map=module_map)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] triton.compiler.errors.CompilationError: at 106:17:
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     if USE_ALIBI_SLOPES:
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         alibi_slope = tl.load(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]             alibi_slopes_ptr + query_head_idx, mask=head_mask, other=0.0
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] 
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     num_blocks = cdiv_fn(seq_len, BLOCK_SIZE)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] 
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     # iterate through tiles
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     for j in range(0, num_blocks):
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         physical_block_idx = tl.load(block_tables_ptr + block_table_offset + j)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] 
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         offs_n = tl.arange(0, BLOCK_SIZE)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]                  ^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] arange's range must be a power of 2
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] Traceback (most recent call last):
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/language/core.py", line 43, in wrapper
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return fn(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/language/core.py", line 1638, in arange
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return _semantic.arange(start, end)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~~^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/language/semantic.py", line 583, in arange
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     raise ValueError("arange's range must be a power of 2")
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ValueError: arange's range must be a power of 2
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] 
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] The above exception was the direct cause of the following exception:
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] 
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] Traceback (most recent call last):
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py", line 821, in worker_busy_loop
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     output = func(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_worker.py", line 459, in compile_or_warm_up_model
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     cuda_graph_memory_bytes = self.model_runner.capture_model()
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4586, in capture_model
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     self._capture_cudagraphs(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ~~~~~~~~~~~~~~~~~~~~~~~~^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         compilation_cases=compilation_cases_decode,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         cudagraph_runtime_mode=CUDAGraphMode.FULL,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         uniform_decode=True,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4664, in _capture_cudagraphs
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     self._dummy_run(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ~~~~~~~~~~~~~~~^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         num_tokens,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ...<6 lines>...
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         activate_lora=activate_lora,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return func(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4198, in _dummy_run
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     outputs = self.model(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         input_ids=input_ids,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ...<3 lines>...
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         **model_kwargs,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/compilation/cuda_graph.py", line 220, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self.runnable(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return forward_call(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/model_executor/models/qwen3_next.py", line 1231, in forward
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     hidden_states = self.model(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         input_ids, positions, intermediate_tensors, inputs_embeds
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/compilation/decorators.py", line 376, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self.aot_compiled_fn(self, *args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/_dynamo/aot_compile.py", line 124, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self.fn(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/model_executor/models/qwen3_next.py", line 997, in forward
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     def forward(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/compilation/caching.py", line 54, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self.optimized_call(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 936, in call_wrapped
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self._wrapped_call(self, *args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 455, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     raise e
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 442, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return forward_call(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "<eval_with_key>.99", line 333, in forward
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     submod_7 = self.submod_7(getitem_21, s72, getitem_22, getitem_23, getitem_24);  getitem_21 = getitem_22 = getitem_23 = submod_7 = None
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 936, in call_wrapped
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self._wrapped_call(self, *args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 455, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     raise e
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 442, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return forward_call(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "<eval_with_key>.107", line 5, in forward
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     unified_attention_with_output = torch.ops.vllm.unified_attention_with_output(query_8, key_8, value_9, output_5, 'model.layers.3.self_attn.attn');  query_8 = key_8 = value_9 = output_5 = unified_attention_with_output = None
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/_ops.py", line 1209, in __call__
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self._op(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/attention/utils/kv_transfer_utils.py", line 39, in wrapper
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return func(*args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/attention/layer.py", line 923, in unified_attention_with_output
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     self.impl.forward(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ~~~~~~~~~~~~~~~~~^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         self,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ...<7 lines>...
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         output_block_scale=output_block_scale,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/attention/backends/rocm_attn.py", line 337, in forward
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     chunked_prefill_paged_decode(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         query=query[:num_actual_tokens],
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ...<17 lines>...
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         sinks=self.sinks,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/attention/ops/chunked_prefill_paged_decode.py", line 356, in chunked_prefill_paged_decode
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     kernel_paged_attention_2d[
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ~~~~~~~~~~~~~~~~~~~~~~~~~~
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ...<3 lines>...
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ~
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ](
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ~^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         output_ptr=output,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ...<37 lines>...
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         USE_FP8=output_scale is not None,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/runtime/jit.py", line 419, in <lambda>
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]                                    ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/runtime/jit.py", line 733, in run
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     kernel = self._do_compile(key, signature, device, constexprs, options, attrs, warmup)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/runtime/jit.py", line 861, in _do_compile
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     kernel = self.compile(src, target=target, options=options.__dict__)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/compiler/compiler.py", line 300, in compile
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     module = src.make_ir(target, options, codegen_fns, module_map, context)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/compiler/compiler.py", line 80, in make_ir
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return ast_to_ttir(self.fn, self, context=context, options=options, codegen_fns=codegen_fns,
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]                        module_map=module_map)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] triton.compiler.errors.CompilationError: at 106:17:
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     if USE_ALIBI_SLOPES:
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         alibi_slope = tl.load(
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]             alibi_slopes_ptr + query_head_idx, mask=head_mask, other=0.0
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         )
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] 
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     num_blocks = cdiv_fn(seq_len, BLOCK_SIZE)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] 
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     # iterate through tiles
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     for j in range(0, num_blocks):
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         physical_block_idx = tl.load(block_tables_ptr + block_table_offset + j)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] 
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         offs_n = tl.arange(0, BLOCK_SIZE)
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826]                  ^
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] arange's range must be a power of 2
(Worker_TP1 pid=76496) ERROR 12-19 17:12:51 [multiproc_executor.py:826] 
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] WorkerProc hit an exception.
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] Traceback (most recent call last):
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/language/core.py", line 43, in wrapper
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return fn(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/language/core.py", line 1638, in arange
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return _semantic.arange(start, end)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~~^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/language/semantic.py", line 583, in arange
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     raise ValueError("arange's range must be a power of 2")
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ValueError: arange's range must be a power of 2
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] 
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] The above exception was the direct cause of the following exception:
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] 
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] Traceback (most recent call last):
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py", line 821, in worker_busy_loop
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     output = func(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_worker.py", line 459, in compile_or_warm_up_model
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     cuda_graph_memory_bytes = self.model_runner.capture_model()
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4586, in capture_model
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     self._capture_cudagraphs(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ~~~~~~~~~~~~~~~~~~~~~~~~^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         compilation_cases=compilation_cases_decode,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         cudagraph_runtime_mode=CUDAGraphMode.FULL,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         uniform_decode=True,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4664, in _capture_cudagraphs
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     self._dummy_run(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ~~~~~~~~~~~~~~~^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         num_tokens,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ...<6 lines>...
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         activate_lora=activate_lora,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return func(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4198, in _dummy_run
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     outputs = self.model(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         input_ids=input_ids,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ...<3 lines>...
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         **model_kwargs,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/compilation/cuda_graph.py", line 220, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self.runnable(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return forward_call(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/model_executor/models/qwen3_next.py", line 1231, in forward
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     hidden_states = self.model(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         input_ids, positions, intermediate_tensors, inputs_embeds
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/compilation/decorators.py", line 376, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self.aot_compiled_fn(self, *args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/_dynamo/aot_compile.py", line 124, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self.fn(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/model_executor/models/qwen3_next.py", line 997, in forward
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     def forward(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/compilation/caching.py", line 54, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self.optimized_call(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 936, in call_wrapped
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self._wrapped_call(self, *args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 455, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     raise e
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 442, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return forward_call(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "<eval_with_key>.99", line 333, in forward
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     submod_7 = self.submod_7(getitem_21, s72, getitem_22, getitem_23, getitem_24);  getitem_21 = getitem_22 = getitem_23 = submod_7 = None
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 936, in call_wrapped
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self._wrapped_call(self, *args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 455, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     raise e
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 442, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return forward_call(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "<eval_with_key>.107", line 5, in forward
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     unified_attention_with_output = torch.ops.vllm.unified_attention_with_output(query_8, key_8, value_9, output_5, 'model.layers.3.self_attn.attn');  query_8 = key_8 = value_9 = output_5 = unified_attention_with_output = None
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/_ops.py", line 1209, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self._op(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/attention/utils/kv_transfer_utils.py", line 39, in wrapper
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return func(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/attention/layer.py", line 923, in unified_attention_with_output
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     self.impl.forward(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ~~~~~~~~~~~~~~~~~^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         self,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ...<7 lines>...
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         output_block_scale=output_block_scale,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/attention/backends/rocm_attn.py", line 337, in forward
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     chunked_prefill_paged_decode(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         query=query[:num_actual_tokens],
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ...<17 lines>...
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         sinks=self.sinks,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/attention/ops/chunked_prefill_paged_decode.py", line 356, in chunked_prefill_paged_decode
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     kernel_paged_attention_2d[
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ~~~~~~~~~~~~~~~~~~~~~~~~~~
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ...<3 lines>...
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ~
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ](
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ~^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         output_ptr=output,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ...<37 lines>...
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         USE_FP8=output_scale is not None,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/runtime/jit.py", line 419, in <lambda>
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]                                    ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/runtime/jit.py", line 733, in run
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     kernel = self._do_compile(key, signature, device, constexprs, options, attrs, warmup)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/runtime/jit.py", line 861, in _do_compile
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     kernel = self.compile(src, target=target, options=options.__dict__)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/compiler/compiler.py", line 300, in compile
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     module = src.make_ir(target, options, codegen_fns, module_map, context)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/compiler/compiler.py", line 80, in make_ir
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return ast_to_ttir(self.fn, self, context=context, options=options, codegen_fns=codegen_fns,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]                        module_map=module_map)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] triton.compiler.errors.CompilationError: at 106:17:
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     if USE_ALIBI_SLOPES:
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         alibi_slope = tl.load(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]             alibi_slopes_ptr + query_head_idx, mask=head_mask, other=0.0
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] 
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     num_blocks = cdiv_fn(seq_len, BLOCK_SIZE)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] 
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     # iterate through tiles
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     for j in range(0, num_blocks):
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         physical_block_idx = tl.load(block_tables_ptr + block_table_offset + j)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] 
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         offs_n = tl.arange(0, BLOCK_SIZE)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]                  ^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] arange's range must be a power of 2
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] Traceback (most recent call last):
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/language/core.py", line 43, in wrapper
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return fn(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/language/core.py", line 1638, in arange
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return _semantic.arange(start, end)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~~^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/language/semantic.py", line 583, in arange
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     raise ValueError("arange's range must be a power of 2")
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] ValueError: arange's range must be a power of 2
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] 
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] The above exception was the direct cause of the following exception:
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] 
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] Traceback (most recent call last):
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py", line 821, in worker_busy_loop
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     output = func(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_worker.py", line 459, in compile_or_warm_up_model
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     cuda_graph_memory_bytes = self.model_runner.capture_model()
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4586, in capture_model
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     self._capture_cudagraphs(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ~~~~~~~~~~~~~~~~~~~~~~~~^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         compilation_cases=compilation_cases_decode,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         cudagraph_runtime_mode=CUDAGraphMode.FULL,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         uniform_decode=True,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4664, in _capture_cudagraphs
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     self._dummy_run(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ~~~~~~~~~~~~~~~^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         num_tokens,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ...<6 lines>...
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         activate_lora=activate_lora,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return func(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4198, in _dummy_run
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     outputs = self.model(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         input_ids=input_ids,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ...<3 lines>...
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         **model_kwargs,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/compilation/cuda_graph.py", line 220, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self.runnable(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return forward_call(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/model_executor/models/qwen3_next.py", line 1231, in forward
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     hidden_states = self.model(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         input_ids, positions, intermediate_tensors, inputs_embeds
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/compilation/decorators.py", line 376, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self.aot_compiled_fn(self, *args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/_dynamo/aot_compile.py", line 124, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self.fn(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/model_executor/models/qwen3_next.py", line 997, in forward
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     def forward(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/compilation/caching.py", line 54, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self.optimized_call(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 936, in call_wrapped
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self._wrapped_call(self, *args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 455, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     raise e
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 442, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return forward_call(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "<eval_with_key>.99", line 333, in forward
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     submod_7 = self.submod_7(getitem_21, s72, getitem_22, getitem_23, getitem_24);  getitem_21 = getitem_22 = getitem_23 = submod_7 = None
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 936, in call_wrapped
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self._wrapped_call(self, *args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 455, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     raise e
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/fx/graph_module.py", line 442, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return forward_call(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "<eval_with_key>.107", line 5, in forward
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     unified_attention_with_output = torch.ops.vllm.unified_attention_with_output(query_8, key_8, value_9, output_5, 'model.layers.3.self_attn.attn');  query_8 = key_8 = value_9 = output_5 = unified_attention_with_output = None
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/torch/_ops.py", line 1209, in __call__
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return self._op(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]            ~~~~~~~~^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/attention/utils/kv_transfer_utils.py", line 39, in wrapper
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return func(*args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/attention/layer.py", line 923, in unified_attention_with_output
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     self.impl.forward(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ~~~~~~~~~~~~~~~~~^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         self,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ...<7 lines>...
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         output_block_scale=output_block_scale,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/attention/backends/rocm_attn.py", line 337, in forward
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     chunked_prefill_paged_decode(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         query=query[:num_actual_tokens],
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ...<17 lines>...
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         sinks=self.sinks,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/vllm/attention/ops/chunked_prefill_paged_decode.py", line 356, in chunked_prefill_paged_decode
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     kernel_paged_attention_2d[
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ~~~~~~~~~~~~~~~~~~~~~~~~~~
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ...<3 lines>...
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ~
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ](
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ~^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         output_ptr=output,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ...<37 lines>...
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         USE_FP8=output_scale is not None,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     ^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/runtime/jit.py", line 419, in <lambda>
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]                                    ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/runtime/jit.py", line 733, in run
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     kernel = self._do_compile(key, signature, device, constexprs, options, attrs, warmup)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/runtime/jit.py", line 861, in _do_compile
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     kernel = self.compile(src, target=target, options=options.__dict__)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/compiler/compiler.py", line 300, in compile
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     module = src.make_ir(target, options, codegen_fns, module_map, context)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]   File "/opt/venv/lib64/python3.13/site-packages/triton/compiler/compiler.py", line 80, in make_ir
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     return ast_to_ttir(self.fn, self, context=context, options=options, codegen_fns=codegen_fns,
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]                        module_map=module_map)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] triton.compiler.errors.CompilationError: at 106:17:
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     if USE_ALIBI_SLOPES:
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         alibi_slope = tl.load(
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]             alibi_slopes_ptr + query_head_idx, mask=head_mask, other=0.0
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         )
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] 
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     num_blocks = cdiv_fn(seq_len, BLOCK_SIZE)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] 
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     # iterate through tiles
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]     for j in range(0, num_blocks):
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         physical_block_idx = tl.load(block_tables_ptr + block_table_offset + j)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] 
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]         offs_n = tl.arange(0, BLOCK_SIZE)
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826]                  ^
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] arange's range must be a power of 2
(Worker_TP0 pid=76495) ERROR 12-19 17:12:51 [multiproc_executor.py:826] 
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] EngineCore failed to start.
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] Traceback (most recent call last):
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core.py", line 857, in run_engine_core
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core.py", line 637, in __init__
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]     super().__init__(
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]     ~~~~~~~~~~~~~~~~^
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]         vllm_config, executor_class, log_stats, executor_fail_callback
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]     )
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]     ^
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core.py", line 109, in __init__
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]     num_gpu_blocks, num_cpu_blocks, kv_cache_config = self._initialize_kv_caches(
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]                                                       ~~~~~~~~~~~~~~~~~~~~~~~~~~^
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]         vllm_config
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]         ^^^^^^^^^^^
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]     )
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]     ^
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core.py", line 256, in _initialize_kv_caches
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]     self.model_executor.initialize_from_config(kv_cache_configs)
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/abstract.py", line 116, in initialize_from_config
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]     self.collective_rpc("compile_or_warm_up_model")
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]     ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py", line 361, in collective_rpc
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]     return aggregate(get_response())
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]                      ~~~~~~~~~~~~^^
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py", line 344, in get_response
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]     raise RuntimeError(
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]     ...<2 lines>...
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]     )
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] RuntimeError: Worker failed with error 'at 106:17:
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]     if USE_ALIBI_SLOPES:
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]         alibi_slope = tl.load(
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]             alibi_slopes_ptr + query_head_idx, mask=head_mask, other=0.0
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]         )
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] 
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]     num_blocks = cdiv_fn(seq_len, BLOCK_SIZE)
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] 
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]     # iterate through tiles
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]     for j in range(0, num_blocks):
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]         physical_block_idx = tl.load(block_tables_ptr + block_table_offset + j)
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] 
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]         offs_n = tl.arange(0, BLOCK_SIZE)
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866]                  ^
(EngineCore_DP0 pid=76413) ERROR 12-19 17:12:51 [core.py:866] arange's range must be a power of 2', please check the stack trace above for the root cause
(EngineCore_DP0 pid=76413) Process EngineCore_DP0:
(EngineCore_DP0 pid=76413) Traceback (most recent call last):
(EngineCore_DP0 pid=76413)   File "/usr/lib64/python3.13/multiprocessing/process.py", line 313, in _bootstrap
(EngineCore_DP0 pid=76413)     self.run()
(EngineCore_DP0 pid=76413)     ~~~~~~~~^^
(EngineCore_DP0 pid=76413)   File "/usr/lib64/python3.13/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=76413)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=76413)     ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=76413)   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core.py", line 870, in run_engine_core
(EngineCore_DP0 pid=76413)     raise e
(EngineCore_DP0 pid=76413)   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core.py", line 857, in run_engine_core
(EngineCore_DP0 pid=76413)     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=76413)   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core.py", line 637, in __init__
(EngineCore_DP0 pid=76413)     super().__init__(
(EngineCore_DP0 pid=76413)     ~~~~~~~~~~~~~~~~^
(EngineCore_DP0 pid=76413)         vllm_config, executor_class, log_stats, executor_fail_callback
(EngineCore_DP0 pid=76413)         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=76413)     )
(EngineCore_DP0 pid=76413)     ^
(EngineCore_DP0 pid=76413)   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core.py", line 109, in __init__
(EngineCore_DP0 pid=76413)     num_gpu_blocks, num_cpu_blocks, kv_cache_config = self._initialize_kv_caches(
(EngineCore_DP0 pid=76413)                                                       ~~~~~~~~~~~~~~~~~~~~~~~~~~^
(EngineCore_DP0 pid=76413)         vllm_config
(EngineCore_DP0 pid=76413)         ^^^^^^^^^^^
(EngineCore_DP0 pid=76413)     )
(EngineCore_DP0 pid=76413)     ^
(EngineCore_DP0 pid=76413)   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core.py", line 256, in _initialize_kv_caches
(EngineCore_DP0 pid=76413)     self.model_executor.initialize_from_config(kv_cache_configs)
(EngineCore_DP0 pid=76413)     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=76413)   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/abstract.py", line 116, in initialize_from_config
(EngineCore_DP0 pid=76413)     self.collective_rpc("compile_or_warm_up_model")
(EngineCore_DP0 pid=76413)     ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=76413)   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py", line 361, in collective_rpc
(EngineCore_DP0 pid=76413)     return aggregate(get_response())
(EngineCore_DP0 pid=76413)                      ~~~~~~~~~~~~^^
(EngineCore_DP0 pid=76413)   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py", line 344, in get_response
(EngineCore_DP0 pid=76413)     raise RuntimeError(
(EngineCore_DP0 pid=76413)     ...<2 lines>...
(EngineCore_DP0 pid=76413)     )
(EngineCore_DP0 pid=76413) RuntimeError: Worker failed with error 'at 106:17:
(EngineCore_DP0 pid=76413)     if USE_ALIBI_SLOPES:
(EngineCore_DP0 pid=76413)         alibi_slope = tl.load(
(EngineCore_DP0 pid=76413)             alibi_slopes_ptr + query_head_idx, mask=head_mask, other=0.0
(EngineCore_DP0 pid=76413)         )
(EngineCore_DP0 pid=76413) 
(EngineCore_DP0 pid=76413)     num_blocks = cdiv_fn(seq_len, BLOCK_SIZE)
(EngineCore_DP0 pid=76413) 
(EngineCore_DP0 pid=76413)     # iterate through tiles
(EngineCore_DP0 pid=76413)     for j in range(0, num_blocks):
(EngineCore_DP0 pid=76413)         physical_block_idx = tl.load(block_tables_ptr + block_table_offset + j)
(EngineCore_DP0 pid=76413) 
(EngineCore_DP0 pid=76413)         offs_n = tl.arange(0, BLOCK_SIZE)
(EngineCore_DP0 pid=76413)                  ^
(EngineCore_DP0 pid=76413) arange's range must be a power of 2', please check the stack trace above for the root cause
(Worker_TP0 pid=76495) INFO 12-19 17:12:51 [multiproc_executor.py:711] Parent process exited, terminating worker
(Worker_TP1 pid=76496) INFO 12-19 17:12:51 [multiproc_executor.py:711] Parent process exited, terminating worker
(APIServer pid=76251) Traceback (most recent call last):
(APIServer pid=76251)   File "/opt/venv/bin/vllm", line 7, in <module>
(APIServer pid=76251)     sys.exit(main())
(APIServer pid=76251)              ~~~~^^
(APIServer pid=76251)   File "/opt/venv/lib64/python3.13/site-packages/vllm/entrypoints/cli/main.py", line 73, in main
(APIServer pid=76251)     args.dispatch_function(args)
(APIServer pid=76251)     ~~~~~~~~~~~~~~~~~~~~~~^^^^^^
(APIServer pid=76251)   File "/opt/venv/lib64/python3.13/site-packages/vllm/entrypoints/cli/serve.py", line 60, in cmd
(APIServer pid=76251)     uvloop.run(run_server(args))
(APIServer pid=76251)     ~~~~~~~~~~^^^^^^^^^^^^^^^^^^
(APIServer pid=76251)   File "/opt/venv/lib64/python3.13/site-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=76251)     return __asyncio.run(
(APIServer pid=76251)            ~~~~~~~~~~~~~^
(APIServer pid=76251)         wrapper(),
(APIServer pid=76251)         ^^^^^^^^^^
(APIServer pid=76251)     ...<2 lines>...
(APIServer pid=76251)         **run_kwargs
(APIServer pid=76251)         ^^^^^^^^^^^^
(APIServer pid=76251)     )
(APIServer pid=76251)     ^
(APIServer pid=76251)   File "/usr/lib64/python3.13/asyncio/runners.py", line 195, in run
(APIServer pid=76251)     return runner.run(main)
(APIServer pid=76251)            ~~~~~~~~~~^^^^^^
(APIServer pid=76251)   File "/usr/lib64/python3.13/asyncio/runners.py", line 118, in run
(APIServer pid=76251)     return self._loop.run_until_complete(task)
(APIServer pid=76251)            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
(APIServer pid=76251)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=76251)   File "/opt/venv/lib64/python3.13/site-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=76251)     return await main
(APIServer pid=76251)            ^^^^^^^^^^
(APIServer pid=76251)   File "/opt/venv/lib64/python3.13/site-packages/vllm/entrypoints/openai/api_server.py", line 1398, in run_server
(APIServer pid=76251)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=76251)   File "/opt/venv/lib64/python3.13/site-packages/vllm/entrypoints/openai/api_server.py", line 1417, in run_server_worker
(APIServer pid=76251)     async with build_async_engine_client(
(APIServer pid=76251)                ~~~~~~~~~~~~~~~~~~~~~~~~~^
(APIServer pid=76251)         args,
(APIServer pid=76251)         ^^^^^
(APIServer pid=76251)         client_config=client_config,
(APIServer pid=76251)         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=76251)     ) as engine_client:
(APIServer pid=76251)     ^
(APIServer pid=76251)   File "/usr/lib64/python3.13/contextlib.py", line 214, in __aenter__
(APIServer pid=76251)     return await anext(self.gen)
(APIServer pid=76251)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=76251)   File "/opt/venv/lib64/python3.13/site-packages/vllm/entrypoints/openai/api_server.py", line 172, in build_async_engine_client
(APIServer pid=76251)     async with build_async_engine_client_from_engine_args(
(APIServer pid=76251)                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
(APIServer pid=76251)         engine_args,
(APIServer pid=76251)         ^^^^^^^^^^^^
(APIServer pid=76251)     ...<2 lines>...
(APIServer pid=76251)         client_config=client_config,
(APIServer pid=76251)         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=76251)     ) as engine:
(APIServer pid=76251)     ^
(APIServer pid=76251)   File "/usr/lib64/python3.13/contextlib.py", line 214, in __aenter__
(APIServer pid=76251)     return await anext(self.gen)
(APIServer pid=76251)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=76251)   File "/opt/venv/lib64/python3.13/site-packages/vllm/entrypoints/openai/api_server.py", line 213, in build_async_engine_client_from_engine_args
(APIServer pid=76251)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=76251)         vllm_config=vllm_config,
(APIServer pid=76251)     ...<6 lines>...
(APIServer pid=76251)         client_index=client_index,
(APIServer pid=76251)     )
(APIServer pid=76251)   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/async_llm.py", line 215, in from_vllm_config
(APIServer pid=76251)     return cls(
(APIServer pid=76251)         vllm_config=vllm_config,
(APIServer pid=76251)     ...<9 lines>...
(APIServer pid=76251)         client_index=client_index,
(APIServer pid=76251)     )
(APIServer pid=76251)   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/async_llm.py", line 134, in __init__
(APIServer pid=76251)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=76251)                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
(APIServer pid=76251)         vllm_config=vllm_config,
(APIServer pid=76251)         ^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=76251)     ...<4 lines>...
(APIServer pid=76251)         client_index=client_index,
(APIServer pid=76251)         ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=76251)     )
(APIServer pid=76251)     ^
(APIServer pid=76251)   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core_client.py", line 121, in make_async_mp_client
(APIServer pid=76251)     return AsyncMPClient(*client_args)
(APIServer pid=76251)   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core_client.py", line 820, in __init__
(APIServer pid=76251)     super().__init__(
(APIServer pid=76251)     ~~~~~~~~~~~~~~~~^
(APIServer pid=76251)         asyncio_mode=True,
(APIServer pid=76251)         ^^^^^^^^^^^^^^^^^^
(APIServer pid=76251)     ...<3 lines>...
(APIServer pid=76251)         client_addresses=client_addresses,
(APIServer pid=76251)         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=76251)     )
(APIServer pid=76251)     ^
(APIServer pid=76251)   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/core_client.py", line 477, in __init__
(APIServer pid=76251)     with launch_core_engines(vllm_config, executor_class, log_stats) as (
(APIServer pid=76251)          ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=76251)   File "/usr/lib64/python3.13/contextlib.py", line 148, in __exit__
(APIServer pid=76251)     next(self.gen)
(APIServer pid=76251)     ~~~~^^^^^^^^^^
(APIServer pid=76251)   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/utils.py", line 903, in launch_core_engines
(APIServer pid=76251)     wait_for_engine_startup(
(APIServer pid=76251)     ~~~~~~~~~~~~~~~~~~~~~~~^
(APIServer pid=76251)         handshake_socket,
(APIServer pid=76251)         ^^^^^^^^^^^^^^^^^
(APIServer pid=76251)     ...<5 lines>...
(APIServer pid=76251)         coordinator.proc if coordinator else None,
(APIServer pid=76251)         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=76251)     )
(APIServer pid=76251)     ^
(APIServer pid=76251)   File "/opt/venv/lib64/python3.13/site-packages/vllm/v1/engine/utils.py", line 960, in wait_for_engine_startup
(APIServer pid=76251)     raise RuntimeError(
(APIServer pid=76251)     ...<3 lines>...
(APIServer pid=76251)     )
(APIServer pid=76251) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}