/opt/venv/lib64/python3.13/site-packages/torch/library.py:357: UserWarning: Warning only once for all operators, other operators may also be overridden. Overriding a previously registered kernel for the same operator and the same dispatch key operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 dispatch key: ADInplaceOrView previous kernel: no debug info new kernel: registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 (Triggered internally at /__w/TheRock/TheRock/external-builds/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.) self.m.impl( (APIServer pid=10267) INFO 12-11 19:27:42 [api_server.py:1351] vLLM API server version 0.11.2.dev690+g67475a6e8.d20251209 (APIServer pid=10267) INFO 12-11 19:27:42 [utils.py:253] non-default args: {'model_tag': 'RedHatAI/gemma-3-12b-it-FP8-dynamic', 'host': '127.0.0.1', 'model': 'RedHatAI/gemma-3-12b-it-FP8-dynamic', 'trust_remote_code': True, 'max_model_len': 9900, 'tensor_parallel_size': 2, 'gpu_memory_utilization': 0.98, 'max_num_seqs': 64} (APIServer pid=10267) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. (APIServer pid=10267) INFO 12-11 19:27:46 [model.py:629] Resolved architecture: Gemma3ForConditionalGeneration (APIServer pid=10267) INFO 12-11 19:27:46 [model.py:1755] Using max model len 9900 (APIServer pid=10267) INFO 12-11 19:27:46 [scheduler.py:228] Chunked prefill is enabled with max_num_batched_tokens=2048. /opt/venv/lib64/python3.13/site-packages/torch/library.py:357: UserWarning: Warning only once for all operators, other operators may also be overridden. Overriding a previously registered kernel for the same operator and the same dispatch key operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 dispatch key: ADInplaceOrView previous kernel: no debug info new kernel: registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 (Triggered internally at /__w/TheRock/TheRock/external-builds/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.) self.m.impl( (EngineCore_DP0 pid=10431) INFO 12-11 19:27:51 [core.py:93] Initializing a V1 LLM engine (v0.11.2.dev690+g67475a6e8.d20251209) with config: model='RedHatAI/gemma-3-12b-it-FP8-dynamic', speculative_config=None, tokenizer='RedHatAI/gemma-3-12b-it-FP8-dynamic', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=9900, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=True, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False), seed=0, served_model_name=RedHatAI/gemma-3-12b-it-FP8-dynamic, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False}, 'local_cache_dir': None} (EngineCore_DP0 pid=10431) WARNING 12-11 19:27:51 [multiproc_executor.py:880] Reducing Torch parallelism from 24 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. /opt/venv/lib64/python3.13/site-packages/torch/library.py:357: UserWarning: Warning only once for all operators, other operators may also be overridden. Overriding a previously registered kernel for the same operator and the same dispatch key operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 dispatch key: ADInplaceOrView previous kernel: no debug info new kernel: registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 (Triggered internally at /__w/TheRock/TheRock/external-builds/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.) self.m.impl( /opt/venv/lib64/python3.13/site-packages/torch/library.py:357: UserWarning: Warning only once for all operators, other operators may also be overridden. Overriding a previously registered kernel for the same operator and the same dispatch key operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 dispatch key: ADInplaceOrView previous kernel: no debug info new kernel: registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 (Triggered internally at /__w/TheRock/TheRock/external-builds/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.) self.m.impl( INFO 12-11 19:27:56 [parallel_state.py:1203] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:56779 backend=nccl INFO 12-11 19:27:56 [parallel_state.py:1203] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:56779 backend=nccl INFO 12-11 19:27:56 [pynccl.py:111] vLLM is using nccl==2.27.3 INFO 12-11 19:27:57 [parallel_state.py:1411] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0 INFO 12-11 19:27:57 [parallel_state.py:1411] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 1, EP rank 1 Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`. Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`. (Worker_TP0 pid=10513) INFO 12-11 19:28:03 [gpu_model_runner.py:3544] Starting to load model RedHatAI/gemma-3-12b-it-FP8-dynamic... (Worker_TP1 pid=10514) WARNING 12-11 19:28:03 [compressed_tensors.py:742] Acceleration for non-quantized schemes is not supported by Compressed Tensors. Falling back to UnquantizedLinearMethod (Worker_TP1 pid=10514) INFO 12-11 19:28:03 [layer.py:524] Using AttentionBackendEnum.TORCH_SDPA for MultiHeadAttention in multimodal encoder. (Worker_TP1 pid=10514) WARNING 12-11 19:28:03 [activation.py:544] [ROCm] PyTorch's native GELU with tanh approximation is unstable. Falling back to GELU(approximate='none'). (Worker_TP1 pid=10514) INFO 12-11 19:28:03 [rocm.py:320] Using Triton Attention backend on V1 engine. (Worker_TP1 pid=10514) WARNING 12-11 19:28:03 [activation.py:220] [ROCm] PyTorch's native GELU with tanh approximation is unstable with torch.compile. For native implementation, fallback to 'none' approximation. The custom kernel implementation is unaffected. (Worker_TP0 pid=10513) WARNING 12-11 19:28:03 [compressed_tensors.py:742] Acceleration for non-quantized schemes is not supported by Compressed Tensors. Falling back to UnquantizedLinearMethod (Worker_TP0 pid=10513) INFO 12-11 19:28:03 [layer.py:524] Using AttentionBackendEnum.TORCH_SDPA for MultiHeadAttention in multimodal encoder. (Worker_TP0 pid=10513) WARNING 12-11 19:28:03 [activation.py:544] [ROCm] PyTorch's native GELU with tanh approximation is unstable. Falling back to GELU(approximate='none'). (Worker_TP0 pid=10513) INFO 12-11 19:28:03 [rocm.py:320] Using Triton Attention backend on V1 engine. (Worker_TP0 pid=10513) WARNING 12-11 19:28:03 [activation.py:220] [ROCm] PyTorch's native GELU with tanh approximation is unstable with torch.compile. For native implementation, fallback to 'none' approximation. The custom kernel implementation is unaffected. (Worker_TP0 pid=10513) Loading safetensors checkpoint shards: 0% Completed | 0/3 [00:00