amd-r9700-vllm-toolboxes/benchmarks/benchmark_results_nvidia-5090/cpatonn_Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit_tp1_server.log at 9acf1322e1cdb4b223af8b771af4841576a308cb

Files

T

cjw a04b9ef9c5 x

2026-03-26 20:50:00 +08:00

115 KiB

Raw Blame History

WARNING 12-09 19:23:27 [argparse_utils.py:195] With `vllm serve`, you should provide the model as a positional argument or in a config file instead of via the `--model` option. The `--model` option will be removed in v0.13.
(APIServer pid=17193) INFO 12-09 19:23:27 [api_server.py:1772] vLLM API server version 0.12.0
(APIServer pid=17193) INFO 12-09 19:23:27 [utils.py:253] non-default args: {'model_tag': 'cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit', 'host': '127.0.0.1', 'model': 'cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit', 'trust_remote_code': True, 'max_model_len': 24576, 'max_num_seqs': 64}
(APIServer pid=17193) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=17193) INFO 12-09 19:23:33 [model.py:637] Resolved architecture: Qwen3MoeForCausalLM
(APIServer pid=17193) INFO 12-09 19:23:33 [model.py:1750] Using max model len 24576
(APIServer pid=17193) INFO 12-09 19:23:35 [scheduler.py:228] Chunked prefill is enabled with max_num_batched_tokens=2048.
(EngineCore_DP0 pid=17457) INFO 12-09 19:23:41 [core.py:93] Initializing a V1 LLM engine (v0.12.0) with config: model='cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit', speculative_config=None, tokenizer='cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=24576, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01), seed=0, served_model_name=cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>}, 'local_cache_dir': None}
(EngineCore_DP0 pid=17457) INFO 12-09 19:23:42 [parallel_state.py:1200] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.17.0.4:60659 backend=nccl
(EngineCore_DP0 pid=17457) INFO 12-09 19:23:42 [parallel_state.py:1408] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
(EngineCore_DP0 pid=17457) INFO 12-09 19:23:42 [gpu_model_runner.py:3467] Starting to load model cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit...
(EngineCore_DP0 pid=17457) INFO 12-09 19:23:42 [compressed_tensors_wNa16.py:114] Using MarlinLinearKernel for CompressedTensorsWNA16
(EngineCore_DP0 pid=17457) INFO 12-09 19:23:43 [cuda.py:411] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']
(EngineCore_DP0 pid=17457) INFO 12-09 19:23:43 [layer.py:379] Enabled separate cuda stream for MoE shared_experts
(EngineCore_DP0 pid=17457) INFO 12-09 19:23:43 [compressed_tensors_moe.py:167] Using CompressedTensorsWNA16MarlinMoEMethod
(EngineCore_DP0 pid=17457) WARNING 12-09 19:23:43 [compressed_tensors.py:721] Acceleration for non-quantized schemes is not supported by Compressed Tensors. Falling back to UnquantizedLinearMethod
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:00<00:01, 1.51it/s]
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:03<00:03, 1.78s/it]
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:06<00:02, 2.30s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:09<00:00, 2.54s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:09<00:00, 2.26s/it]
(EngineCore_DP0 pid=17457)
(EngineCore_DP0 pid=17457) INFO 12-09 19:23:53 [default_loader.py:308] Loading weights took 9.15 seconds
(EngineCore_DP0 pid=17457) INFO 12-09 19:23:54 [gpu_model_runner.py:3549] Model loading took 15.6117 GiB memory and 11.126164 seconds
(EngineCore_DP0 pid=17457) INFO 12-09 19:24:02 [backends.py:655] Using cache directory: /root/.cache/vllm/torch_compile_cache/06a729e350/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=17457) INFO 12-09 19:24:02 [backends.py:715] Dynamo bytecode transform time: 7.30 s
(EngineCore_DP0 pid=17457) INFO 12-09 19:24:03 [backends.py:257] Cache the graph for dynamic shape for later use
(EngineCore_DP0 pid=17457) INFO 12-09 19:24:09 [backends.py:288] Compiling a graph for dynamic shape takes 6.57 s
(EngineCore_DP0 pid=17457) INFO 12-09 19:24:11 [monitor.py:34] torch.compile takes 13.87 s in total
(EngineCore_DP0 pid=17457) INFO 12-09 19:24:12 [gpu_worker.py:359] Available KV cache memory: 12.06 GiB
(EngineCore_DP0 pid=17457) INFO 12-09 19:24:12 [kv_cache_utils.py:1286] GPU KV cache size: 131,696 tokens
(EngineCore_DP0 pid=17457) INFO 12-09 19:24:12 [kv_cache_utils.py:1291] Maximum concurrency for 24,576 tokens per request: 5.36x
(EngineCore_DP0 pid=17457) 2025-12-09 19:24:12,801 - INFO - autotuner.py:256 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(EngineCore_DP0 pid=17457) 2025-12-09 19:24:12,826 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process ends
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:01<00:00, 17.17it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:00<00:00, 15.41it/s]
(EngineCore_DP0 pid=17457) INFO 12-09 19:24:15 [gpu_model_runner.py:4466] Graph capturing finished in 3 secs, took 0.08 GiB
(EngineCore_DP0 pid=17457) INFO 12-09 19:24:15 [core.py:254] init engine (profile, create kv cache, warmup model) took 20.79 seconds
(APIServer pid=17193) INFO 12-09 19:24:17 [api_server.py:1520] Supported tasks: ['generate']
(APIServer pid=17193) WARNING 12-09 19:24:17 [model.py:1576] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=17193) INFO 12-09 19:24:17 [serving_responses.py:194] Using default chat sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=17193) INFO 12-09 19:24:17 [serving_chat.py:133] Using default chat sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=17193) INFO 12-09 19:24:17 [serving_completion.py:73] Using default completion sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=17193) INFO 12-09 19:24:17 [serving_chat.py:133] Using default chat sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=17193) INFO 12-09 19:24:17 [api_server.py:1847] Starting vLLM API server 0 on http://127.0.0.1:8000
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:38] Available routes are:
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /pause, Methods: POST
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /resume, Methods: POST
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /is_paused, Methods: GET
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /v1/audio/transcriptions, Methods: POST
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /v1/audio/translations, Methods: POST
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /classify, Methods: POST
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /v1/embeddings, Methods: POST
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /score, Methods: POST
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /v1/score, Methods: POST
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /rerank, Methods: POST
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /v1/rerank, Methods: POST
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /v2/rerank, Methods: POST
(APIServer pid=17193) INFO 12-09 19:24:17 [launcher.py:46] Route: /pooling, Methods: POST
(APIServer pid=17193) INFO: Started server process [17193]
(APIServer pid=17193) INFO: Waiting for application startup.
(APIServer pid=17193) INFO: Application startup complete.
(APIServer pid=17193) INFO: 127.0.0.1:44312 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:50102 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:50102 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:50102 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:50116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:24:38 [loggers.py:236] Engine 000: Avg prompt throughput: 7.4 tokens/s, Avg generation throughput: 45.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0%
(APIServer pid=17193) INFO: 127.0.0.1:50118 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:50116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:50116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:50118 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:50118 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:50118 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:50116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:24:48 [loggers.py:236] Engine 000: Avg prompt throughput: 131.2 tokens/s, Avg generation throughput: 218.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0%
(APIServer pid=17193) INFO: 127.0.0.1:50118 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:50116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:50118 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:50118 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:50116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:50116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:50116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:50118 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:24:58 [loggers.py:236] Engine 000: Avg prompt throughput: 224.6 tokens/s, Avg generation throughput: 169.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=17193) INFO: 127.0.0.1:48658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:52054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:52054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:52054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:25:08 [loggers.py:236] Engine 000: Avg prompt throughput: 279.4 tokens/s, Avg generation throughput: 173.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 0.0%
(APIServer pid=17193) INFO: 127.0.0.1:52054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:52054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:52054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44582 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:52054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:52054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:25:18 [loggers.py:236] Engine 000: Avg prompt throughput: 217.3 tokens/s, Avg generation throughput: 232.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
(APIServer pid=17193) INFO: 127.0.0.1:48658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44582 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:52054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44582 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:52054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44582 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:52054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:25:28 [loggers.py:236] Engine 000: Avg prompt throughput: 375.3 tokens/s, Avg generation throughput: 169.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=17193) INFO: 127.0.0.1:44582 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44582 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44720 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44720 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44720 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44720 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:50934 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:50942 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:50934 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:50952 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:50952 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:25:38 [loggers.py:236] Engine 000: Avg prompt throughput: 405.3 tokens/s, Avg generation throughput: 292.2 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 0.0%
(APIServer pid=17193) INFO: 127.0.0.1:44582 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:50952 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44720 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:50952 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:25:48 [loggers.py:236] Engine 000: Avg prompt throughput: 215.3 tokens/s, Avg generation throughput: 85.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=17193) INFO: 127.0.0.1:50952 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:50952 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:50952 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:53208 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:53212 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:53224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:50952 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:53224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:53208 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:53212 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:25:58 [loggers.py:236] Engine 000: Avg prompt throughput: 298.4 tokens/s, Avg generation throughput: 362.5 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 0.0%
(APIServer pid=17193) INFO: 127.0.0.1:56520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:53212 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:53224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56536 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:53224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:53212 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56520 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:26:08 [loggers.py:236] Engine 000: Avg prompt throughput: 225.0 tokens/s, Avg generation throughput: 387.5 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 0.0%
(APIServer pid=17193) INFO: 127.0.0.1:53224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:53224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:53224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:26:18 [loggers.py:236] Engine 000: Avg prompt throughput: 232.7 tokens/s, Avg generation throughput: 137.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:53224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:53224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:53224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:26:28 [loggers.py:236] Engine 000: Avg prompt throughput: 41.1 tokens/s, Avg generation throughput: 209.6 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0%
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:53224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:53224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:26:38 [loggers.py:236] Engine 000: Avg prompt throughput: 232.2 tokens/s, Avg generation throughput: 173.7 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0%
(APIServer pid=17193) INFO: 127.0.0.1:53224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:53224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:53224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:53224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:55508 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:55508 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:53224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:53224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:55508 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:26:48 [loggers.py:236] Engine 000: Avg prompt throughput: 219.2 tokens/s, Avg generation throughput: 378.3 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0%
(APIServer pid=17193) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:53224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:55508 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:55508 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:26:58 [loggers.py:236] Engine 000: Avg prompt throughput: 152.8 tokens/s, Avg generation throughput: 149.2 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=17193) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:42868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:42868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:27:08 [loggers.py:236] Engine 000: Avg prompt throughput: 70.6 tokens/s, Avg generation throughput: 31.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0%
(APIServer pid=17193) INFO: 127.0.0.1:42880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:42894 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:42868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:42868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:42894 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:42868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:42894 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:42880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:42868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:42894 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:42868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:27:18 [loggers.py:236] Engine 000: Avg prompt throughput: 172.9 tokens/s, Avg generation throughput: 315.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 0.0%
(APIServer pid=17193) INFO: 127.0.0.1:42868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:42880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:42894 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:42880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:42894 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:42880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:42894 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44956 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:27:28 [loggers.py:236] Engine 000: Avg prompt throughput: 130.7 tokens/s, Avg generation throughput: 128.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0%
(APIServer pid=17193) INFO: 127.0.0.1:44956 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:42880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:42894 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44976 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44976 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:42880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44956 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:44976 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:27:38 [loggers.py:236] Engine 000: Avg prompt throughput: 205.6 tokens/s, Avg generation throughput: 313.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=17193) INFO 12-09 19:27:48 [loggers.py:236] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:27:58 [loggers.py:236] Engine 000: Avg prompt throughput: 643.5 tokens/s, Avg generation throughput: 585.6 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 13.8%
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:28:08 [loggers.py:236] Engine 000: Avg prompt throughput: 1223.3 tokens/s, Avg generation throughput: 825.2 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 31.5%
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:28:18 [loggers.py:236] Engine 000: Avg prompt throughput: 786.0 tokens/s, Avg generation throughput: 1045.9 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 39.2%
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:28:28 [loggers.py:236] Engine 000: Avg prompt throughput: 684.0 tokens/s, Avg generation throughput: 784.1 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 44.7%
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:28:38 [loggers.py:236] Engine 000: Avg prompt throughput: 819.3 tokens/s, Avg generation throughput: 915.2 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 46.1%
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:28:48 [loggers.py:236] Engine 000: Avg prompt throughput: 1072.9 tokens/s, Avg generation throughput: 861.7 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 40.6%
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:28:58 [loggers.py:236] Engine 000: Avg prompt throughput: 761.7 tokens/s, Avg generation throughput: 838.3 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 37.5%
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:29:08 [loggers.py:236] Engine 000: Avg prompt throughput: 580.1 tokens/s, Avg generation throughput: 807.0 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 35.4%
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:29:18 [loggers.py:236] Engine 000: Avg prompt throughput: 853.1 tokens/s, Avg generation throughput: 1296.5 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 32.7%
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:29:28 [loggers.py:236] Engine 000: Avg prompt throughput: 677.4 tokens/s, Avg generation throughput: 744.9 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 30.8%
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56794 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:29:38 [loggers.py:236] Engine 000: Avg prompt throughput: 1176.6 tokens/s, Avg generation throughput: 773.8 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 28.1%
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:29:48 [loggers.py:236] Engine 000: Avg prompt throughput: 428.8 tokens/s, Avg generation throughput: 638.3 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 27.2%
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:29:58 [loggers.py:236] Engine 000: Avg prompt throughput: 826.0 tokens/s, Avg generation throughput: 697.1 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 25.8%
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:30:08 [loggers.py:236] Engine 000: Avg prompt throughput: 936.3 tokens/s, Avg generation throughput: 831.8 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 24.2%
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:30:18 [loggers.py:236] Engine 000: Avg prompt throughput: 701.2 tokens/s, Avg generation throughput: 591.3 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 23.1%
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:30:28 [loggers.py:236] Engine 000: Avg prompt throughput: 682.6 tokens/s, Avg generation throughput: 893.5 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 22.2%
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:30:38 [loggers.py:236] Engine 000: Avg prompt throughput: 674.3 tokens/s, Avg generation throughput: 756.9 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 21.3%
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:45980 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36254 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:48814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46010 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:46018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:55776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:55786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:55802 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:55806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:55812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:55822 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:55836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:55802 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36244 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO 12-09 19:30:48 [loggers.py:236] Engine 000: Avg prompt throughput: 1070.2 tokens/s, Avg generation throughput: 907.6 tokens/s, Running: 18 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.2%, Prefix cache hit rate: 20.1%
(APIServer pid=17193) INFO: 127.0.0.1:55786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:55822 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:55776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:55842 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:55842 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:56762 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:55812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=17193) INFO: 127.0.0.1:36236 - "POST /v1/completions HTTP/1.1" 200 OK
/usr/lib/python3.12/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '

115 KiB Raw Blame History

115 KiB

Raw Blame History