cjw/amd-r9700-vllm-toolboxes
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
9acf1322e1cdb4b223af8b771af4841576a308cb
amd-r9700-vllm-toolboxes/benchmarks/benchmark_results_nvidia-3090/cpatonn_Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit_tp1_server.log
T
cjw a04b9ef9c5 x
2026-03-26 20:50:00 +08:00

115 KiB
Raw Blame History

WARNING 12-10 09:44:20 [argparse_utils.py:195] With `vllm serve`, you should provide the model as a positional argument or in a config file instead of via the `--model` option. The `--model` option will be removed in v0.13.
(APIServer pid=3923) INFO 12-10 09:44:20 [api_server.py:1772] vLLM API server version 0.12.0
(APIServer pid=3923) INFO 12-10 09:44:20 [utils.py:253] non-default args: {'model_tag': 'cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit', 'host': '127.0.0.1', 'model': 'cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit', 'trust_remote_code': True, 'max_model_len': 24576, 'max_num_seqs': 64}
(APIServer pid=3923) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=3923) INFO 12-10 09:44:29 [model.py:637] Resolved architecture: Qwen3MoeForCausalLM
(APIServer pid=3923) INFO 12-10 09:44:29 [model.py:1750] Using max model len 24576
(APIServer pid=3923) INFO 12-10 09:44:29 [scheduler.py:228] Chunked prefill is enabled with max_num_batched_tokens=2048.
(EngineCore_DP0 pid=3963) INFO 12-10 09:44:37 [core.py:93] Initializing a V1 LLM engine (v0.12.0) with config: model='cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit', speculative_config=None, tokenizer='cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=24576, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01), seed=0, served_model_name=cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>}, 'local_cache_dir': None}
(EngineCore_DP0 pid=3963) INFO 12-10 09:44:38 [parallel_state.py:1200] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.17.0.2:49511 backend=nccl
(EngineCore_DP0 pid=3963) INFO 12-10 09:44:38 [parallel_state.py:1408] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
(EngineCore_DP0 pid=3963) INFO 12-10 09:44:38 [gpu_model_runner.py:3467] Starting to load model cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit...
(EngineCore_DP0 pid=3963) INFO 12-10 09:44:39 [compressed_tensors_wNa16.py:114] Using MarlinLinearKernel for CompressedTensorsWNA16
(EngineCore_DP0 pid=3963) INFO 12-10 09:44:39 [cuda.py:411] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']
(EngineCore_DP0 pid=3963) INFO 12-10 09:44:39 [layer.py:379] Enabled separate cuda stream for MoE shared_experts
(EngineCore_DP0 pid=3963) INFO 12-10 09:44:39 [compressed_tensors_moe.py:167] Using CompressedTensorsWNA16MarlinMoEMethod
(EngineCore_DP0 pid=3963) WARNING 12-10 09:44:39 [compressed_tensors.py:721] Acceleration for non-quantized schemes is not supported by Compressed Tensors. Falling back to UnquantizedLinearMethod
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:01<00:03, 1.25s/it]
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:07<00:08, 4.48s/it]
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:13<00:04, 4.95s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:18<00:00, 5.14s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:18<00:00, 4.73s/it]
(EngineCore_DP0 pid=3963)
(EngineCore_DP0 pid=3963) INFO 12-10 09:44:59 [default_loader.py:308] Loading weights took 19.02 seconds
(EngineCore_DP0 pid=3963) INFO 12-10 09:45:02 [gpu_model_runner.py:3549] Model loading took 15.6116 GiB memory and 22.703472 seconds
(EngineCore_DP0 pid=3963) INFO 12-10 09:45:16 [backends.py:655] Using cache directory: /root/.cache/vllm/torch_compile_cache/950831accb/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=3963) INFO 12-10 09:45:16 [backends.py:715] Dynamo bytecode transform time: 13.68 s
(EngineCore_DP0 pid=3963) INFO 12-10 09:45:17 [backends.py:257] Cache the graph for dynamic shape for later use
(EngineCore_DP0 pid=3963) INFO 12-10 09:45:33 [backends.py:288] Compiling a graph for dynamic shape takes 16.19 s
(EngineCore_DP0 pid=3963) INFO 12-10 09:45:37 [monitor.py:34] torch.compile takes 29.88 s in total
(EngineCore_DP0 pid=3963) INFO 12-10 09:45:38 [gpu_worker.py:359] Available KV cache memory: 5.08 GiB
(EngineCore_DP0 pid=3963) INFO 12-10 09:45:38 [kv_cache_utils.py:1286] GPU KV cache size: 55,504 tokens
(EngineCore_DP0 pid=3963) INFO 12-10 09:45:38 [kv_cache_utils.py:1291] Maximum concurrency for 24,576 tokens per request: 2.26x
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:02<00:00, 9.33it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:01<00:00, 9.06it/s]
(EngineCore_DP0 pid=3963) INFO 12-10 09:45:42 [gpu_model_runner.py:4466] Graph capturing finished in 4 secs, took 0.32 GiB
(EngineCore_DP0 pid=3963) INFO 12-10 09:45:42 [core.py:254] init engine (profile, create kv cache, warmup model) took 40.65 seconds
(APIServer pid=3923) INFO 12-10 09:45:44 [api_server.py:1520] Supported tasks: ['generate']
(APIServer pid=3923) WARNING 12-10 09:45:44 [model.py:1576] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=3923) INFO 12-10 09:45:44 [serving_responses.py:194] Using default chat sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=3923) INFO 12-10 09:45:44 [serving_chat.py:133] Using default chat sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=3923) INFO 12-10 09:45:45 [serving_completion.py:73] Using default completion sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=3923) INFO 12-10 09:45:45 [serving_chat.py:133] Using default chat sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=3923) INFO 12-10 09:45:45 [api_server.py:1847] Starting vLLM API server 0 on http://127.0.0.1:8000
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:38] Available routes are:
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /docs, Methods: HEAD, GET
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /pause, Methods: POST
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /resume, Methods: POST
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /is_paused, Methods: GET
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /v1/audio/transcriptions, Methods: POST
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /v1/audio/translations, Methods: POST
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /classify, Methods: POST
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /v1/embeddings, Methods: POST
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /score, Methods: POST
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /v1/score, Methods: POST
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /rerank, Methods: POST
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /v1/rerank, Methods: POST
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /v2/rerank, Methods: POST
(APIServer pid=3923) INFO 12-10 09:45:45 [launcher.py:46] Route: /pooling, Methods: POST
(APIServer pid=3923) INFO: Started server process [3923]
(APIServer pid=3923) INFO: Waiting for application startup.
(APIServer pid=3923) INFO: Application startup complete.
(APIServer pid=3923) INFO: 127.0.0.1:48268 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:36798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:46:05 [loggers.py:236] Engine 000: Avg prompt throughput: 1.2 tokens/s, Avg generation throughput: 11.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:36798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:36798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:36804 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:36820 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:36830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:36804 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:36830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:36820 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:46:15 [loggers.py:236] Engine 000: Avg prompt throughput: 115.5 tokens/s, Avg generation throughput: 208.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:36798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:36820 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:36804 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:36798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:36804 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:36820 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:36804 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:36798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:36804 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:46:25 [loggers.py:236] Engine 000: Avg prompt throughput: 207.6 tokens/s, Avg generation throughput: 185.4 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:36798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:36804 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:46:35 [loggers.py:236] Engine 000: Avg prompt throughput: 193.4 tokens/s, Avg generation throughput: 169.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:36804 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:36804 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:36804 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:58676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:46:45 [loggers.py:236] Engine 000: Avg prompt throughput: 309.4 tokens/s, Avg generation throughput: 153.0 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:58676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:36804 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:58676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:36804 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:58676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:46:55 [loggers.py:236] Engine 000: Avg prompt throughput: 205.4 tokens/s, Avg generation throughput: 216.8 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:47788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:58676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:58676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:58676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54042 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:36804 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54042 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:58676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:47:05 [loggers.py:236] Engine 000: Avg prompt throughput: 438.1 tokens/s, Avg generation throughput: 182.5 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:58676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54042 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:58676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:46190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:58676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:36804 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:46192 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:46206 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:46192 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:46206 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54042 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:36804 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:47:15 [loggers.py:236] Engine 000: Avg prompt throughput: 257.7 tokens/s, Avg generation throughput: 253.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:46190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:46190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55740 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55750 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55740 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47696 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47700 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:47:25 [loggers.py:236] Engine 000: Avg prompt throughput: 369.4 tokens/s, Avg generation throughput: 207.2 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:47696 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55750 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47700 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55740 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47696 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:46190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55750 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47696 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55740 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:47:35 [loggers.py:236] Engine 000: Avg prompt throughput: 174.9 tokens/s, Avg generation throughput: 381.0 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:47700 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55750 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47700 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:46190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47700 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:47:45 [loggers.py:236] Engine 000: Avg prompt throughput: 332.3 tokens/s, Avg generation throughput: 256.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:55740 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55750 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55740 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55750 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55740 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55740 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:47:55 [loggers.py:236] Engine 000: Avg prompt throughput: 45.4 tokens/s, Avg generation throughput: 243.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:55732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55750 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55750 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:48:05 [loggers.py:236] Engine 000: Avg prompt throughput: 201.2 tokens/s, Avg generation throughput: 163.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:55732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55750 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55750 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55750 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54550 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54564 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54550 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54550 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:48:15 [loggers.py:236] Engine 000: Avg prompt throughput: 242.5 tokens/s, Avg generation throughput: 279.2 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:55732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55750 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54544 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55750 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54550 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:42490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54564 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:48:25 [loggers.py:236] Engine 000: Avg prompt throughput: 103.3 tokens/s, Avg generation throughput: 257.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:55732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55750 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55750 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:48:35 [loggers.py:236] Engine 000: Avg prompt throughput: 93.6 tokens/s, Avg generation throughput: 57.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:59334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59336 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59350 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59360 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59374 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59360 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59350 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59336 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59360 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59374 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:48:45 [loggers.py:236] Engine 000: Avg prompt throughput: 106.2 tokens/s, Avg generation throughput: 227.5 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:59350 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59336 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59374 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59360 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:48:55 [loggers.py:236] Engine 000: Avg prompt throughput: 140.1 tokens/s, Avg generation throughput: 182.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:59374 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59360 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59374 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59360 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:39204 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:39204 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:39214 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:39224 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:39238 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:39238 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59374 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59360 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59374 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59360 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:39214 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:49:05 [loggers.py:236] Engine 000: Avg prompt throughput: 299.7 tokens/s, Avg generation throughput: 275.1 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO 12-10 09:49:15 [loggers.py:236] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 100.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:55070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47320 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:49:25 [loggers.py:236] Engine 000: Avg prompt throughput: 84.0 tokens/s, Avg generation throughput: 62.7 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47320 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47342 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47372 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47408 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47408 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47342 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56468 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47320 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47320 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:49:35 [loggers.py:236] Engine 000: Avg prompt throughput: 904.2 tokens/s, Avg generation throughput: 551.0 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.0%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:47386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47320 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56468 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56468 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47342 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56468 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47342 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47308 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47320 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47408 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:49:45 [loggers.py:236] Engine 000: Avg prompt throughput: 1189.1 tokens/s, Avg generation throughput: 781.8 tokens/s, Running: 21 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.3%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:56474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47372 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47320 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47342 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47342 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56468 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47342 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47408 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:49:55 [loggers.py:236] Engine 000: Avg prompt throughput: 799.4 tokens/s, Avg generation throughput: 982.2 tokens/s, Running: 28 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.1%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:56900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56908 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47372 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47408 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56908 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47372 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47320 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56468 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47320 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:50:05 [loggers.py:236] Engine 000: Avg prompt throughput: 534.6 tokens/s, Avg generation throughput: 871.2 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.7%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:56908 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47408 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47372 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47320 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56908 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47372 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47320 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47320 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56908 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38296 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:50:15 [loggers.py:236] Engine 000: Avg prompt throughput: 902.5 tokens/s, Avg generation throughput: 893.7 tokens/s, Running: 24 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.3%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:56512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38296 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47320 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56908 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47408 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56908 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38296 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:50:25 [loggers.py:236] Engine 000: Avg prompt throughput: 1066.5 tokens/s, Avg generation throughput: 896.8 tokens/s, Running: 19 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.8%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:47386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47372 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56484 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47320 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38296 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:50:35 [loggers.py:236] Engine 000: Avg prompt throughput: 615.8 tokens/s, Avg generation throughput: 847.9 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.7%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:56908 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47320 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38296 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56908 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38296 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54304 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38296 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54326 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:50:45 [loggers.py:236] Engine 000: Avg prompt throughput: 820.1 tokens/s, Avg generation throughput: 860.9 tokens/s, Running: 31 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.6%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:54334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54342 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47320 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54342 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54342 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56908 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54342 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54304 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47320 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38296 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:50:55 [loggers.py:236] Engine 000: Avg prompt throughput: 702.4 tokens/s, Avg generation throughput: 1158.2 tokens/s, Running: 26 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.3%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:38268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54326 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47320 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54342 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54304 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54326 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56908 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38296 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:51:05 [loggers.py:236] Engine 000: Avg prompt throughput: 906.1 tokens/s, Avg generation throughput: 802.7 tokens/s, Running: 22 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.2%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:56522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:55070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54326 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54342 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56908 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54304 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54326 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:51:15 [loggers.py:236] Engine 000: Avg prompt throughput: 916.0 tokens/s, Avg generation throughput: 747.6 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.1%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:38296 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54326 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54304 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:51:25 [loggers.py:236] Engine 000: Avg prompt throughput: 364.6 tokens/s, Avg generation throughput: 724.6 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.5%, Prefix cache hit rate: 0.0%
(APIServer pid=3923) INFO: 127.0.0.1:38296 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54342 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54326 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54304 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38296 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54304 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54342 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38296 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54326 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54342 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:51:35 [loggers.py:236] Engine 000: Avg prompt throughput: 834.5 tokens/s, Avg generation throughput: 747.1 tokens/s, Running: 19 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.9%, Prefix cache hit rate: 0.1%
(APIServer pid=3923) INFO: 127.0.0.1:47424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54342 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54326 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:33102 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54304 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54326 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:33102 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54326 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54304 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:51:45 [loggers.py:236] Engine 000: Avg prompt throughput: 1052.8 tokens/s, Avg generation throughput: 747.8 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.5%, Prefix cache hit rate: 0.1%
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:33102 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54342 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54326 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38296 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54304 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:51:55 [loggers.py:236] Engine 000: Avg prompt throughput: 551.0 tokens/s, Avg generation throughput: 680.2 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.5%, Prefix cache hit rate: 0.1%
(APIServer pid=3923) INFO: 127.0.0.1:54342 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54304 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38296 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:33102 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54342 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:33102 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:52:05 [loggers.py:236] Engine 000: Avg prompt throughput: 780.0 tokens/s, Avg generation throughput: 811.9 tokens/s, Running: 19 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.1%, Prefix cache hit rate: 0.1%
(APIServer pid=3923) INFO: 127.0.0.1:54304 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38296 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54326 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54342 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54326 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38296 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54304 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54342 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54326 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:52:15 [loggers.py:236] Engine 000: Avg prompt throughput: 846.9 tokens/s, Avg generation throughput: 791.3 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.7%, Prefix cache hit rate: 0.1%
(APIServer pid=3923) INFO: 127.0.0.1:54342 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:33102 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54304 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47334 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47356 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47416 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54342 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56910 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47424 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56876 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47346 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54304 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54342 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:60264 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:60272 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:60276 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:56474 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:33102 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:47386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:59572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:38296 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:60280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:60280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:49038 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:60272 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO: 127.0.0.1:54304 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3923) INFO 12-10 09:52:25 [loggers.py:236] Engine 000: Avg prompt throughput: 798.4 tokens/s, Avg generation throughput: 920.5 tokens/s, Running: 23 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.1%, Prefix cache hit rate: 0.1%
Reference in New Issue View Git Blame Copy Permalink