cjw/amd-r9700-vllm-toolboxes
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
9acf1322e1cdb4b223af8b771af4841576a308cb
amd-r9700-vllm-toolboxes/benchmarks/benchmark_results_nvidia-4090/RedHatAI_Qwen3-14B-FP8-dynamic_tp1_server.log
T
cjw a04b9ef9c5 x
2026-03-26 20:50:00 +08:00

113 KiB
Raw Blame History

WARNING 12-09 21:03:27 [argparse_utils.py:195] With `vllm serve`, you should provide the model as a positional argument or in a config file instead of via the `--model` option. The `--model` option will be removed in v0.13.
(APIServer pid=31169) INFO 12-09 21:03:27 [api_server.py:1772] vLLM API server version 0.12.0
(APIServer pid=31169) INFO 12-09 21:03:27 [utils.py:253] non-default args: {'model_tag': 'RedHatAI/Qwen3-14B-FP8-dynamic', 'host': '127.0.0.1', 'model': 'RedHatAI/Qwen3-14B-FP8-dynamic', 'trust_remote_code': True, 'max_model_len': 4096, 'gpu_memory_utilization': 0.86, 'max_num_seqs': 32}
(APIServer pid=31169) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=31169) INFO 12-09 21:03:34 [model.py:637] Resolved architecture: Qwen3ForCausalLM
(APIServer pid=31169) INFO 12-09 21:03:34 [model.py:1750] Using max model len 4096
(APIServer pid=31169) INFO 12-09 21:03:34 [scheduler.py:228] Chunked prefill is enabled with max_num_batched_tokens=2048.
(EngineCore_DP0 pid=31480) INFO 12-09 21:03:41 [core.py:93] Initializing a V1 LLM engine (v0.12.0) with config: model='RedHatAI/Qwen3-14B-FP8-dynamic', speculative_config=None, tokenizer='RedHatAI/Qwen3-14B-FP8-dynamic', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01), seed=0, served_model_name=RedHatAI/Qwen3-14B-FP8-dynamic, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 64, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>}, 'local_cache_dir': None}
(EngineCore_DP0 pid=31480) INFO 12-09 21:03:42 [parallel_state.py:1200] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.17.0.4:34731 backend=nccl
(EngineCore_DP0 pid=31480) INFO 12-09 21:03:42 [parallel_state.py:1408] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
(EngineCore_DP0 pid=31480) INFO 12-09 21:03:42 [gpu_model_runner.py:3467] Starting to load model RedHatAI/Qwen3-14B-FP8-dynamic...
(EngineCore_DP0 pid=31480) INFO 12-09 21:03:43 [cuda.py:411] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:00<00:00, 4.87it/s]
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:00<00:00, 2.32it/s]
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:01<00:00, 1.78it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:02<00:00, 1.59it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:02<00:00, 1.78it/s]
(EngineCore_DP0 pid=31480)
(EngineCore_DP0 pid=31480) INFO 12-09 21:03:46 [default_loader.py:308] Loading weights took 2.41 seconds
(EngineCore_DP0 pid=31480) INFO 12-09 21:03:46 [gpu_model_runner.py:3549] Model loading took 15.3388 GiB memory and 3.593561 seconds
(EngineCore_DP0 pid=31480) INFO 12-09 21:03:59 [backends.py:655] Using cache directory: /root/.cache/vllm/torch_compile_cache/57323a05ea/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=31480) INFO 12-09 21:03:59 [backends.py:715] Dynamo bytecode transform time: 12.33 s
(EngineCore_DP0 pid=31480) INFO 12-09 21:04:00 [backends.py:257] Cache the graph for dynamic shape for later use
(EngineCore_DP0 pid=31480) INFO 12-09 21:04:09 [backends.py:288] Compiling a graph for dynamic shape takes 8.80 s
(EngineCore_DP0 pid=31480) INFO 12-09 21:04:15 [monitor.py:34] torch.compile takes 21.13 s in total
(EngineCore_DP0 pid=31480) INFO 12-09 21:04:16 [gpu_worker.py:359] Available KV cache memory: 4.43 GiB
(EngineCore_DP0 pid=31480) INFO 12-09 21:04:16 [kv_cache_utils.py:1286] GPU KV cache size: 29,008 tokens
(EngineCore_DP0 pid=31480) INFO 12-09 21:04:16 [kv_cache_utils.py:1291] Maximum concurrency for 4,096 tokens per request: 7.08x
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 11/11 [00:00<00:00, 22.34it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 7/7 [00:00<00:00, 23.77it/s]
(EngineCore_DP0 pid=31480) INFO 12-09 21:04:18 [gpu_model_runner.py:4466] Graph capturing finished in 1 secs, took 0.16 GiB
(EngineCore_DP0 pid=31480) INFO 12-09 21:04:18 [core.py:254] init engine (profile, create kv cache, warmup model) took 31.35 seconds
(APIServer pid=31169) INFO 12-09 21:04:20 [api_server.py:1520] Supported tasks: ['generate']
(APIServer pid=31169) WARNING 12-09 21:04:20 [model.py:1576] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=31169) INFO 12-09 21:04:20 [serving_responses.py:194] Using default chat sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
(APIServer pid=31169) INFO 12-09 21:04:20 [serving_chat.py:133] Using default chat sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
(APIServer pid=31169) INFO 12-09 21:04:20 [serving_completion.py:73] Using default completion sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
(APIServer pid=31169) INFO 12-09 21:04:20 [serving_chat.py:133] Using default chat sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
(APIServer pid=31169) INFO 12-09 21:04:20 [api_server.py:1847] Starting vLLM API server 0 on http://127.0.0.1:8000
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:38] Available routes are:
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /pause, Methods: POST
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /resume, Methods: POST
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /is_paused, Methods: GET
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /v1/audio/transcriptions, Methods: POST
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /v1/audio/translations, Methods: POST
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /classify, Methods: POST
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /v1/embeddings, Methods: POST
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /score, Methods: POST
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /v1/score, Methods: POST
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /rerank, Methods: POST
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /v1/rerank, Methods: POST
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /v2/rerank, Methods: POST
(APIServer pid=31169) INFO 12-09 21:04:20 [launcher.py:46] Route: /pooling, Methods: POST
(APIServer pid=31169) INFO: Started server process [31169]
(APIServer pid=31169) INFO: Waiting for application startup.
(APIServer pid=31169) INFO: Application startup complete.
(APIServer pid=31169) INFO: 127.0.0.1:40408 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59962 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:04:41 [loggers.py:236] Engine 000: Avg prompt throughput: 1.2 tokens/s, Avg generation throughput: 8.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:59962 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59962 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:52006 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:52012 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:52018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:52018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:04:51 [loggers.py:236] Engine 000: Avg prompt throughput: 115.5 tokens/s, Avg generation throughput: 124.6 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:52006 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59962 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:52018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:52006 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:52018 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:05:01 [loggers.py:236] Engine 000: Avg prompt throughput: 206.3 tokens/s, Avg generation throughput: 190.9 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.6%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:38054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59962 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:05:11 [loggers.py:236] Engine 000: Avg prompt throughput: 118.5 tokens/s, Avg generation throughput: 163.7 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.2%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:42066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42074 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42074 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42074 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:05:21 [loggers.py:236] Engine 000: Avg prompt throughput: 323.1 tokens/s, Avg generation throughput: 183.8 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:42074 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59962 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59962 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59962 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:05:31 [loggers.py:236] Engine 000: Avg prompt throughput: 253.3 tokens/s, Avg generation throughput: 210.5 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.9%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:40828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42074 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:40844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:40844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:40844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59962 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:40844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:05:41 [loggers.py:236] Engine 000: Avg prompt throughput: 386.6 tokens/s, Avg generation throughput: 210.6 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:42074 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:40844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42074 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:40844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38054 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42074 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:48642 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:48658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:48642 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:48658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:48642 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59962 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:05:51 [loggers.py:236] Engine 000: Avg prompt throughput: 323.8 tokens/s, Avg generation throughput: 268.6 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.8%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:42074 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:48642 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:40844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:48642 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:40844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42074 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:40844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:39932 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:39944 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:48642 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:39958 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:06:01 [loggers.py:236] Engine 000: Avg prompt throughput: 343.8 tokens/s, Avg generation throughput: 145.9 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:39958 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:39958 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:39972 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:39944 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:48642 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:39944 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42074 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:40844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:39958 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:40844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42074 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:39932 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:39972 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:06:11 [loggers.py:236] Engine 000: Avg prompt throughput: 172.3 tokens/s, Avg generation throughput: 359.6 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.3%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:39932 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:60400 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:39944 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:48642 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:48642 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:06:21 [loggers.py:236] Engine 000: Avg prompt throughput: 358.8 tokens/s, Avg generation throughput: 338.5 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.2%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:40844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:39972 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:39932 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:39944 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:40844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:06:31 [loggers.py:236] Engine 000: Avg prompt throughput: 34.6 tokens/s, Avg generation throughput: 187.3 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:39944 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:39972 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:39944 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:40844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:40844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:06:41 [loggers.py:236] Engine 000: Avg prompt throughput: 213.8 tokens/s, Avg generation throughput: 212.6 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.6%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:39932 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:39972 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:40844 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:39944 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:33370 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:39944 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:33370 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:39972 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:39944 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:39932 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:06:51 [loggers.py:236] Engine 000: Avg prompt throughput: 234.9 tokens/s, Avg generation throughput: 213.0 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:59978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:39944 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:33386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:39972 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:39944 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:45454 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:07:01 [loggers.py:236] Engine 000: Avg prompt throughput: 110.0 tokens/s, Avg generation throughput: 318.7 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.2%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:33386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:39944 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:39972 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:07:11 [loggers.py:236] Engine 000: Avg prompt throughput: 94.5 tokens/s, Avg generation throughput: 119.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:33386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34912 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:33386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:33148 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:33152 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:33168 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:33152 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:33148 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:07:21 [loggers.py:236] Engine 000: Avg prompt throughput: 81.6 tokens/s, Avg generation throughput: 118.2 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:33180 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:33186 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:33188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:33152 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:33180 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:33148 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:33180 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:33168 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:07:31 [loggers.py:236] Engine 000: Avg prompt throughput: 164.7 tokens/s, Avg generation throughput: 243.8 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.8%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:33386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:33168 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:33186 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:33386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:33152 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:33152 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:44944 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:33180 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:44946 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:44946 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:33168 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:33168 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:33168 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:07:41 [loggers.py:236] Engine 000: Avg prompt throughput: 251.1 tokens/s, Avg generation throughput: 222.4 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.8%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:33386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:33186 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:07:51 [loggers.py:236] Engine 000: Avg prompt throughput: 48.6 tokens/s, Avg generation throughput: 190.7 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO 12-09 21:08:01 [loggers.py:236] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.7 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:42556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58770 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58770 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:08:11 [loggers.py:236] Engine 000: Avg prompt throughput: 220.0 tokens/s, Avg generation throughput: 98.3 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.2%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:58788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58770 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34100 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34130 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34130 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58770 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34152 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34172 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:08:21 [loggers.py:236] Engine 000: Avg prompt throughput: 1019.5 tokens/s, Avg generation throughput: 588.5 tokens/s, Running: 20 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.1%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:34162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34152 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34100 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34152 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34100 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34194 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34130 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34130 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34194 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34172 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34130 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43126 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34130 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:08:31 [loggers.py:236] Engine 000: Avg prompt throughput: 1216.1 tokens/s, Avg generation throughput: 853.1 tokens/s, Running: 28 reqs, Waiting: 0 reqs, GPU KV cache usage: 40.8%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:34130 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34172 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34152 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34130 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34172 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58770 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34100 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34172 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34152 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59496 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34152 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:08:41 [loggers.py:236] Engine 000: Avg prompt throughput: 648.4 tokens/s, Avg generation throughput: 995.1 tokens/s, Running: 30 reqs, Waiting: 0 reqs, GPU KV cache usage: 38.6%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:59496 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34100 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34152 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43126 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59496 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43126 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34100 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34194 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:08:51 [loggers.py:236] Engine 000: Avg prompt throughput: 534.4 tokens/s, Avg generation throughput: 819.4 tokens/s, Running: 20 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.1%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:58736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34130 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34100 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34130 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59496 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34130 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58786 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59496 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34130 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59496 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59496 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:09:01 [loggers.py:236] Engine 000: Avg prompt throughput: 962.2 tokens/s, Avg generation throughput: 959.0 tokens/s, Running: 27 reqs, Waiting: 0 reqs, GPU KV cache usage: 37.4%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:43126 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34194 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43126 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34156 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34130 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43126 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34130 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59496 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:09:11 [loggers.py:236] Engine 000: Avg prompt throughput: 1031.9 tokens/s, Avg generation throughput: 893.3 tokens/s, Running: 21 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.4%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:34194 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34130 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34100 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59496 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34194 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43126 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:09:21 [loggers.py:236] Engine 000: Avg prompt throughput: 570.4 tokens/s, Avg generation throughput: 799.3 tokens/s, Running: 18 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.4%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:59502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34100 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34130 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59496 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59496 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34800 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34816 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34130 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34100 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34816 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34194 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59496 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:09:31 [loggers.py:236] Engine 000: Avg prompt throughput: 869.6 tokens/s, Avg generation throughput: 991.5 tokens/s, Running: 32 reqs, Waiting: 3 reqs, GPU KV cache usage: 32.5%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:34194 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34816 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34194 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59496 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43126 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:09:41 [loggers.py:236] Engine 000: Avg prompt throughput: 636.1 tokens/s, Avg generation throughput: 1121.4 tokens/s, Running: 25 reqs, Waiting: 0 reqs, GPU KV cache usage: 33.8%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:34130 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34100 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59494 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34816 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59496 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43126 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58798 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59490 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:09:51 [loggers.py:236] Engine 000: Avg prompt throughput: 859.3 tokens/s, Avg generation throughput: 786.8 tokens/s, Running: 18 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.8%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:59496 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34800 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58758 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43126 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34194 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34800 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:10:01 [loggers.py:236] Engine 000: Avg prompt throughput: 948.0 tokens/s, Avg generation throughput: 688.8 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.9%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:34800 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34100 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34194 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34800 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43126 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34100 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34194 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34800 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:10:11 [loggers.py:236] Engine 000: Avg prompt throughput: 574.2 tokens/s, Avg generation throughput: 701.7 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.7%, Prefix cache hit rate: 0.0%
(APIServer pid=31169) INFO: 127.0.0.1:58752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34800 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:60810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34800 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:60810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34194 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34100 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43126 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:60810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:10:21 [loggers.py:236] Engine 000: Avg prompt throughput: 854.5 tokens/s, Avg generation throughput: 852.7 tokens/s, Running: 28 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.4%, Prefix cache hit rate: 0.1%
(APIServer pid=31169) INFO: 127.0.0.1:58836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:60810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34100 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43126 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34800 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43126 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:10:31 [loggers.py:236] Engine 000: Avg prompt throughput: 846.6 tokens/s, Avg generation throughput: 765.6 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.1%, Prefix cache hit rate: 0.1%
(APIServer pid=31169) INFO: 127.0.0.1:34196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34800 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34100 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:60810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:60810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43126 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:60810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:10:41 [loggers.py:236] Engine 000: Avg prompt throughput: 613.5 tokens/s, Avg generation throughput: 696.1 tokens/s, Running: 19 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.5%, Prefix cache hit rate: 0.1%
(APIServer pid=31169) INFO: 127.0.0.1:34832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:60810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34100 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:60810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34800 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43126 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34100 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:10:51 [loggers.py:236] Engine 000: Avg prompt throughput: 730.2 tokens/s, Avg generation throughput: 825.2 tokens/s, Running: 26 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.3%, Prefix cache hit rate: 0.1%
(APIServer pid=31169) INFO: 127.0.0.1:34128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:60810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34100 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34800 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43126 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34100 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58752 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34116 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34800 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34196 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58774 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:11:01 [loggers.py:236] Engine 000: Avg prompt throughput: 1098.9 tokens/s, Avg generation throughput: 786.1 tokens/s, Running: 22 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.8%, Prefix cache hit rate: 0.1%
(APIServer pid=31169) INFO: 127.0.0.1:43128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34800 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58838 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59144 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58736 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34800 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:43126 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:60810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:58828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:42556 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34340 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34146 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:47978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:47994 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:47996 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:48012 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34128 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:34330 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO: 127.0.0.1:59488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=31169) INFO 12-09 21:11:11 [loggers.py:236] Engine 000: Avg prompt throughput: 434.8 tokens/s, Avg generation throughput: 979.0 tokens/s, Running: 20 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.6%, Prefix cache hit rate: 0.1%
(APIServer pid=31169) INFO 12-09 21:11:21 [loggers.py:236] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 362.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 0.1%
Reference in New Issue View Git Blame Copy Permalink