cjw/amd-r9700-vllm-toolboxes
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a04b9ef9c5807c3937429e93fd2166aebb347518
amd-r9700-vllm-toolboxes/benchmarks/benchmark_results_nvidia-5090/meta-llama_Meta-Llama-3.1-8B-Instruct_tp1_server.log
T
cjw a04b9ef9c5 x
2026-03-26 20:50:00 +08:00

113 KiB
Raw Blame History

WARNING 12-09 18:18:16 [argparse_utils.py:195] With `vllm serve`, you should provide the model as a positional argument or in a config file instead of via the `--model` option. The `--model` option will be removed in v0.13.
(APIServer pid=4406) INFO 12-09 18:18:16 [api_server.py:1772] vLLM API server version 0.12.0
(APIServer pid=4406) INFO 12-09 18:18:16 [utils.py:253] non-default args: {'model_tag': 'meta-llama/Meta-Llama-3.1-8B-Instruct', 'host': '127.0.0.1', 'model': 'meta-llama/Meta-Llama-3.1-8B-Instruct', 'max_model_len': 65536, 'gpu_memory_utilization': 0.95, 'max_num_seqs': 64}
(APIServer pid=4406) INFO 12-09 18:18:23 [model.py:637] Resolved architecture: LlamaForCausalLM
(APIServer pid=4406) INFO 12-09 18:18:23 [model.py:1750] Using max model len 65536
(APIServer pid=4406) INFO 12-09 18:18:24 [scheduler.py:228] Chunked prefill is enabled with max_num_batched_tokens=2048.
(EngineCore_DP0 pid=4670) INFO 12-09 18:18:31 [core.py:93] Initializing a V1 LLM engine (v0.12.0) with config: model='meta-llama/Meta-Llama-3.1-8B-Instruct', speculative_config=None, tokenizer='meta-llama/Meta-Llama-3.1-8B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=65536, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01), seed=0, served_model_name=meta-llama/Meta-Llama-3.1-8B-Instruct, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>}, 'local_cache_dir': None}
(EngineCore_DP0 pid=4670) INFO 12-09 18:18:31 [parallel_state.py:1200] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.17.0.4:34945 backend=nccl
(EngineCore_DP0 pid=4670) INFO 12-09 18:18:31 [parallel_state.py:1408] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
(EngineCore_DP0 pid=4670) INFO 12-09 18:18:32 [gpu_model_runner.py:3467] Starting to load model meta-llama/Meta-Llama-3.1-8B-Instruct...
(EngineCore_DP0 pid=4670) INFO 12-09 18:18:32 [cuda.py:411] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']
(EngineCore_DP0 pid=4670) INFO 12-09 18:18:34 [weight_utils.py:487] Time spent downloading weights for meta-llama/Meta-Llama-3.1-8B-Instruct: 0.680210 seconds
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:00<00:00, 9.79it/s]
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:00<00:00, 3.49it/s]
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:00<00:00, 2.71it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:01<00:00, 2.47it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:01<00:00, 2.77it/s]
(EngineCore_DP0 pid=4670)
(EngineCore_DP0 pid=4670) INFO 12-09 18:18:36 [default_loader.py:308] Loading weights took 1.51 seconds
(EngineCore_DP0 pid=4670) INFO 12-09 18:18:36 [gpu_model_runner.py:3549] Model loading took 14.9889 GiB memory and 3.826109 seconds
(EngineCore_DP0 pid=4670) INFO 12-09 18:18:40 [backends.py:655] Using cache directory: /root/.cache/vllm/torch_compile_cache/6d8c7a30a1/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=4670) INFO 12-09 18:18:40 [backends.py:715] Dynamo bytecode transform time: 3.46 s
(EngineCore_DP0 pid=4670) INFO 12-09 18:18:40 [backends.py:257] Cache the graph for dynamic shape for later use
(EngineCore_DP0 pid=4670) INFO 12-09 18:18:44 [backends.py:288] Compiling a graph for dynamic shape takes 3.77 s
(EngineCore_DP0 pid=4670) INFO 12-09 18:18:45 [monitor.py:34] torch.compile takes 7.23 s in total
(EngineCore_DP0 pid=4670) INFO 12-09 18:18:46 [gpu_worker.py:359] Available KV cache memory: 14.31 GiB
(EngineCore_DP0 pid=4670) INFO 12-09 18:18:46 [kv_cache_utils.py:1286] GPU KV cache size: 117,232 tokens
(EngineCore_DP0 pid=4670) INFO 12-09 18:18:46 [kv_cache_utils.py:1291] Maximum concurrency for 65,536 tokens per request: 1.79x
(EngineCore_DP0 pid=4670) 2025-12-09 18:18:46,684 - INFO - autotuner.py:256 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(EngineCore_DP0 pid=4670) 2025-12-09 18:18:46,695 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process ends
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:00<00:00, 29.38it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:00<00:00, 25.00it/s]
(EngineCore_DP0 pid=4670) INFO 12-09 18:18:48 [gpu_model_runner.py:4466] Graph capturing finished in 2 secs, took -0.07 GiB
(EngineCore_DP0 pid=4670) INFO 12-09 18:18:48 [core.py:254] init engine (profile, create kv cache, warmup model) took 11.78 seconds
(APIServer pid=4406) INFO 12-09 18:18:50 [api_server.py:1520] Supported tasks: ['generate']
(APIServer pid=4406) WARNING 12-09 18:18:51 [model.py:1576] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=4406) INFO 12-09 18:18:51 [serving_responses.py:194] Using default chat sampling params from model: {'temperature': 0.6, 'top_p': 0.9}
(APIServer pid=4406) INFO 12-09 18:18:51 [serving_chat.py:133] Using default chat sampling params from model: {'temperature': 0.6, 'top_p': 0.9}
(APIServer pid=4406) INFO 12-09 18:18:51 [serving_completion.py:73] Using default completion sampling params from model: {'temperature': 0.6, 'top_p': 0.9}
(APIServer pid=4406) INFO 12-09 18:18:51 [serving_chat.py:133] Using default chat sampling params from model: {'temperature': 0.6, 'top_p': 0.9}
(APIServer pid=4406) INFO 12-09 18:18:51 [api_server.py:1847] Starting vLLM API server 0 on http://127.0.0.1:8000
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:38] Available routes are:
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /pause, Methods: POST
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /resume, Methods: POST
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /is_paused, Methods: GET
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /v1/audio/transcriptions, Methods: POST
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /v1/audio/translations, Methods: POST
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /classify, Methods: POST
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /v1/embeddings, Methods: POST
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /score, Methods: POST
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /v1/score, Methods: POST
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /rerank, Methods: POST
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /v1/rerank, Methods: POST
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /v2/rerank, Methods: POST
(APIServer pid=4406) INFO 12-09 18:18:51 [launcher.py:46] Route: /pooling, Methods: POST
(APIServer pid=4406) INFO: Started server process [4406]
(APIServer pid=4406) INFO: Waiting for application startup.
(APIServer pid=4406) INFO: Application startup complete.
(APIServer pid=4406) INFO: 127.0.0.1:47774 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:38058 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:38058 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:19:12 [loggers.py:236] Engine 000: Avg prompt throughput: 5.1 tokens/s, Avg generation throughput: 26.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=4406) INFO: 127.0.0.1:38058 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43446 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43462 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43468 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43462 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:38058 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:38058 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:19:22 [loggers.py:236] Engine 000: Avg prompt throughput: 133.1 tokens/s, Avg generation throughput: 197.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 0.0%
(APIServer pid=4406) INFO: 127.0.0.1:38058 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43468 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45516 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:38058 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:19:32 [loggers.py:236] Engine 000: Avg prompt throughput: 183.1 tokens/s, Avg generation throughput: 190.1 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0%
(APIServer pid=4406) INFO: 127.0.0.1:43468 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43468 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:19:42 [loggers.py:236] Engine 000: Avg prompt throughput: 317.0 tokens/s, Avg generation throughput: 146.9 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 0.0%
(APIServer pid=4406) INFO: 127.0.0.1:59162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43468 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43468 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43468 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:19:52 [loggers.py:236] Engine 000: Avg prompt throughput: 191.1 tokens/s, Avg generation throughput: 169.4 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 0.0%
(APIServer pid=4406) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43468 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43468 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40772 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:20:02 [loggers.py:236] Engine 000: Avg prompt throughput: 331.1 tokens/s, Avg generation throughput: 223.2 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 0.0%
(APIServer pid=4406) INFO: 127.0.0.1:40776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43468 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:38070 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43468 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:20:12 [loggers.py:236] Engine 000: Avg prompt throughput: 312.8 tokens/s, Avg generation throughput: 217.8 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 0.0%
(APIServer pid=4406) INFO: 127.0.0.1:59174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:50990 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:50994 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:50994 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:50990 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:20:22 [loggers.py:236] Engine 000: Avg prompt throughput: 236.2 tokens/s, Avg generation throughput: 225.7 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=4406) INFO: 127.0.0.1:50990 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:50990 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45630 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45632 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45630 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:50990 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45654 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45654 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:20:32 [loggers.py:236] Engine 000: Avg prompt throughput: 373.5 tokens/s, Avg generation throughput: 250.2 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 0.0%
(APIServer pid=4406) INFO: 127.0.0.1:45654 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45630 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:50990 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45632 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45630 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45654 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45632 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:20:42 [loggers.py:236] Engine 000: Avg prompt throughput: 158.1 tokens/s, Avg generation throughput: 331.3 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 0.0%
(APIServer pid=4406) INFO: 127.0.0.1:45638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:50990 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:50990 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45630 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45654 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45632 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:20:52 [loggers.py:236] Engine 000: Avg prompt throughput: 341.1 tokens/s, Avg generation throughput: 253.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=4406) INFO: 127.0.0.1:45654 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45632 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:48880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45654 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45632 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:48882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:48882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:48880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:21:02 [loggers.py:236] Engine 000: Avg prompt throughput: 33.7 tokens/s, Avg generation throughput: 205.1 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 0.0%
(APIServer pid=4406) INFO: 127.0.0.1:48880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:48882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:48880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45632 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45654 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:48882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:21:12 [loggers.py:236] Engine 000: Avg prompt throughput: 92.8 tokens/s, Avg generation throughput: 150.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
(APIServer pid=4406) INFO: 127.0.0.1:45654 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:48880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45632 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:48882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:33166 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:48880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:33182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:48882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:48880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45632 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:48882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:33182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:48880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45654 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:48880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:33182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:21:22 [loggers.py:236] Engine 000: Avg prompt throughput: 418.1 tokens/s, Avg generation throughput: 306.9 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 0.0%
(APIServer pid=4406) INFO: 127.0.0.1:45654 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:48882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:33166 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:33182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:45632 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:33182 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:48882 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:48880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:21:32 [loggers.py:236] Engine 000: Avg prompt throughput: 68.9 tokens/s, Avg generation throughput: 261.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
(APIServer pid=4406) INFO: 127.0.0.1:45632 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:33166 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:33166 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:54582 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:21:42 [loggers.py:236] Engine 000: Avg prompt throughput: 96.3 tokens/s, Avg generation throughput: 64.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=4406) INFO: 127.0.0.1:33166 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:54582 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:54582 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:33166 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:54582 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:49268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:49270 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:49270 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:49280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:49268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:33166 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:21:52 [loggers.py:236] Engine 000: Avg prompt throughput: 138.8 tokens/s, Avg generation throughput: 192.0 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 0.0%
(APIServer pid=4406) INFO: 127.0.0.1:49280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:33166 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:49270 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:33166 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:49280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:49280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:22:02 [loggers.py:236] Engine 000: Avg prompt throughput: 111.5 tokens/s, Avg generation throughput: 178.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 0.0%
(APIServer pid=4406) INFO: 127.0.0.1:33166 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:54290 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:54290 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:33166 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51430 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51430 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:49270 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:54290 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:49280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:33166 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:22:12 [loggers.py:236] Engine 000: Avg prompt throughput: 243.1 tokens/s, Avg generation throughput: 280.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0%
(APIServer pid=4406) INFO 12-09 18:22:22 [loggers.py:236] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 30.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39988 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39988 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:22:32 [loggers.py:236] Engine 000: Avg prompt throughput: 319.1 tokens/s, Avg generation throughput: 183.8 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 7.4%
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39988 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39988 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39988 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39988 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56548 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:22:42 [loggers.py:236] Engine 000: Avg prompt throughput: 909.5 tokens/s, Avg generation throughput: 756.3 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 23.5%
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56548 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56548 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39988 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56548 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39988 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39988 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:22:52 [loggers.py:236] Engine 000: Avg prompt throughput: 1132.5 tokens/s, Avg generation throughput: 930.0 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 36.9%
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56548 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39988 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:23:02 [loggers.py:236] Engine 000: Avg prompt throughput: 779.4 tokens/s, Avg generation throughput: 946.8 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 43.5%
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56548 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39988 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56548 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39988 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39988 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:23:12 [loggers.py:236] Engine 000: Avg prompt throughput: 635.8 tokens/s, Avg generation throughput: 727.9 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 48.0%
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39988 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56548 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39988 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59200 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39988 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:23:22 [loggers.py:236] Engine 000: Avg prompt throughput: 844.9 tokens/s, Avg generation throughput: 1070.0 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 43.2%
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56548 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39988 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59200 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56548 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56548 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59200 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56548 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39988 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:23:32 [loggers.py:236] Engine 000: Avg prompt throughput: 1011.7 tokens/s, Avg generation throughput: 784.2 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 38.6%
(APIServer pid=4406) INFO: 127.0.0.1:56548 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39988 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59200 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56548 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:23:42 [loggers.py:236] Engine 000: Avg prompt throughput: 552.5 tokens/s, Avg generation throughput: 738.9 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 36.4%
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39988 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56548 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59200 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43262 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43262 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43282 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39988 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56548 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43282 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39988 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43296 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43296 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39988 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59200 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:23:52 [loggers.py:236] Engine 000: Avg prompt throughput: 978.7 tokens/s, Avg generation throughput: 1092.9 tokens/s, Running: 21 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.8%, Prefix cache hit rate: 33.2%
(APIServer pid=4406) INFO: 127.0.0.1:43262 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43282 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59200 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39988 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43282 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59200 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43296 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56548 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43262 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59200 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:24:02 [loggers.py:236] Engine 000: Avg prompt throughput: 418.4 tokens/s, Avg generation throughput: 1030.0 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 32.0%
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43296 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43282 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39988 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43262 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59200 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56548 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43282 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43262 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43296 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56548 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39988 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43282 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:24:12 [loggers.py:236] Engine 000: Avg prompt throughput: 832.9 tokens/s, Avg generation throughput: 711.1 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 29.8%
(APIServer pid=4406) INFO: 127.0.0.1:43280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59200 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43296 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43262 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56530 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56548 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59200 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:24:22 [loggers.py:236] Engine 000: Avg prompt throughput: 868.6 tokens/s, Avg generation throughput: 622.1 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 27.8%
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59200 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59200 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:24:32 [loggers.py:236] Engine 000: Avg prompt throughput: 487.4 tokens/s, Avg generation throughput: 626.6 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.7%, Prefix cache hit rate: 26.8%
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59200 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59200 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59200 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59200 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59200 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:34286 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:24:42 [loggers.py:236] Engine 000: Avg prompt throughput: 927.5 tokens/s, Avg generation throughput: 1013.2 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 25.2%
(APIServer pid=4406) INFO: 127.0.0.1:43280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56518 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:34286 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43294 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59200 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:34286 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:24:52 [loggers.py:236] Engine 000: Avg prompt throughput: 733.6 tokens/s, Avg generation throughput: 654.3 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 24.0%
(APIServer pid=4406) INFO: 127.0.0.1:59216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:34286 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:34286 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:34286 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:25:02 [loggers.py:236] Engine 000: Avg prompt throughput: 889.4 tokens/s, Avg generation throughput: 635.6 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 22.7%
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:34286 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:34286 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:34286 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:25:12 [loggers.py:236] Engine 000: Avg prompt throughput: 639.9 tokens/s, Avg generation throughput: 816.2 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 21.8%
(APIServer pid=4406) INFO: 127.0.0.1:59202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:34286 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:34286 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40020 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39978 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:34286 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:25:22 [loggers.py:236] Engine 000: Avg prompt throughput: 926.1 tokens/s, Avg generation throughput: 852.2 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 20.7%
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39966 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:34286 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40032 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:34286 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:56534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:44608 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:44624 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:44634 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:34286 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59216 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:59202 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:43266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40026 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:40008 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:34286 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO: 127.0.0.1:39992 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=4406) INFO 12-09 18:25:32 [loggers.py:236] Engine 000: Avg prompt throughput: 693.4 tokens/s, Avg generation throughput: 932.3 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 19.9%
(APIServer pid=4406) INFO 12-09 18:25:36 [launcher.py:110] Shutting down FastAPI HTTP server.
[rank0]:[W1209 18:28:56.997423394 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
[rank0]:[W1209 18:28:56.102110333 AllocatorConfig.cpp:28] Warning: PYTORCH_CUDA_ALLOC_CONF is deprecated, use PYTORCH_ALLOC_CONF instead (function operator())
Reference in New Issue View Git Blame Copy Permalink