cjw/amd-r9700-vllm-toolboxes
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
9acf1322e1cdb4b223af8b771af4841576a308cb
amd-r9700-vllm-toolboxes/benchmarks/benchmark_results_nvidia-3090/RedHatAI_Qwen3-14B-FP8-dynamic_tp1_server.log
T
cjw a04b9ef9c5 x
2026-03-26 20:50:00 +08:00

113 KiB
Raw Blame History

WARNING 12-10 09:23:01 [argparse_utils.py:195] With `vllm serve`, you should provide the model as a positional argument or in a config file instead of via the `--model` option. The `--model` option will be removed in v0.13.
(APIServer pid=3017) INFO 12-10 09:23:01 [api_server.py:1772] vLLM API server version 0.12.0
(APIServer pid=3017) INFO 12-10 09:23:01 [utils.py:253] non-default args: {'model_tag': 'RedHatAI/Qwen3-14B-FP8-dynamic', 'host': '127.0.0.1', 'model': 'RedHatAI/Qwen3-14B-FP8-dynamic', 'trust_remote_code': True, 'max_model_len': 4096, 'gpu_memory_utilization': 0.86, 'max_num_seqs': 32}
(APIServer pid=3017) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=3017) INFO 12-10 09:23:09 [model.py:637] Resolved architecture: Qwen3ForCausalLM
(APIServer pid=3017) INFO 12-10 09:23:09 [model.py:1750] Using max model len 4096
(APIServer pid=3017) INFO 12-10 09:23:09 [scheduler.py:228] Chunked prefill is enabled with max_num_batched_tokens=2048.
(EngineCore_DP0 pid=3057) INFO 12-10 09:23:18 [core.py:93] Initializing a V1 LLM engine (v0.12.0) with config: model='RedHatAI/Qwen3-14B-FP8-dynamic', speculative_config=None, tokenizer='RedHatAI/Qwen3-14B-FP8-dynamic', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01), seed=0, served_model_name=RedHatAI/Qwen3-14B-FP8-dynamic, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 64, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>}, 'local_cache_dir': None}
(EngineCore_DP0 pid=3057) INFO 12-10 09:23:19 [parallel_state.py:1200] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.17.0.2:36297 backend=nccl
(EngineCore_DP0 pid=3057) INFO 12-10 09:23:19 [parallel_state.py:1408] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
(EngineCore_DP0 pid=3057) INFO 12-10 09:23:19 [gpu_model_runner.py:3467] Starting to load model RedHatAI/Qwen3-14B-FP8-dynamic...
(EngineCore_DP0 pid=3057) INFO 12-10 09:23:19 [cuda.py:411] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:00<00:00, 3.56it/s]
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:01<00:01, 1.52it/s]
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:02<00:00, 1.18it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00, 1.11it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00, 1.23it/s]
(EngineCore_DP0 pid=3057)
(EngineCore_DP0 pid=3057) INFO 12-10 09:23:24 [default_loader.py:308] Loading weights took 3.34 seconds
(EngineCore_DP0 pid=3057) WARNING 12-10 09:23:24 [marlin_utils_fp8.py:98] Your GPU does not have native support for FP8 computation but FP8 quantization is being used. Weight-only FP8 compression will be used leveraging the Marlin kernel. This may degrade performance for compute-heavy workloads.
(EngineCore_DP0 pid=3057) INFO 12-10 09:23:24 [gpu_model_runner.py:3549] Model loading took 15.3291 GiB memory and 4.892989 seconds
(EngineCore_DP0 pid=3057) INFO 12-10 09:23:37 [backends.py:655] Using cache directory: /root/.cache/vllm/torch_compile_cache/882f1c097b/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=3057) INFO 12-10 09:23:37 [backends.py:715] Dynamo bytecode transform time: 12.64 s
(EngineCore_DP0 pid=3057) INFO 12-10 09:23:38 [backends.py:257] Cache the graph for dynamic shape for later use
(EngineCore_DP0 pid=3057) INFO 12-10 09:23:52 [backends.py:288] Compiling a graph for dynamic shape takes 14.28 s
(EngineCore_DP0 pid=3057) INFO 12-10 09:23:55 [monitor.py:34] torch.compile takes 26.92 s in total
(EngineCore_DP0 pid=3057) INFO 12-10 09:23:57 [gpu_worker.py:359] Available KV cache memory: 4.64 GiB
(EngineCore_DP0 pid=3057) INFO 12-10 09:23:57 [kv_cache_utils.py:1286] GPU KV cache size: 30,384 tokens
(EngineCore_DP0 pid=3057) INFO 12-10 09:23:57 [kv_cache_utils.py:1291] Maximum concurrency for 4,096 tokens per request: 7.42x
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 11/11 [00:00<00:00, 16.75it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 7/7 [00:00<00:00, 15.38it/s]
(EngineCore_DP0 pid=3057) INFO 12-10 09:23:59 [gpu_model_runner.py:4466] Graph capturing finished in 2 secs, took 0.14 GiB
(EngineCore_DP0 pid=3057) INFO 12-10 09:23:59 [core.py:254] init engine (profile, create kv cache, warmup model) took 34.60 seconds
(APIServer pid=3017) INFO 12-10 09:24:01 [api_server.py:1520] Supported tasks: ['generate']
(APIServer pid=3017) WARNING 12-10 09:24:01 [model.py:1576] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=3017) INFO 12-10 09:24:01 [serving_responses.py:194] Using default chat sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
(APIServer pid=3017) INFO 12-10 09:24:01 [serving_chat.py:133] Using default chat sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
(APIServer pid=3017) INFO 12-10 09:24:01 [serving_completion.py:73] Using default completion sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
(APIServer pid=3017) INFO 12-10 09:24:01 [serving_chat.py:133] Using default chat sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
(APIServer pid=3017) INFO 12-10 09:24:01 [api_server.py:1847] Starting vLLM API server 0 on http://127.0.0.1:8000
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:38] Available routes are:
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /pause, Methods: POST
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /resume, Methods: POST
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /is_paused, Methods: GET
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /v1/audio/transcriptions, Methods: POST
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /v1/audio/translations, Methods: POST
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /classify, Methods: POST
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /v1/embeddings, Methods: POST
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /score, Methods: POST
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /v1/score, Methods: POST
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /rerank, Methods: POST
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /v1/rerank, Methods: POST
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /v2/rerank, Methods: POST
(APIServer pid=3017) INFO 12-10 09:24:01 [launcher.py:46] Route: /pooling, Methods: POST
(APIServer pid=3017) INFO: Started server process [3017]
(APIServer pid=3017) INFO: Waiting for application startup.
(APIServer pid=3017) INFO: Application startup complete.
(APIServer pid=3017) INFO: 127.0.0.1:60512 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:43808 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:24:21 [loggers.py:236] Engine 000: Avg prompt throughput: 1.2 tokens/s, Avg generation throughput: 3.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:43808 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:41338 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:41354 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:43808 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:41370 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:41386 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:41394 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:41394 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:24:31 [loggers.py:236] Engine 000: Avg prompt throughput: 82.8 tokens/s, Avg generation throughput: 114.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:41370 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:43808 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:41394 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:41370 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:57514 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:41394 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:57514 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:57528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:57538 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:24:41 [loggers.py:236] Engine 000: Avg prompt throughput: 239.0 tokens/s, Avg generation throughput: 155.7 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.8%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:57514 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:41338 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:43808 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:57528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:24:51 [loggers.py:236] Engine 000: Avg prompt throughput: 111.0 tokens/s, Avg generation throughput: 200.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:41370 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:57538 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:57514 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:57514 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47654 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:41370 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:57538 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47654 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:57528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:25:01 [loggers.py:236] Engine 000: Avg prompt throughput: 326.7 tokens/s, Avg generation throughput: 170.1 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.9%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:41370 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47654 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:57538 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:43808 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:43808 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:57514 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:57538 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:41370 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:57514 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:25:11 [loggers.py:236] Engine 000: Avg prompt throughput: 228.1 tokens/s, Avg generation throughput: 202.9 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.3%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:47638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:43808 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:57538 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39912 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39926 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39936 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39926 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39912 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:57528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:57538 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:41370 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:43808 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:25:21 [loggers.py:236] Engine 000: Avg prompt throughput: 276.5 tokens/s, Avg generation throughput: 218.5 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:57528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39936 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39912 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:43808 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39936 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39912 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39936 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:57528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47960 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:57528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47960 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:57528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39936 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:25:31 [loggers.py:236] Engine 000: Avg prompt throughput: 463.0 tokens/s, Avg generation throughput: 259.7 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.1%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:57528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39936 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39912 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39936 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39912 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:57528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:41370 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44586 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:57528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44602 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:25:41 [loggers.py:236] Engine 000: Avg prompt throughput: 268.5 tokens/s, Avg generation throughput: 141.9 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:44612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39936 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39936 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44602 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:41370 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44586 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44586 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:25:51 [loggers.py:236] Engine 000: Avg prompt throughput: 185.4 tokens/s, Avg generation throughput: 341.8 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.1%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:41370 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:60604 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:60612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39936 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:60604 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:57528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39818 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:57528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39912 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44602 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:26:01 [loggers.py:236] Engine 000: Avg prompt throughput: 368.5 tokens/s, Avg generation throughput: 339.9 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.2%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:39912 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:57528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44586 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39912 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:60612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:60604 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39818 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:26:11 [loggers.py:236] Engine 000: Avg prompt throughput: 87.1 tokens/s, Avg generation throughput: 222.4 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:57528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:60604 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39818 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44586 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:57528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:57528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:60604 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:26:21 [loggers.py:236] Engine 000: Avg prompt throughput: 190.0 tokens/s, Avg generation throughput: 199.0 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.2%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:39912 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:60612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39818 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44586 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:60604 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:60604 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39818 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:60604 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:60612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44586 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:56162 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:26:31 [loggers.py:236] Engine 000: Avg prompt throughput: 253.5 tokens/s, Avg generation throughput: 203.6 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.7%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:39912 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44586 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39912 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:57528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44586 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:60612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:60604 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44586 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:49932 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:26:41 [loggers.py:236] Engine 000: Avg prompt throughput: 79.6 tokens/s, Avg generation throughput: 317.7 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.7%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:39912 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44586 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:60612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39912 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:60604 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:26:51 [loggers.py:236] Engine 000: Avg prompt throughput: 130.1 tokens/s, Avg generation throughput: 164.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:44586 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:51262 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44586 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:51264 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:51272 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:51284 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:51272 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:27:01 [loggers.py:236] Engine 000: Avg prompt throughput: 81.0 tokens/s, Avg generation throughput: 91.3 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:51298 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:51264 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:53738 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:53754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:51272 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:51264 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:51298 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:51264 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:27:11 [loggers.py:236] Engine 000: Avg prompt throughput: 139.6 tokens/s, Avg generation throughput: 240.5 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.4%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:53754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:53754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44406 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:53738 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44586 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:53754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:51272 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:53754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44422 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44426 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44426 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:51264 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44406 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:27:21 [loggers.py:236] Engine 000: Avg prompt throughput: 275.0 tokens/s, Avg generation throughput: 214.3 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.3%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:51264 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44406 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44586 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:27:31 [loggers.py:236] Engine 000: Avg prompt throughput: 50.4 tokens/s, Avg generation throughput: 215.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.2%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO 12-10 09:27:41 [loggers.py:236] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:32938 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32938 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32942 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32958 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32970 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32982 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:27:51 [loggers.py:236] Engine 000: Avg prompt throughput: 8.3 tokens/s, Avg generation throughput: 21.1 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:32994 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:33000 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:33000 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:33000 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50140 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32938 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50152 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50168 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32982 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50206 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50210 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50168 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:33000 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32938 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32970 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50226 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50230 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32958 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50232 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32970 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50232 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50238 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50252 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50226 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50232 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50270 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50282 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:28:01 [loggers.py:236] Engine 000: Avg prompt throughput: 851.6 tokens/s, Avg generation throughput: 340.8 tokens/s, Running: 22 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.8%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:50232 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50290 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32970 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50232 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37558 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37564 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37568 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37600 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37604 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50230 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37564 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37558 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37636 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37648 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37674 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37686 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32938 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50230 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50206 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37604 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32982 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50232 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50252 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37686 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32938 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37702 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37708 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50210 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32958 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50282 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50140 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37604 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:28:11 [loggers.py:236] Engine 000: Avg prompt throughput: 1107.8 tokens/s, Avg generation throughput: 558.2 tokens/s, Running: 32 reqs, Waiting: 12 reqs, GPU KV cache usage: 43.5%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:50268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37564 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50252 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50270 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50152 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50290 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46270 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46298 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37568 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37600 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50210 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37674 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:33000 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37564 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50290 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50206 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50230 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37708 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37702 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37600 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50210 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:28:21 [loggers.py:236] Engine 000: Avg prompt throughput: 652.5 tokens/s, Avg generation throughput: 784.0 tokens/s, Running: 31 reqs, Waiting: 13 reqs, GPU KV cache usage: 41.1%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:37636 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37686 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50290 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37568 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32942 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46306 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32994 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50230 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32958 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37702 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:40234 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32938 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50226 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:40246 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32970 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37648 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50238 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:40250 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50168 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:40266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37636 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50282 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37674 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37686 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37558 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:33000 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46306 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37604 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50206 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:28:31 [loggers.py:236] Engine 000: Avg prompt throughput: 540.5 tokens/s, Avg generation throughput: 844.7 tokens/s, Running: 32 reqs, Waiting: 19 reqs, GPU KV cache usage: 41.3%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:50190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50270 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32994 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32970 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46270 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50140 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32942 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37708 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39460 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50238 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:40266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37636 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50210 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37702 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50252 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37686 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50226 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50168 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50232 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50152 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50206 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50270 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:33000 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50140 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32938 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:40250 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39468 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39482 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39496 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39460 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:40234 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:40266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39504 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37636 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46298 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:28:41 [loggers.py:236] Engine 000: Avg prompt throughput: 676.0 tokens/s, Avg generation throughput: 761.4 tokens/s, Running: 32 reqs, Waiting: 29 reqs, GPU KV cache usage: 38.2%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:37620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32982 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37568 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37686 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:40246 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37674 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50230 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32994 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37600 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37564 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50270 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46270 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50140 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50168 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:40250 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37648 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47738 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39460 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47746 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47756 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32938 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47782 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47802 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32958 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47820 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32970 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:28:51 [loggers.py:236] Engine 000: Avg prompt throughput: 626.5 tokens/s, Avg generation throughput: 812.6 tokens/s, Running: 32 reqs, Waiting: 41 reqs, GPU KV cache usage: 50.9%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:50174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37558 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37568 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32982 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:40246 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32942 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50090 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50102 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46306 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37636 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32994 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50118 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50124 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50130 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50142 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37686 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39468 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37564 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37604 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37600 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46298 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:40234 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50282 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50252 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:29:01 [loggers.py:236] Engine 000: Avg prompt throughput: 842.0 tokens/s, Avg generation throughput: 681.5 tokens/s, Running: 32 reqs, Waiting: 34 reqs, GPU KV cache usage: 36.7%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:50140 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39460 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:33000 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39496 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47746 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37702 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50238 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47756 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50168 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50290 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50206 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37708 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50226 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50210 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47820 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39504 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37674 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32982 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:40246 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:40266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32942 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50230 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37636 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50232 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47738 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46306 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50142 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39468 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37568 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50090 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:40250 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:29:11 [loggers.py:236] Engine 000: Avg prompt throughput: 790.7 tokens/s, Avg generation throughput: 748.6 tokens/s, Running: 32 reqs, Waiting: 42 reqs, GPU KV cache usage: 31.1%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:40234 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50152 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50140 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39482 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46298 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37648 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32970 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36818 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32938 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36848 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36850 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37702 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50270 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36862 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36864 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50130 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47756 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50206 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50118 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47782 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37564 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36878 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37558 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50102 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39496 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:33000 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37600 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39460 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:40266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32942 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32982 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32958 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:29:21 [loggers.py:236] Engine 000: Avg prompt throughput: 495.8 tokens/s, Avg generation throughput: 905.6 tokens/s, Running: 31 reqs, Waiting: 55 reqs, GPU KV cache usage: 34.4%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:50252 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50230 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47802 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50282 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50290 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50142 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37686 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50226 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47746 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46270 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50124 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37604 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37674 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50238 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:40246 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37708 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:52174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50140 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36818 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50168 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:52178 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36850 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50232 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36862 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36864 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37702 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32994 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:29:31 [loggers.py:236] Engine 000: Avg prompt throughput: 713.1 tokens/s, Avg generation throughput: 790.4 tokens/s, Running: 32 reqs, Waiting: 55 reqs, GPU KV cache usage: 30.0%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:50190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50210 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36848 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50152 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:52184 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:52200 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:52208 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39504 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37648 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47756 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47820 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50102 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37636 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50090 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37568 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39460 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32982 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32942 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39468 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32938 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:53676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:53682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39482 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:53690 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50290 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50142 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:29:41 [loggers.py:236] Engine 000: Avg prompt throughput: 608.2 tokens/s, Avg generation throughput: 847.9 tokens/s, Running: 32 reqs, Waiting: 57 reqs, GPU KV cache usage: 34.9%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:46270 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50226 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37686 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50270 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37604 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32958 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47782 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50238 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37708 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37558 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32970 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:40266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:40234 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36818 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50230 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36878 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36850 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46298 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50232 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50124 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46306 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:29:51 [loggers.py:236] Engine 000: Avg prompt throughput: 874.2 tokens/s, Avg generation throughput: 687.9 tokens/s, Running: 32 reqs, Waiting: 49 reqs, GPU KV cache usage: 37.8%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:37702 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37674 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:33000 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50252 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47746 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36848 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47802 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39504 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:52174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47820 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37564 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47756 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47738 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50102 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50140 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37568 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50206 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39468 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32938 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39460 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50282 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:53690 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36864 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50130 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:52200 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37600 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37648 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46270 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37686 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32994 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:40250 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32958 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:30:01 [loggers.py:236] Engine 000: Avg prompt throughput: 842.1 tokens/s, Avg generation throughput: 700.7 tokens/s, Running: 32 reqs, Waiting: 55 reqs, GPU KV cache usage: 35.1%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:53682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:53676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37558 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50168 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50152 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37604 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32982 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39482 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50270 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50118 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50230 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46438 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:52208 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:52184 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50290 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32942 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:40246 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46306 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37708 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50124 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50210 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:33000 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39496 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37702 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36848 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37636 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47782 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46450 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46464 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50226 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:40266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50142 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47820 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:30:11 [loggers.py:236] Engine 000: Avg prompt throughput: 754.0 tokens/s, Avg generation throughput: 764.7 tokens/s, Running: 31 reqs, Waiting: 61 reqs, GPU KV cache usage: 32.4%, Prefix cache hit rate: 0.0%
(APIServer pid=3017) INFO: 127.0.0.1:36868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47738 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32970 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32938 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37568 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:52174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50206 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37564 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50102 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:53690 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36850 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36878 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37600 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50140 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32994 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:52200 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37554 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36818 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37674 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:53682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36864 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39504 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47746 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:30:21 [loggers.py:236] Engine 000: Avg prompt throughput: 650.4 tokens/s, Avg generation throughput: 803.2 tokens/s, Running: 32 reqs, Waiting: 54 reqs, GPU KV cache usage: 31.5%, Prefix cache hit rate: 0.1%
(APIServer pid=3017) INFO: 127.0.0.1:37604 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50252 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39482 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50090 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50238 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46270 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37648 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37558 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50270 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:52184 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50290 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32942 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:40246 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50230 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37686 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50124 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50152 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:52208 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36848 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50168 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50282 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39496 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:40234 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:40266 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:30:31 [loggers.py:236] Engine 000: Avg prompt throughput: 886.2 tokens/s, Avg generation throughput: 684.7 tokens/s, Running: 32 reqs, Waiting: 49 reqs, GPU KV cache usage: 36.0%, Prefix cache hit rate: 0.1%
(APIServer pid=3017) INFO: 127.0.0.1:47756 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50142 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46438 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32982 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39468 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46464 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46450 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47782 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37658 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32970 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37708 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:33000 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32958 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50130 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50206 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:52174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36850 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46298 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37702 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:53690 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:53676 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47802 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36818 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37674 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:52200 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37620 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50232 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32938 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47746 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50228 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47776 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50210 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39504 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50190 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:30:41 [loggers.py:236] Engine 000: Avg prompt throughput: 661.1 tokens/s, Avg generation throughput: 767.9 tokens/s, Running: 31 reqs, Waiting: 55 reqs, GPU KV cache usage: 34.6%, Prefix cache hit rate: 0.1%
(APIServer pid=3017) INFO: 127.0.0.1:50174 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47764 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50118 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:52184 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39482 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46306 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50226 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47836 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50230 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47820 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50288 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50124 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37562 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44426 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44434 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47788 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:53682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36848 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39460 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50168 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37604 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44444 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44450 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:37558 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36864 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44460 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44462 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:36878 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50238 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:46438 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47738 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:32994 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44470 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50102 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44476 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44480 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44502 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50090 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:47724 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:39528 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:40250 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50140 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44512 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44534 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44542 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:44552 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO: 127.0.0.1:50270 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3017) INFO 12-10 09:30:51 [loggers.py:236] Engine 000: Avg prompt throughput: 557.5 tokens/s, Avg generation throughput: 880.0 tokens/s, Running: 31 reqs, Waiting: 78 reqs, GPU KV cache usage: 36.0%, Prefix cache hit rate: 0.1%
(APIServer pid=3017) INFO 12-10 09:31:01 [loggers.py:236] Engine 000: Avg prompt throughput: 944.9 tokens/s, Avg generation throughput: 614.4 tokens/s, Running: 31 reqs, Waiting: 40 reqs, GPU KV cache usage: 34.8%, Prefix cache hit rate: 0.1%
(APIServer pid=3017) INFO 12-10 09:31:11 [loggers.py:236] Engine 000: Avg prompt throughput: 585.6 tokens/s, Avg generation throughput: 895.5 tokens/s, Running: 31 reqs, Waiting: 0 reqs, GPU KV cache usage: 30.5%, Prefix cache hit rate: 0.1%
(APIServer pid=3017) INFO 12-10 09:31:21 [loggers.py:236] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 611.0 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.7%, Prefix cache hit rate: 0.1%
Reference in New Issue View Git Blame Copy Permalink